A method, system, device, and storage medium for visual localization of pathological images.

Abstract

This application provides a method, system, device, and storage medium for visual localization of pathological images, belonging to the field of image recognition technology. The method includes: extracting visual features based on a target pathological image; determining semantic feature vectors and knowledge feature vectors based on a first text description; the target pathological image is the pathological image for which target region localization is to be performed; the knowledge feature vectors are used to represent knowledge information associated with the content of the target pathological image; fusing the semantic feature vectors and knowledge feature vectors to obtain fused text features; performing cross-modal fusion of the fused text features and visual features to obtain fused multimodal features; obtaining a fused representation based on the fused multimodal features; and, based on the fused representation, locating the target region in the target pathological image using a multilayer perceptron to obtain the position information of the bounding box of the target region. This application can improve the ability to accurately and flexibly locate regions at the pathological image level.

Description

Technical Field

[0001] This application belongs to the field of image recognition technology, and more specifically, relates to a method, system, device, and storage medium for visual localization of pathological images. Background Technology

[0002] Pathology is the cornerstone of modern medicine, playing a crucial role in the diagnosis and understanding of diseases. With the development of artificial intelligence technology, computational pathology has made significant progress, with tasks such as cancer subtyping and survival prediction from whole-slice images, cell nucleus segmentation, and pathology visual question answering (PathVQA) being extensively studied.

[0003] However, existing technologies still suffer from low accuracy in pathological image recognition and localization. Therefore, a new visual localization method for pathological images is needed to improve the ability to accurately and flexibly locate pathological image regions. Summary of the Invention

[0004] The purpose of this application is to provide a method, system, device, and storage medium for visual localization of pathological images, so as to improve the ability to accurately and flexibly locate pathological image regions.

[0005] A first aspect of this application provides a method for visual localization of pathological images, comprising:

[0006] Visual features are extracted based on the target pathological image, and semantic feature vectors and knowledge feature vectors are determined based on the first text description; the target pathological image is a pathological image for target region localization, and the first text description is a natural language description of the features to be identified in the target pathological image; the knowledge feature vector is used to represent knowledge information associated with the content of the target pathological image.

[0007] The semantic feature vector and the knowledge feature vector are fused to obtain fused text features; the fused text features and the visual features are fused across modally to obtain fused multimodal features; and a fused representation is obtained based on the fused multimodal features.

[0008] Based on the fusion representation, the target region in the target pathological image is located by a multilayer perceptron to obtain the position information of the bounding box of the target region, and the target region contains the feature to be identified.

[0009] A second aspect of this application provides a system for visual localization of pathological images, comprising:

[0010] A multimodal feature extraction module is used to extract visual features based on a target pathological image and determine semantic feature vectors and knowledge feature vectors based on a first text description; the target pathological image is a pathological image for target region localization, the first text description is a natural language description of the features to be identified in the target pathological image; the knowledge feature vector is used to represent knowledge information associated with the content of the target pathological image;

[0011] The feature fusion module is used to fuse the semantic feature vector and the knowledge feature vector to obtain fused text features; to perform cross-modal fusion of the fused text features and the visual features to obtain fused multimodal features; and to obtain a fused representation based on the fused multimodal features.

[0012] The localization module is used to locate the target region in the target pathological image based on the fused representation using a multilayer perceptron, and obtain the position information of the bounding box of the target region, wherein the target region contains the feature to be identified.

[0013] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the above-described method for visual localization of pathological images.

[0014] In a fourth aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for visual localization of pathological images.

[0015] The beneficial effects of the pathological image visual localization method, system, device, and storage medium provided in this application embodiment are as follows: This application embodiment introduces natural language description, and by fusing visual features with text semantics and knowledge features, the localization results are more consistent with clinical diagnostic logic, improving localization accuracy. The fusion of semantics and knowledge features in this application embodiment realizes the transformation from shallow text semantics to deep medical knowledge; cross-modal fusion further aligns text knowledge with visual features, accurately capturing the morphology and spatial location relationship of lesions, effectively addressing the problem of locating complex lesions. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a method for visual localization of pathological images provided in an embodiment of this application;

[0018] Figure 2 A schematic diagram of the structure of a pathological knowledge-enhanced visual localization model provided in an embodiment of this application;

[0019] Figure 3 A structural block diagram of a pathological image visual localization system provided in an embodiment of this application;

[0020] Figure 4 This is a schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0021] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0022] Considering that existing medical visual localization technologies are mainly aimed at natural medical images and have not fully taken into account the two major challenges unique to pathological images: (1) pathological images have multi-scale features, and the details at different magnifications vary significantly; (2) pathological expressions contain a large number of implicit semantic professional terms, which are difficult for ordinary models to understand and convert into visual localization cues. Therefore, the main technical objectives of this application are: (1) to propose a visual localization benchmark system suitable for pathological images, which supports accurate localization of specific pathological regions in images through diverse natural language expressions; (2) to propose a visual language model (Pathology Knowledge-enhanced Network, PKNet) that integrates pathological knowledge, which enhances the implicit medical terms in the expression by introducing a large language model to explicitly enhance the model's understanding and localization capabilities under complex expressions. Through this method, this embodiment can overcome the technical bottlenecks of existing pathological visual localization tasks in terms of flexibility, interpretability, and professional knowledge understanding, and significantly improve the practical value of the model in clinical auxiliary diagnosis scenarios.

[0023] In real clinical settings, doctors or pathologists often use subjective descriptions, such as "dense clusters of red blood cells in the lower right corner" or "acinar-like structures with cell infiltration," to indicate suspicious lesions or areas of interest. However, existing technologies (such as pathological visual localization and cell nuclear segmentation) cannot parse these complex terms and accurately point back to the image region, resulting in a lack of effective bridge between doctors' subjective judgments and image analysis systems. On the other hand, existing visual localization studies focus on natural or general medical images, ignoring the three special characteristics of pathological images: (1) multi-scale heterogeneity: the same lesion appears drastically different at different magnifications (such as cell structure and tissue structure); (2) implicit professional expressions: highly compressed professional terms are often embedded in the language description (such as "tumor infiltration" and "solid papillary structure"), which are difficult for the model to understand; (3) regional semantic granularity: unlike whole-image classification tasks, this task focuses on the one-to-one correspondence between the expression and the "local region of the image."

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0025] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a method for visual localization of pathological images according to an embodiment of this application. The method can be executed by an electronic device, and specifically, the method may include S101 to S103.

[0026] S101: Extract visual features based on the target pathological image, and determine semantic feature vector and knowledge feature vector based on the first text description; the target pathological image is the pathological image to be used for target region localization, the first text description is the natural language description text of the features to be identified in the target pathological image; the knowledge feature vector is used to represent the knowledge information associated with the content of the target pathological image.

[0027] In this embodiment, the target pathological image is a standardized pathological image used for localization after preprocessing. Visual features refer to visual information in the target pathological image related to lesion localization. The first text description refers to natural language descriptive text for the target region in the target pathological image. The first text description can come from pathological diagnosis reports, clinical records, or annotation text, such as "densely arranged clusters of red blood cells in the lower right corner" or "aciniform structures with cell infiltration," etc. The role of the first text description is to supplement the pathological semantics that are difficult to directly represent through image visual features by using textual information. The semantic feature vector refers to the semantic encoding of the first text description, including the contextual relationship of diagnostic keywords in the first text description. The knowledge feature vector refers to the structured representation of medical knowledge, including anatomical knowledge, pathological knowledge, and the features and spatial relationships shown in the slide image.

[0028] This embodiment transforms image visual information and text semantic-knowledge information into computable feature vectors, providing input for subsequent fusion localization.

[0029] For example, for visual feature extraction, this implementation can extract multi-scale feature maps (such as 160×120, 48×38 pixel resolution, etc.) from the target pathological image using ResNet50. For each feature point in each feature map, a positional encoding containing coordinate information is generated (e.g., x=0.3, y=0.7 indicates a position slightly to the right of the image center), which is then concatenated with the visual features to form a feature-position pair sequence. This feature-position pair sequence is then input into a 6-layer Transformer for spatial context modeling to extract visual features. Specifically, the Transformer can capture the relationships between different regions in the image through an attention mechanism and a feedforward neural network, ultimately outputting a high-dimensional visual feature sequence representing the image content.

[0030] For semantic feature extraction, this embodiment can segment the first text description into words, label medical entities (such as "red blood cell clusters" and "acinar-like structures"), and identify and locate keywords (such as "bottom right corner"). This embodiment can encode the segmented text to generate a semantic vector containing contextual relationships. This embodiment can also increase the weight of the located keywords through an attention mechanism to generate a weighted semantic feature vector.

[0031] For knowledge feature extraction, this embodiment can design prompts, such as "Generate anatomical location knowledge of this lesion: {pathological text}, requiring the inclusion of tissue layers and adjacent relationships." This prompt and the first text description are input into a medical large language model to obtain a structured knowledge text description, which is then encoded into a knowledge feature vector. For example, the prompt could be set to "Can you explain the visual effect of pathological specific terms on a slide?".

[0032] In this embodiment, the target pathological image is obtained by: acquiring a pathological slide image, determining the image type and multiple pathological tissue regions based on the pathological slide image, and performing image preprocessing on the multiple pathological tissue regions based on the image type to obtain the target pathological image.

[0033] In this embodiment, image preprocessing is performed on multiple pathological tissue regions based on image type to obtain a target pathological image. Specifically, if the image type is a local lesion image, the multiple pathological tissue regions are divided into lesion regions and tissue background regions; edge enhancement is performed on the lesion regions, and noise reduction is performed on the tissue background regions to obtain the target pathological image.

[0034] If the image type is a diffuse lesion image, calculate the probability value of each pathological tissue region belonging to the lesion region; mark the pathological tissue regions with probability values ​​greater than or equal to the probability threshold as lesion regions to obtain the marked target pathological image.

[0035] In this embodiment, the target pathological image is a preprocessed, standardized pathological image used for visual localization. Pathological tissue regions are areas of pathological significance identified through image segmentation techniques, such as lesion areas and tissue background areas. Pathological tissue regions can be divided according to biological characteristics and localization requirements. Localized lesion images and diffuse lesion images are two major image types classified based on the spatial distribution characteristics of lesions, used for differentiated preprocessing strategies and localization logic. Lesions in localized lesion images have clear boundaries and can be located using quadrilateral bounding boxes. Lesions in diffuse lesion images have no clear boundaries and are diffusely distributed, requiring probabilistic region marking for localization. A lesion region may include a lesion core area and an infiltrative edge area (a blurred transition zone); the tissue background area refers to normal tissue or non-diagnostic areas outside the lesion (such as interstitial fibers and adipose tissue).

[0036] In this embodiment, lesion edge enhancement refers to strengthening the boundary contrast between the lesion and the background through algorithms, facilitating subsequent bounding box localization. Tissue background denoising refers to suppressing background noise (such as staining particles and bubble artifacts) to reduce interference with lesion detection. Probability value calculation assigns a lesion probability value of 0-1 to each pixel or pathological tissue region, reflecting the likelihood of a lesion, where 0 represents normal and 1 represents a lesion. The probability threshold and area threshold are preset thresholds.

[0037] In this embodiment, the purpose of pathological image preprocessing is to differentiate and enhance key diagnostic features based on the spatial distribution characteristics of lesions, suppress noise interference, and provide high-quality input for subsequent visual localization. For localized lesions (such as solid tumors): their boundaries are relatively clear, and edge enhancement combined with background denoising is needed to highlight the lesion outline, adapting to the dependence of bounding box localization on shape and position features; for diffuse lesions (such as chronic inflammation): their boundaries are not clear, and regional abnormalities need to be quantified through probabilistic labeling and connected component screening.

[0038] For example, in the preprocessing of images of localized lesions, this embodiment can utilize pre-trained tissue segmentation models such as U-Net or threshold segmentation algorithms to divide the image into lesion regions and tissue background regions. The segmentation results are then cross-validated using morphological features (cell density, nucleocytoplasmic ratio), for example, the nuclear density of the tumor region must be significantly higher than that of the background region (a factor greater than or equal to 2).

[0039] For the lesion area, this embodiment can use an edge-preserving filtering algorithm to suppress image noise while preserving cell boundaries; this embodiment can also enhance the gradient magnitude of pixels at the edge of the area. For example, the edge gradient is calculated using the Sobel operator, and the contrast of the top 20% of pixels with the highest gradient magnitude is stretched, such as expanding the grayscale range [100, 200] to [50, 250] to highlight the boundary between the lesion and the background.

[0040] For the tissue background area, this embodiment can use low-pass filtering to reduce the texture complexity of the image in this area and reduce the interference of structures such as collagen fibers and blood vessels on lesion localization; this embodiment can also use color space normalization to unify the staining intensity of the tissue background area and avoid misjudgment due to uneven staining.

[0041] For example, in the preprocessing of diffuse lesion images, this embodiment can extract features such as cell nuclear morphology, staining intensity, and tissue structure for each pathological tissue region. Based on support vector machine or random forest, the extracted features are mapped to lesion probability values ​​(0-1). For example, regions with nuclear atypia greater than 60% and CD3+ density greater than 40 cells / high-power field are assigned a probability value of 0.8. This embodiment can use the DBSCAN algorithm to cluster pixels with similar probability values ​​and spatial proximity to form continuous probability regions. This embodiment can set a neighborhood radius (e.g., 50 μm) and a minimum sample size (e.g., 10 pixels) to ensure that the core lesion region is completely identified. This embodiment can set a probability threshold (e.g., 0.4) to mark regions with probability values ​​greater than the probability threshold as lesion regions.

[0042] S102: The semantic feature vector and the knowledge feature vector are fused to obtain the fused text feature; the fused text feature and the visual feature are fused across modally to obtain the fused multimodal feature; and the fused representation is obtained based on the fused multimodal feature.

[0043] In this embodiment, fused text features refer to the feature vectors that integrate the semantic information of pathological texts with the knowledge in the medical field, and are generated by fusing the semantic feature vectors and knowledge feature vectors through a knowledge fusion module.

[0044] For example, this embodiment can capture the deep association between semantics and knowledge based on semantic feature vectors and knowledge feature vectors through a two-layer Transformer structure, utilizing the Transformer's multi-head attention mechanism. The final output is a feature vector containing a joint semantic-knowledge representation, which can be used for subsequent cross-modal fusion. This embodiment can solve the problem of understanding the implicit semantics of professional terms in pathological texts, providing text-level semantic enhancement and knowledge guidance for visual localization.

[0045] In this embodiment, the semantic feature vector and the knowledge feature vector are fused to obtain fused text features. Specifically, this may include: generating a semantic calibration weight matrix based on the knowledge feature vector, and calibrating the fuzzy semantic features in the semantic feature vector based on the semantic calibration weight matrix.

[0046] The knowledge feature vector is decomposed into structural knowledge features and relational knowledge features; structural knowledge features include morphological pathological features associated with the first text description; relational knowledge features include spatial positional relationship features between various pathological entities in the first text description.

[0047] The calibrated semantic feature vector, structural knowledge feature, and relational knowledge feature are concatenated to obtain the fused text feature.

[0048] In this embodiment, the semantic calibration weight matrix is ​​an adjustment parameter matrix generated from the knowledge feature vector, used to correct the deviation of the semantic feature vector. Structural knowledge features are morphological features decomposed from the knowledge feature vector, corresponding to the inherent attributes of pathological entities. Structural knowledge features can specifically include morphological parameters (such as mass size, edge clarity, and color intensity) and structural types (such as tissue morphology labels like "aciniform" and "nested"), reflecting the shape of the entity. Relational knowledge features are spatial association features decomposed from the knowledge feature vector, corresponding to the positional relationships between pathological entities. Relational knowledge features can specifically include spatial distance (such as the pixel distance between the mass and the pleura) and orientation labels (such as relative positional identifiers like "above" and "surrounding"), reflecting where and how the entities are located.

[0049] For example, this embodiment performs natural language parsing on the first text description, transforming the text into a quantifiable semantic feature vector through methods such as word segmentation and semantic encoding. For instance, when parsing "lump edge spiculation," the semantic associations of "lump," "edge," and "spiculation" are extracted to form a preliminary semantic representation. This step needs to preserve the original descriptive logic of the text, but it will contain ambiguous information such as "spiculation" not having a clear medical definition (e.g., length, density).

[0050] In this embodiment, the first text description and prompt words (such as "generate pathological knowledge related to lung masses") are input into the medical large language model, and professional knowledge text (such as "spiculated signs indicate tumor infiltration, commonly seen in lung squamous cell carcinoma, and spiculation length greater than 5mm indicates a high risk of malignancy") are output. This embodiment then converts this into a knowledge feature vector. This step can introduce medical professional knowledge to make up for the lack of professionalism in the text description.

[0051] This embodiment can calculate a semantic calibration weight matrix based on professional terms (such as "the length of the burr is greater than 5mm") in the knowledge feature vector. Specifically, this embodiment assigns high weights to semantic terms in the text that match professional knowledge (such as "burr") and low weights to vague expressions (such as "a bit long"). By adjusting the semantic feature vector through the matrix, the association between "burr" and "infiltration" and "malignant risk" is strengthened, while the influence of non-professional expressions is weakened, making the semantic features more consistent with the medical context.

[0052] This embodiment can filter the portion of knowledge features describing the morphology of entities, such as "mass diameter 3cm, spiky edges, uneven density," and extract morphological parameters (size, edge condition, density) to form features that include the entity's own morphology. This embodiment can also filter the portion of knowledge features describing the location between entities, such as "mass located subpleurally, adjacent bronchus compressed," and extract spatial parameters (distance, orientation, interaction) to form features that include the locational relationships between entities.

[0053] This embodiment concatenates the calibrated semantic feature vector (corrected exact semantics), structural knowledge features (morphological details), and relational knowledge features (spatial associations) according to feature dimensions to form fused text features. For example, the fused features contain not only the semantics of "subpleural mass," but also the morphological information of "3cm size, spiky edges," and the spatial relationship of "3mm from the pleura," providing accurate textual guidance for subsequent fusion with image features.

[0054] In this embodiment, the fusion representation refers to a multimodal feature vector that integrates visual features and text features. The fusion representation is the final feature input used for bounding box prediction.

[0055] For example, this embodiment can map visual features and fused text features to a unified dimension through linear projection, introduce learnable [REG] tags to aggregate global semantics, and then model the spatial semantic alignment of vision and text through a six-layer cross-modal Transformer to finally output a fused representation. This embodiment can realize cross-modal association between visual information and text semantics, provide multi-dimensional feature support for bounding box coordinate prediction, and improve localization accuracy.

[0056] S103: Based on fusion representation, the target region in the target pathological image is located by a multilayer perceptron to obtain the position information of the bounding box of the target region, and the target region contains the features to be identified.

[0057] In this embodiment, the fused representation refers to the high-dimensional feature vector output by the cross-modal fusion module, which integrates visual features, semantic features, and knowledge features. The Multilayer Perceptron (MLP) is a neural network composed of fully connected layers, used to map the fused representation to bounding box coordinates. A four-dimensional vector is used to represent the positional information of the target region's bounding box, in the format (x, y, w, h), where (x, y) are the coordinates of the top-left corner of the target region's bounding box, and (w, h) are the width and height of the target region's bounding box. This embodiment uses the MLP to convert abstract fused features into concrete spatial coordinates, essentially establishing a regression mapping from semantic-visual features to spatial location.

[0058] For example, such as Figure 2 As shown, to address the aforementioned issues of multi-scale heterogeneity, implicit professional expressions, and regional semantic granularity, this embodiment proposes a method for visual localization of pathological images by constructing a Pathology Knowledge-enhanced Network (PKNet). Specifically, PKNet can include a visual branch, a text expression branch, a pathology knowledge branch, a knowledge fusion module, and a cross-modal fusion module. The visual branch extracts visual features from the target pathological image; the expression branch extracts semantic feature vectors from the first text description; the knowledge branch generates knowledge feature vectors corresponding to the first text description by calling an auxiliary medical large language model, thus enhancing the knowledge of the first text description; the knowledge fusion module fuses the semantic feature vectors and the knowledge feature vectors; and the cross-modal fusion module performs cross-modal fusion of visual and text features and locates the target region.

[0059] Specifically, the input to the visual branch can be pathological slide images (such as HE-stained tissue slides, containing structures like cells, glands, and stroma). The visual branch extracts low-level visual features (such as cell morphology, tissue texture, and density distribution) through a visual encoder. These low-level visual features are then input into a visual Transformer to model spatial relationships between structures (such as the locational association between tumors and blood vessels, and glandular arrangement patterns), thus enhancing the visual features. Finally, linear projection maps the enhanced visual features to a specified dimension for dimensionality matching with text features.

[0060] The input to the text representation branch can be a natural language description from a pathology report (e.g., "isolated round cell clusters with a micropapillary pattern far from the main tumor mass"). The text representation branch uses a text encoder to decompose the text into word vectors, capture surface semantic relationships, and generate text representation embeddings.

[0061] The input for the pathology knowledge branch can be specialized knowledge in the pathology field, such as "cell clusters: appearing as a group of tightly packed cells in a slide, with boundaries between cells that may be clear or blurred. Micropapillary pattern: visually appearing as small papillary structures lacking a fibrovascular core, with cells arranged in clusters, often appearing to float or float. Major tumor mass: refers to the area far from the primary tumor body, showing scattered tumor cells or cell clusters." The specialized knowledge in the pathology field used as input for this branch is specifically generated by a medical large language model. After inputting the first text description and the prompt "Can you explain the visual effect of pathology-specific terms on a slide?" into the medical large language model, the specialized knowledge in the pathology field used as input is obtained. Furthermore, the pathology knowledge branch can transform the input specialized knowledge in the pathology field into pathology knowledge embeddings through a text encoder. Finally, the model concatenates and fuses the text expression embeddings with the pathology knowledge embeddings to obtain a text embedding, supplementing the text with medical expertise.

[0062] The core components of the knowledge fusion module can include Multi-Head Attention, Norm (layer normalization), and MLP (Multilayer Perceptron). The output of the text representation branch is first normalized by Norm to stabilize the mean and variance of the features, avoiding distribution bias during subsequent attention calculations. Then, multi-head attention guides the weight adjustment of text features by allowing pathological knowledge features to strengthen knowledge-matching terms in the text and weaken vague / non-professional expressions. For example, pathological knowledge guides semantic correction of the text, strengthening the medical definition of "small nipple-like structures" and weakening vague expressions in the text. The corrected and fused text embedding is then normalized by Norm to stabilize its distribution, and then enhanced by MLP to output the fused text-knowledge features. Finally, linear projection maps the fused text-knowledge features to a dimension consistent with the visual features.

[0063] The cross-modal fusion module can take visual features and fused text-knowledge features as input. It models cross-modal relationships between images and text using a visual-language Transformer (e.g., aligning "microscopic papillary structures" in an image with text descriptions and knowledge definitions). Then, it uses the [BEIG] token as a localization anchor to help the model focus on the target region (e.g., indicating the search range for "microscopic papillary structures"). Finally, the regression layer of the prediction head outputs the bounding box coordinates (x, y, w, h) and confidence level b, achieving spatial localization of the pathological target.

[0064] This embodiment employs a combined loss function to perform end-to-end training of the pathology knowledge-enhanced visual localization model. The combined loss function is:

[0065]

[0066] in, Here, P represents the bounding box predicted by the model, and GT represents the ground truth bounding box. This represents the L1 loss for bounding box regression. This represents the IoU loss between the bounding box and the ground truth box. , These are preset weighting coefficients. Responsible for ensuring the accuracy of the bounding box position. This system is responsible for monitoring the accuracy of region coverage and can set different parameters based on different downstream diseases. This design ensures that the model balances bounding box position accuracy and region coverage accuracy, improving the accuracy of the final localization results.

[0067] This embodiment not only proposes new tasks and architectures, but also establishes a structured modeling path between pathological image language understanding and image region localization for the first time. The underlying technical principles include: (1) Multimodal semantic alignment theory: language and image embedding are aligned to a unified space to support region localization; (2) Knowledge explicit modeling theory: a large language model is introduced to enhance the knowledge explicit semantics of medical terms, realizing "visual translation of terms"; (3) Cross-scale region feature fusion: multi-resolution representations are learned through visual branches to adapt to the diversity of pathological image scales. In summary, this embodiment can solve the problem that existing pathological image analysis cannot flexibly respond to subjective expressions, and can be widely used in scenarios such as pathological auxiliary diagnosis, human-computer interactive pathological annotation, and intelligent question-answering systems.

[0068] For example, the preparation of training data may include: (1) Image input: multi-magnification slice images (20× and 40×) of size 1024×1024, 3-channel RGB. (2) Text input: descriptive expression of the target region in the image, such as: "Lower right corner region, loosely arranged cell nuclei with mild infiltration". (3) Labeled data: bounding boxes representing the corresponding regions, in the format (x, y, w, h).

[0069] The model modules are configured as follows: (1) Visual branch: The input image is processed by ResNet50 to extract two-dimensional feature maps, and then by 6 layers of Transformer to perform spatial context modeling and output visual features. (2) Expression branch: The input text expression is encoded by BERT to generate expression features. (3) Knowledge branch: LLM is called to interpret the expression statement using professional terms to generate knowledge text, and then BERT is used to extract knowledge features. (4) Knowledge fusion module: The input expression features F_e and knowledge features F_k are fused by two layers of Transformer to output language features F_l. (5) Cross-modal fusion module: The expression features and knowledge features are mapped to a unified dimension, modeled by 6 layers of cross-modal Transformer, and the position representation of the localization intention is added at the beginning and end; finally, the prediction head outputs the coordinates of the prediction box.

[0070] The loss function settings include: using a weighted combined loss function for regression optimization, with the specific loss function settings as follows:

[0071]

[0072] The specific training process may include: (1) loading pre-trained model parameters (ResNet and BERT); (2) using the AdamW optimizer, with an initial learning rate of 1e-5 and a training cycle of 90 rounds; (3) each batch of input includes an image, a segment of expression, and a corresponding region box; (4) calculating L1 and IoU losses between the model output and the GT box, and performing backpropagation optimization; (5) using the validation set for model selection, with accuracy and mean intersection-union ratio as evaluation metrics.

[0073] The usage can include: (1) The user inputs a pathological image (single image) and natural language expression, such as: "There are dense glandular structures in the upper left corner, accompanied by cells with abundant cytoplasm". (2) The model extracts semantic features through expression and knowledge branches and fuses them with image features. (3) The model outputs a predicted bounding box (x, y, w, h) and marks the target area in the image. (4) It can be combined with downstream modules for automatic analysis (such as regional classification, risk prediction, etc.).

[0074] As can be seen from the above, this embodiment introduces natural language description, and by fusing visual features with textual semantics and knowledge features, it overcomes the limitations of traditional methods that rely solely on visual information. This makes the localization results more consistent with clinical diagnostic logic and improves localization accuracy. The fusion of semantic and knowledge features in this embodiment realizes the transformation from shallow textual semantics to deep medical knowledge; cross-modal fusion further aligns textual knowledge with visual features, accurately capturing the morphology and spatial location relationship of lesions, and effectively addressing the problem of locating complex lesions.

[0075] In one embodiment of this application, visual features are extracted based on a target pathological image, including: extracting features from the target pathological image using a convolutional neural network to obtain a first feature map; flattening the first feature map to obtain a one-dimensional feature sequence; and encoding the one-dimensional feature sequence to obtain visual features, which can be used to represent the image content corresponding to the target pathological image.

[0076] In this embodiment, a convolutional neural network (CNN) is used as the foundational network for extracting visual features from images; ResNet-50 can be employed in this embodiment. The first feature map is a low-resolution two-dimensional feature map generated by the CNN, preserving mid-level visual information such as image edges and textures. The one-dimensional feature sequence is a sequence of one-dimensional vectors obtained by flattening the two-dimensional feature map, used for subsequent global relational modeling. This embodiment extracts local image features through a CNN, and then captures the global context through flattening and sequence modeling, adapting to the multi-scale, high-detail characteristics of pathological images.

[0077] For example, this embodiment can standardize the target pathological image, including size unification and color normalization to eliminate the influence of uneven staining and enhance key visual features such as cell nuclei and tissue structures. This embodiment can use ResNet-50 as the base network. The target pathological image is processed through multiple convolutional layers and residual blocks to progressively extract low-level edge textures (such as cell nucleus contours) and high-level semantic features (such as glandular structures), ultimately generating a first feature map that preserves the morphological details of the lesion. This embodiment can flatten the two-dimensional feature map into a one-dimensional feature sequence in spatial order. This embodiment can input the one-dimensional feature sequence into a six-layer Transformer encoder. Each layer uses a multi-head attention mechanism and a feedforward neural network to capture long-distance dependencies (such as the association between the lesion core and the infiltrative edge), outputting a high-dimensional visual feature sequence for subsequent cross-modal fusion with textual features, ensuring global contextual consistency of visual information.

[0078] This embodiment extracts local features through a convolutional neural network, then flattens them into a one-dimensional sequence, and finally captures the global context through sequence modeling. This effectively adapts to the multi-scale and high-detail characteristics of pathological images. This embodiment can preserve mid-level information such as image edges and textures, and strengthens the association between lesions and surrounding tissues through global relationship modeling. It provides high-dimensional visual features containing spatial location for cross-modal fusion, improving the accuracy of regional localization and semantic understanding capabilities of pathological images.

[0079] In one embodiment of this application, determining a semantic feature vector based on a first text description includes: segmenting the first text description into words; converting each word after segmentation into a one-hot representation; mapping the one-hot representation corresponding to each word to a language tag to obtain a language tag corresponding to each word; performing deep semantic feature extraction on the language tags corresponding to each word to obtain a semantic feature vector; the semantic feature vector is a high-dimensional vector sequence representing the semantics of the first text description.

[0080] The process of determining the knowledge feature vector based on the first text description includes: generating the knowledge feature vector through a medical big language model based on the first text description and target prompt words; the target prompt words are used to instruct the medical big language model to generate knowledge text associated with the pathological image content based on the first text description.

[0081] In this embodiment, one-hot representation refers to converting the segmented text into binary vectors, with each word corresponding to a unique dimension. The embedding layer is used to map high-dimensional one-hot vectors to low-dimensional dense vectors (word embeddings) to capture semantic relationships. Target prompt words are instructions used to guide the medical big language model in generating knowledge, such as "parse the pathological morphological features of this description: {first text description}". The prompt word structure (e.g., hierarchical question type) and domain constraints (e.g., limiting the generation of anatomy / pathology knowledge) can be set according to actual needs.

[0082] For example, suppose the first text description is “isolated circular cell clusters in a microtext encoder papillary pattern, far from the main tumor mass.” In this embodiment, the first text description can be segmented into words and converted into one-hot representations (with vector dimensions consistent with the vocabulary). Then, it can be mapped to language tags through an embedding layer (e.g., 768-dimensional). The language tags are then input into a 12-layer BERT encoder, and deep semantics are extracted through a self-attention mechanism to generate a high-dimensional semantic feature vector sequence containing contextual relationships.

[0083] This embodiment can design target prompts, assuming the prompt is "Can you explain the visual effect of pathological specific terms on a slide?", and input this target prompt along with the first text description into a medical big language model to generate knowledge text: "Cells: Displayed as a group of tightly packed cells in a slide, the boundaries between cells may be clear or blurred. Micropapillary pattern: Visually small papillary structures, lacking a fibrovascular core, cells are arranged in clusters, often appearing to float or float. Major tumor mass: Refers to the area far from the primary tumor body, showing scattered tumor cells or cell clusters." The knowledge text is then encoded using BERT, and global average pooling is used to obtain a knowledge feature vector, preserving the anatomical and pathological knowledge dimensions, providing professional knowledge support for cross-modal fusion.

[0084] This embodiment generates semantic feature vectors through one-hot representation, embedding layers, and BERT encoding. It then combines target prompts with a medical big language model to generate knowledge feature vectors, which can capture deep semantics and professional knowledge of text, solve the problem of understanding pathological terminology, provide semantic and knowledge support for cross-modal fusion, and improve the accuracy and interpretability of pathological image region localization.

[0085] In one embodiment of this application, cross-modal fusion of fused text features and visual features is performed to obtain fused multimodal features, including: mapping the fused text features and visual features to a unified feature dimension through linear projection to obtain target text features and target visual features respectively; concatenating the target text features and target visual features, and inserting a first marker into the concatenated multimodal feature sequence to obtain fused multimodal features; the first marker is used to focus global semantic information.

[0086] In this embodiment, the first label refers to a learnable special vector used to aggregate global semantic information of multimodal features, such as [REG] shown above. This embodiment achieves cross-modal semantic alignment between textual semantics and visual information through dimension alignment, feature concatenation, and global labeling, resolving the mapping ambiguity problem between textual descriptions and visual regions in pathological images. Since textual features and visual features belong to different semantic spaces—textual features containing pathological terminology and visual features containing image texture structure—semantic space alignment needs to be achieved through linear projection.

[0087] For example, this embodiment can learn the mapping from text semantics to a shared space through a fully connected layer, and then normalize the feature distribution (mean 0, variance 1) using Norm to enhance the semantic representation of pathological terms; similarly, visual features are processed to align texture features with text semantics in the same dimension. This embodiment concatenates the projected target text features (e.g., 1024-dimensional) and target visual features (e.g., 1024-dimensional) into a 2048-dimensional sequence, inserting a learnable [REG] marker (e.g., 1024-dimensional) at the beginning of the feature sequence to form a 3072-dimensional fusion sequence. This marker captures global semantic associations through a self-attention mechanism, for example, integrating the text location of "right breast upper outer quadrant" with the structural features of the corresponding image region. This embodiment can input the fusion sequence into a multi-layer Transformer encoder, using a multi-head attention mechanism to enhance the spatial semantic alignment of text and visual features, and finally output fused multimodal features, providing a cross-modal joint representation containing global semantics for bounding box prediction.

[0088] This embodiment achieves modal semantic space alignment through linear projection, resolving ambiguities in text and visual feature mapping. Through feature splicing and insertion of learnable markers, and by utilizing Transformer to enhance cross-modal semantic alignment, it can accurately integrate the semantics of pathological terminology with the texture structure of images, providing global semantic support for bounding box prediction and improving the accuracy and robustness of pathological image region localization.

[0089] Corresponding to the above embodiment, a method for visual localization of pathological images, Figure 3 This is a structural block diagram of a system for visual localization of pathological images, provided in one embodiment of this application. For ease of explanation, only the parts relevant to the embodiment of this application are shown. References Figure 3 The system 20 for visual localization of pathological images includes: a multimodal feature extraction module 21, a feature fusion module 22, and a localization module 23.

[0090] The multimodal feature extraction module 21 is used to extract visual features based on the target pathological image and determine semantic feature vectors and knowledge feature vectors based on the first text description. The target pathological image is a pathological image for target region localization, and the first text description is a natural language description of the features to be identified in the target pathological image. The knowledge feature vector is used to represent knowledge information associated with the content of the target pathological image.

[0091] The feature fusion module 22 is used to fuse semantic feature vectors and knowledge feature vectors to obtain fused text features; to perform cross-modal fusion of fused text features and visual features to obtain fused multimodal features; and to obtain fused representation based on fused multimodal features.

[0092] The localization module 23 is used to locate the target region in the target pathological image based on the fusion representation and through a multilayer perceptron, to obtain the position information of the bounding box of the target region, and the target region contains the features to be identified.

[0093] In one embodiment of this application, the multimodal feature extraction module 21 is specifically used to extract features based on the target pathological image through a convolutional neural network to obtain a first feature map; flatten the first feature map to obtain a one-dimensional feature sequence; encode the one-dimensional feature sequence to obtain visual features, which can be used to represent the image content corresponding to the target pathological image.

[0094] In one embodiment of this application, the multimodal feature extraction module 21 is further configured to perform word segmentation on the first text description; convert each word after word segmentation into a one-hot representation; map the one-hot representation corresponding to each word to a language tag to obtain the language tag corresponding to each word; perform deep semantic feature extraction on the language tags corresponding to each word to obtain a semantic feature vector; the semantic feature vector is a high-dimensional vector sequence representing the semantics of the first text description.

[0095] The process of determining the knowledge feature vector based on the first text description includes: generating the knowledge feature vector through a medical big language model based on the first text description and target prompt words; the target prompt words are used to instruct the medical big language model to generate knowledge text associated with the pathological image content based on the first text description.

[0096] In one embodiment of this application, the feature fusion module 22 is specifically used to generate a semantic calibration weight matrix based on the knowledge feature vector, calibrate the fuzzy semantic features in the semantic feature vector based on the semantic calibration weight matrix, decompose the knowledge feature vector into structural knowledge features and relational knowledge features; the structural knowledge features include morphological pathological features associated with the first text description; the relational knowledge features include spatial positional relationship features between various pathological entities in the first text description; and concatenate the calibrated semantic feature vector, structural knowledge features, and relational knowledge features to obtain fused text features.

[0097] In one embodiment of this application, the feature fusion module 22 is further configured to map the fused text features and visual features to a unified feature dimension through linear projection to obtain target text features and target visual features respectively; to concatenate the target text features and target visual features, and to insert a first marker into the concatenated multimodal feature sequence to obtain fused multimodal features; the first marker is used to focus global semantic information.

[0098] In one embodiment of this application, the target pathological image is obtained by: acquiring a pathological slide image; determining the image type and multiple pathological tissue regions based on the pathological slide image; and performing image preprocessing on the multiple pathological tissue regions based on the image type to obtain the target pathological image.

[0099] In one embodiment of this application, image preprocessing is performed on multiple pathological tissue regions based on image type to obtain a target pathological image, including: if the image type is a local lesion image, then the multiple pathological tissue regions are divided into lesion regions and tissue background regions; edge enhancement is performed on the lesion regions, and noise reduction processing is performed on the tissue background regions to obtain the target pathological image;

[0100] If the image type is a diffuse lesion image, calculate the probability value of each pathological tissue region belonging to the lesion region; mark the pathological tissue regions with probability values ​​greater than or equal to the probability threshold as lesion regions to obtain the marked target pathological image.

[0101] See Figure 4 , Figure 4 This is a schematic block diagram of an electronic device provided according to an embodiment of this application. Figure 4 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of the modules in the aforementioned system embodiments, for example... Figure 3 The functions of the multimodal feature extraction module 21, feature fusion module 22, and localization module 23 are shown.

[0102] It should be understood that, in the embodiments of this application, the processor 301 may be a central processing unit (CPU), but it may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0103] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0104] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store information about pathological images.

[0105] In specific implementations, the processor 301, input device 302, and output device 303 described in the embodiments of this application can execute the implementation methods described in the embodiments of the pathological image visual positioning method provided in this application, or they can execute the implementation methods of the electronic device 300 described in the embodiments of this application, which will not be repeated here.

[0106] In another embodiment of this application, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. Alternatively, the computer program can instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0107] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD) card, flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0108] Those skilled in the art will recognize that the modules / units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0109] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0110] In the embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules, units, or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or modules / units, or it may be an electrical, mechanical, or other form of connection.

[0111] The modules / units described as separate components may or may not be physically separate. Similarly, the components shown as modules / units may or may not be physical modules / units; they may be located in one place or distributed across multiple network modules / units. Some or all of the modules / units can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0112] Furthermore, the functional modules / units in the various embodiments of this application can be integrated into one processing module / unit, or each module / unit can exist physically separately, or two or more modules / units can be integrated into one module / unit. The integrated modules / units described above can be implemented in hardware or in the form of software functional modules / units.

[0113] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method of pathological image visual positioning, characterized in that, The method comprises the following steps: extracting visual features based on a target pathological image, determining a semantic feature vector and a knowledge feature vector based on a first text description; the target pathological image is a pathological image to be subjected to target region positioning, and the first text description is a natural language description text of a feature to be recognized in the target pathological image; the knowledge feature vector is used to represent knowledge information associated with the content of the target pathological image; generating a semantic calibration weight matrix based on the knowledge feature vector, and calibrating ambiguous semantic features in the semantic feature vector based on the semantic calibration weight matrix; decomposing the knowledge feature vector into a structural knowledge feature and a relational knowledge feature; the structural knowledge feature comprises a morphological pathological feature associated with the first text description; the relational knowledge feature comprises a spatial position relationship feature between pathological entities in the first text description; concatenating the calibrated semantic feature vector, the structural knowledge feature and the relational knowledge feature to obtain a fusion text feature; cross-modal fusion of the fusion text feature and the visual feature to obtain a fusion multi-modal feature, and obtaining a fusion representation based on the fusion multi-modal feature; based on the fusion representation, positioning a target region in the target pathological image through a multi-layer perception machine to obtain position information of a bounding box of the target region, wherein the target region contains the feature to be recognized, and the target region contains the feature to be recognized; the target pathological image is obtained by the following method: acquiring a pathological section image, determining an image type and a plurality of pathological tissue regions based on the pathological section image; if the image type is a local lesion image, the plurality of pathological tissue regions are divided into a lesion region and a tissue background region; edge enhancement is performed on the lesion region, and denoising processing is performed on the tissue background region to obtain the target pathological image; if the image type is a diffuse lesion image, a probability value of each pathological tissue region belonging to a lesion region is calculated; pathological tissue regions with a probability value greater than or equal to a probability threshold value are marked as lesion regions to obtain a marked target pathological image.

2. A method of visual positioning of pathological images as claimed in claim 1, characterized in that, The method comprises the following steps: based on the target pathological image, feature extraction is performed through a convolutional neural network to obtain a first feature map; the first feature map is flattened to obtain a one-dimensional feature sequence; the one-dimensional feature sequence is encoded to obtain the visual feature, which can be used to represent the image content corresponding to the target pathological image.

3. A method of visual positioning of pathological images as claimed in claim 1, characterized in that, The method comprises the following steps: performing word segmentation processing on the first text description; converting each word after word segmentation processing into a one-hot representation; mapping each one-hot representation corresponding to a word into a language token to obtain a language token corresponding to each word; performing deep semantic feature extraction on the language token corresponding to each word to obtain a semantic feature vector; the semantic feature vector is a high-dimensional vector sequence representing the semantics of the first text description; wherein, determining the knowledge feature vector based on the first text description comprises: generating the knowledge feature vector by a medical large language model based on the first text description and a target prompt word; the target prompt word is used to instruct the medical large language model to generate knowledge text associated with the content of the pathological image based on the first text description.

4. A method of visual positioning of pathological images as claimed in claim 1, characterized in that, The cross-modal fusion of the fusion text feature and the visual feature to obtain a fusion multi-modal feature comprises: mapping the fusion text feature and the visual feature to a unified feature dimension by linear projection to obtain a target text feature and a target visual feature, respectively; splicing the target text feature and the target visual feature, inserting a first marker in the spliced multi-modal feature sequence to obtain the fusion multi-modal feature; the first marker is used to focus on global semantic information.

5. A system for visual positioning of pathological images, characterized by comprises: a multi-modal feature extraction module configured to extract a visual feature based on a target pathological image, and determine a semantic feature vector and a knowledge feature vector based on a first text description; the target pathological image is a pathological image to be subjected to target region positioning, and the first text description is a natural language description text of a feature to be recognized in the target pathological image; the knowledge feature vector is used to represent knowledge information associated with the content of the target pathological image; a feature fusion module configured to generate a semantic calibration weight matrix based on the knowledge feature vector, and calibrate ambiguous semantic features in the semantic feature vector based on the semantic calibration weight matrix; decomposing the knowledge feature vector into a structural knowledge feature and a relational knowledge feature; the structural knowledge feature comprises a morphological pathological feature associated with the first text description; the relational knowledge feature comprises a spatial position relationship feature associated with each pathological entity in the first text description; splicing the calibrated semantic feature vector, the structural knowledge feature, and the relational knowledge feature to obtain a fusion text feature; cross-modal fusion of the fusion text feature and the visual feature to obtain a fusion multi-modal feature, and obtaining a fusion representation based on the fusion multi-modal feature; a positioning module configured to position a target region in a target pathological image based on the fusion representation by a multi-layer perception machine to obtain position information of a bounding box of the target region, the target region containing the feature to be recognized; the target pathological image is obtained by: acquiring a pathological section image, determining an image type and a plurality of pathological tissue regions based on the pathological section image; if the image type is a local lesion image, dividing the plurality of pathological tissue regions into a lesion region and a tissue background region; edge enhancement is performed on the lesion region, and denoising processing is performed on the tissue background region to obtain the target pathological image; if the image type is a diffuse lesion image, a probability value that each pathological tissue region belongs to a lesion region is calculated; pathological tissue regions with a probability value greater than or equal to a probability threshold value are marked as lesion regions to obtain a marked target pathological image.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, wherein the computer program comprises the following steps of: receiving a request for a resource from a client; determining whether the client is authorized to access the resource; and if the client is authorized to access the resource, providing the resource to the client. The computer program, which is executed by a processor, implements the steps of the method according to any one of claims 1 to 4.

Images