Skin cancer multi-modal case retrieval method, device and equipment based on global-local alignment, medium and product

Abstract

The application discloses a skin cancer multi-modal case retrieval method and device based on global-local alignment, equipment, medium and product, relates to the field of multi-modal information fusion, and comprises the following steps: constructing a case database containing dermoscope images and diagnosis labels and a composite query; extracting multi-level visual features of the query image by using a Swin Transformer; encoding the query text by using BERT, and interacting and fusing the multi-level visual features and the text features by using a cross-modal Transformer to obtain a query feature of the text-image fusion; respectively calculating global similarity and local similarity of the query feature of the text-image fusion and target image features; weighting and fusing the global and local similarity to obtain a final similarity score, and performing sorting retrieval and label consistency verification on the database based on the score. The application breaks through the limitation of traditional single-modal retrieval, realizes fine-grained semantic alignment, and significantly improves the accuracy of skin cancer case retrieval and the value of clinical decision support.

Description

Technical Field

[0001] This application relates to the field of multimodal information fusion, and in particular to a method, apparatus, device, medium and product for retrieving multimodal skin cancer cases based on global-local alignment. Background Technology

[0002] Early screening and accurate diagnosis of skin cancer are crucial for timely treatment planning, reducing morbidity, and improving cure rates. With the rapid development of deep learning technology, AI-based classification systems have demonstrated near-dermatologist-level performance in various skin lesion recognition scenarios. However, directly translating end-to-end classification models into daily clinical workflows still faces core challenges. Compared to simple classification tasks, case-based retrieval provides clinicians with a more intuitive and interpretable form of decision support by comparing similar historical cases, aiding in comparative evaluation, clinical teaching, and quality control. In actual clinical practice, doctors typically combine reference lesion images with concise clinical text descriptions (such as dermoscopy features or standards based on examination checklists) to construct retrieval queries. Therefore, building an effective vision-language hybrid query system to fully utilize multimodal information is particularly critical. In recent years, the Transformer architecture and its extensions in the vision domain have laid a solid technical foundation for multimodal synthetic image retrieval. However, existing technologies, when performing case matching, often only focus on global semantic features or perform superficial feature splicing, making it difficult to accurately capture highly diagnostic local clues in skin lesions (such as stripes, irregular pigment distribution, and degenerative patterns). Furthermore, existing methods lack a joint alignment mechanism that considers both global semantic consistency and local discriminative evidence, resulting in insufficient retrieval accuracy and unstable ranking quality under complex clinical queries, failing to meet the actual clinical needs for high-precision, interpretable case retrieval. Summary of the Invention

[0003] The purpose of this application is to provide a method, device, equipment, medium, and product for retrieving multimodal cases of skin cancer based on global-local alignment, which can solve the problems of existing multimodal medical image retrieval methods that are difficult to balance global semantics and local discriminative features, as well as insufficient retrieval accuracy and clinical interpretability.

[0004] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a multimodal skin cancer case retrieval method based on global-local alignment, including: An image database is constructed and a retrieval query is defined. The image database contains multiple case samples, each case sample including a lesion image and its corresponding diagnostic label. The retrieval query includes: a reference lesion image and a clinical text description associated with the reference lesion image. Hierarchical visual encoding is performed on the reference lesion image to extract multi-level visual features of the reference lesion image; Text encoding is performed on the clinical text description to obtain text semantic features; The multi-level visual features and the text semantic features are input into a cross-modal Transformer for feature interaction and fusion, and the resulting image-text fusion query features are output. The target case image is obtained from the image database and its features are extracted to obtain the target image features; Based on the image-text fusion query features and the target image features, local similarity and global similarity are calculated respectively; The local similarity and the global similarity are weighted and fused to obtain the final similarity score between the search query and each target case sample; Based on the final similarity score, all case samples in the image database are sorted in descending order, and a predetermined number of cases with the highest ranking are recalled as search results; The accuracy of the retrieval is verified by comparing the consistency of the retrieval results with the diagnostic tags of the retrieval query.

[0005] Optionally, performing hierarchical visual coding on the reference lesion image and extracting multi-level visual features of the reference lesion image specifically includes: A multi-level visual backbone network based on the Swin Transformer architecture is used to extract features from the reference lesion image. The multi-level visual features include low-level features, mid-level features and high-level features, which are extracted from the 10th, 11th and 12th layers of the visual backbone network, respectively. The dimension of each level feature is 768.

[0006] Optionally, text encoding of the clinical text description to obtain text semantic features specifically includes: The clinical text description was encoded using a pre-trained BERT model to obtain a word sequence and corresponding text semantic features; The multi-level visual features and the text semantic features are input into a cross-modal Transformer for feature interaction and fusion, and the output image-text fusion query features specifically include: Construct a cross-modal Transformer containing three cross-attention blocks; The multi-level visual features are used as query vectors Q, and the text semantic features are used as key vectors K and value vectors V. The interaction of low, medium and high-level image and text features is realized through the cross attention block. The output is an image and text fusion query feature that is aligned with the dimensions of the multi-level visual features. The image and text fusion query feature includes low-level fusion features, medium-level fusion features and high-level fusion features.

[0007] Optionally, the process of calculating local similarity specifically includes: For each level of the image-text fusion query features and the target image features, generate k spatial attention masks; The spatial attention mask is subjected to a Hadamard product with the features of the corresponding level, and a region descriptor is generated by mean pooling. The local feature similarity is calculated by substituting the region descriptor of the image-text fusion query feature and the region descriptor of the target image feature into the cosine similarity formula, and then aggregating the local feature similarities of k regions to obtain the local similarity.

[0008] Optionally, the process of calculating the global similarity is as follows: Global mean pooling is performed on the image-text fusion query features and the target image features respectively to obtain global feature representations of the query and the target; The global similarity is obtained by substituting the global feature representation of the query and the global feature representation of the target into the cosine similarity formula.

[0009] Optionally, the calculation process for the final similarity score is as follows: ; in, This indicates the final similarity score; Coefficients representing the trade-off between local and global signals; Indicates local similarity; Indicates global similarity.

[0010] Secondly, this application provides a skin cancer multimodal case retrieval device based on global-local alignment, comprising: The database construction module is used to construct an image database and define retrieval queries. The image database contains multiple case samples, each case sample including a lesion image and its corresponding diagnostic label. The retrieval query includes: a reference lesion image and a clinical text description associated with the reference lesion image. A visual encoding module is used to perform hierarchical visual encoding on the reference lesion image and extract multi-level visual features of the reference lesion image. The text encoding module is used to encode the clinical text description to obtain text semantic features; The feature interaction fusion module is used to input the multi-level visual features and the text semantic features into the cross-modal Transformer for feature interaction fusion, and output the image-text fusion query features. The feature extraction module is used to obtain target case images from the image database and perform feature extraction to obtain target image features; The local similarity and global similarity calculation module is used to calculate local similarity and global similarity based on the image-text fusion query features and the target image features, respectively. The final similarity score calculation module is used to perform weighted fusion of the local similarity and the global similarity to obtain the final similarity score between the search query and each target case sample; The descending sorting module is used to sort all case samples in the image database in descending order according to the final similarity score, and recall a preset number of cases with the highest ranking as search results; The retrieval verification module is used to verify the accuracy of the retrieval by comparing the retrieval results with the diagnostic tags of the retrieval query.

[0011] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the skin cancer multimodal case retrieval method based on global-local alignment as described above.

[0012] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the skin cancer multimodal case retrieval method based on global-local alignment as described above.

[0013] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the skin cancer multimodal case retrieval method based on global-local alignment as described above.

[0014] According to the specific embodiments provided in this application, this application has the following technical effects: This application provides a method, device, equipment, medium, and product for retrieving multimodal skin cancer cases based on global-local alignment. It overcomes the limitations of traditional single-modal or shallow fusion methods in representing complex clinical queries. It proposes a joint alignment mechanism that balances global semantic consistency with local discriminative evidence. Through dynamic aggregation of clinically discriminative image regions via local alignment and high-level semantic supervision using global alignment, it achieves fine-grained, multi-scale feature measurement, significantly improving the model's precision and comprehensiveness in representing high-dimensional case images. A cross-modal Transformer-based image-text feature interaction fusion strategy is introduced, enabling strict dimensional and semantic alignment between query features and target image features in the visual feature space, allowing for more accurate quantification of feature distances between cases. Combined with an adaptive weighted similarity calculation mechanism, it highlights clinically significant local evidence in similarity assessment while maintaining the integrity of the global semantic structure, achieving high-precision matching without relying on explicit lesion level annotations.

[0015] Experimental results demonstrate that this invention achieves sustained and significant performance improvements over existing state-of-the-art methods on publicly available skin cancer datasets. It achieves 79.3% accuracy@1, with an overall average of 83.0% and a mAP of 81.7%, proving its superiority in early retrieval accuracy and overall ranking quality. This method provides a reliable new paradigm for intelligent medical image retrieval, possessing good generalization ability and clinical translational value. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a multimodal skin cancer case retrieval method based on global-local alignment, provided as an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] In one exemplary embodiment, such as Figure 1 As shown, a multimodal skin cancer case retrieval method based on global-local alignment is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method includes the following steps: Step 101: Construct an image database and define retrieval queries. The image database contains multiple case samples, each case sample including a lesion image and its corresponding diagnostic label. The retrieval query includes: a reference lesion image and a clinical text description associated with the reference lesion image.

[0021] Let the image database be ,in Represents an image of a lesion, while This refers to its diagnostic label. The search query is defined as... It includes a reference lesion image, also known as a query image. and the query tag Related text description (For example, based on checklist standards). In the data setup of this application, each case sample in the image database consists of two modalities: a dermoscopic image for visually displaying the lesion, and a corresponding structured clinical information text.

[0022] Define positive samples at the label level for: ; Given a query Learn a similarity function This paper constructs a cosine similarity function based on dermoscopic images (i.e., images representing actual lesions) and different levels of text features. The model is trained using the diagnostic labels of cases as supervision to fit discriminative representations of different types of features. During training, triplet loss is used as the optimized loss function to distinguish case features with different labels. This function can analyze all images in the database. Sort the cases and return the K most relevant cases associated with the same diagnostic label.

[0023] The cosine similarity function is expressed as follows: .

[0024] Step 102: Perform hierarchical visual coding on the reference lesion image to extract multi-level visual features of the reference lesion image.

[0025] A multi-level visual backbone based on the Swin Transformer architecture was used to extract multi-level visual features. A tiny model was used, with a patch size of 4×4, a window size of 7×7, and an embedding dimension of 96. There were four Transformer blocks with layers of 2, 2, 6, and 2, and multi-head attention heads of 3, 6, 12, and 24, respectively. Images were sourced from the Derm7pt skin cancer dataset, resulting in 888 RGB images of skin lesions covering melanoma, nevi, and keratoid lesions, with a size of 768×512 pixels. Black boundary artifacts were removed during preprocessing before being input into the model. Multi-level visual features were extracted from the reference lesion images. like: ; in, , representing low-level, mid-level, and high-level feature mappings, respectively. These three levels of features (distinguished by L, M, and H) are extracted from layers 10, 11, and 12 of the model, each with a dimension of 768. For the query image... and target database images Perform the following calculations: ; in, This indicates the image to be queried. After being extracted by the Swing Transformer, at the... Feature representation corresponding to the level; This indicates that the target case images in the database, after being extracted via the same network described above, are at the [number]th [time / period]. Feature representation corresponding to the level; Indicates the height of image features in the spatial dimension. It represents the width of the image feature in the spatial dimension. The channel depth or feature dimension (Depth) represents the image features; the subscripts L, M, and H correspond to three different levels of image features: Low, Medium, and High, respectively. That is, the subscripts L, M, and H are used to distinguish between low, medium, and high levels of image features. Indicates the high level of image features, The width of the image features. Indicates feature dimension, subscript and Used to distinguish the source attributes of case images. Among them, (Query) represents the current case to be queried; (Target) represents the target (candidate) case in the database.

[0026] Step 103: Perform text encoding on the clinical text description to obtain text semantic features.

[0027] Step 104: Input the multi-level visual features and the text semantic features into the cross-modal Transformer for feature interaction fusion, and output the image-text fusion query features.

[0028] Based on the meta-meta-data of the dataset, which includes information such as gender for each case and a 7-point checklist of lesion details, text paragraphs are obtained through structured text description transformation. The pre-trained BERT is used as the language encoder to obtain word embeddings. The number of layers is 12, and the hidden dimension is 768. The parameters of the BERT-base-uncased pre-trained model are frozen during use. ; in, It is the number of lexical units. This refers to the embedding dimension of the lexical units. To construct a combined query that reflects both the reference appearance and the specified textual attributes, a cross-modal Transformer is used to inject textual information from the reference visual features. The cross-modal Transformer has 4 layers, 8 heads for multi-head cross-attention, and 768 hidden layer dimensions. Three cross-attention blocks are used in the cross-modal Transformer to achieve low-, mid-, and high-level feature interaction between the image and text. Specifically, the low-, mid-, and high-level visual features of the input image are used as Q, and the low-, mid-, and high-level semantic features of the text information are used as K and V. An attention mechanism is employed to achieve interactive fusion of image and text features. For each feature level... The calculation formula is as follows: ; in, These are the BERT-processed text features of the query cases. With the Layer (of which) There are three levels of visual features: L, M, and H. Extract the image-text fusion features obtained from the cross-modal Transformer fusion output. These features are then compared with the features of the query target case in the visual feature space. Having the same dimensions (are dimension-aligned) allows them to be directly used for similarity calculation and retrieval.

[0029] Step 105: Obtain the target case image from the image database and perform feature extraction to obtain the target image features.

[0030] Step 106: Calculate the local similarity and global similarity based on the image-text fusion query features and the target image features.

[0031] Quantifying combined queries using two complementary terms With target image Similarity between features. Global alignment compares the aggregated representations to capture the overall semantics at the lesion level (such as morphology and color distribution). In contrast, local alignment focuses on learning multiple region masks and aggregating region descriptors to focus on discriminative sub-regions (such as stripes, irregular pigment distributions, and degenerative-like patterns). Combining these two perspectives can generate a similarity function that is both semantically coherent and locally sensitive.

[0032] First, spatial attention masks of the same size are generated for the regions corresponding to k indices to model the contribution of local visual features to the discriminative representation. This is then combined with aggregated region descriptors to construct a local discriminative pattern model. For each feature level... and regional index The process of calculating a region descriptor is as follows: ; in, Generate a spatial attention mask representation of the Hadamard product operation, and It is a mean pooling operator that generates a one-dimensional vector; it applies the region description to... and available and ; The final aggregated case feature representation is a high-dimensional feature vector generated by pooling operations after combining the image features at different levels with the attention mask for importance modeling. Indicates the first extracted by the network i The image feature tensor of the hierarchy (in this application, it specifically includes three different depths: low, medium and high). This represents a learnable attention mask whose spatial dimensions (height h and width w) correspond to the image features of the corresponding layer. Maintaining strict consistency, in its core physical sense, means adaptively quantifying and refining the contribution (weight) of each local visual feature to the overall feature representation.

[0033] Next, this application calculates local similarity by aggregating k region descriptors and measuring cosine similarity: ; in Represents cosine similarity. This is used to aggregate region descriptors (e.g., the average value of k), which enables the model to align patterns with diagnostic value without relying on explicit lesion level labels. Query features of image-text fusion obtained from the output of a cross-modal Transformer Image features of the target case The dimensions are aligned, meaning the global features of the query case combined with the text description of the image are aligned with the global features of the target case image. To achieve global alignment, capture the overall semantic meaning, and provide stable supervisory information to complement local matching, this application uses mean pooling to obtain the feature representations of each case from the global features of the derived query and target cases, and applies cosine similarity to calculate global similarity. ; Step 107: The local similarity and the global similarity are weighted and fused to obtain the final similarity score between the search query and each target case sample.

[0034] In the clinical diagnosis of lesions, global features provide the similarity of the overall visual features of the lesions, while local features ensure the specific details of the local features of lesions in a particular disease type. By combining the two perspectives of global feature similarity and local feature similarity, complementary information is provided, thereby obtaining the quantitative similarity between the query case and the target case, which is the final query-target similarity score.

[0035] ; in, These are coefficients representing the trade-off between local and global signals, and larger ones... This indicates a stronger match to the query target. In skin cancer case searches, significant diagnostic clues are often limited to specific sub-regions of lesions (e.g., streaks, irregular pigmentation distribution, and similar fading patterns); therefore, a priority is given to these. of Parameter value, and at the same time It is used as a holistic semantic constraint.

[0036] Step 108: Sort all case samples in the image database in descending order according to the final similarity score, and recall a preset number of cases with the highest ranking as the search results.

[0037] Step 109: Compare the search results with the diagnostic tags of the search query to verify the accuracy of the search.

[0038] Based on the same inventive concept, this application also provides a global-local alignment-based multimodal skin cancer case retrieval device for implementing the aforementioned global-local alignment-based multimodal skin cancer case retrieval method. The solution provided by this device is similar to the implementation described in the above method. Therefore, the specific limitations of one or more embodiments of the global-local alignment-based multimodal skin cancer case retrieval device provided below can be found in the limitations of the global-local alignment-based multimodal skin cancer case retrieval method described above, and will not be repeated here.

[0039] In one exemplary embodiment, a skin cancer multimodal case retrieval device based on global-local alignment is provided, comprising: The database construction module is used to construct an image database and define retrieval queries. The image database contains multiple case samples, each case sample including a lesion image and its corresponding diagnostic label. The retrieval query includes: a reference lesion image and a clinical text description associated with the reference lesion image. A visual encoding module is used to perform hierarchical visual encoding on the reference lesion image and extract multi-level visual features of the reference lesion image. The text encoding module is used to encode the clinical text description to obtain text semantic features; The feature interaction fusion module is used to input the multi-level visual features and the text semantic features into the cross-modal Transformer for feature interaction fusion, and output the image-text fusion query features. The feature extraction module is used to obtain target case images from the image database and perform feature extraction to obtain target image features; The local similarity and global similarity calculation module is used to calculate local similarity and global similarity based on the image-text fusion query features and the target image features, respectively. The final similarity score calculation module is used to perform weighted fusion of the local similarity and the global similarity to obtain the final similarity score between the search query and each target case sample; The descending sorting module is used to sort all case samples in the image database in descending order according to the final similarity score, and recall a preset number of cases with the highest ranking as search results; The retrieval verification module is used to verify the accuracy of the retrieval by comparing the retrieval results with the diagnostic tags of the retrieval query.

[0040] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 2 As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs in the non-volatile storage media to run. The database stores multimodal skin cancer case retrieval data based on global-local alignment. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a multimodal skin cancer case retrieval method based on global-local alignment.

[0041] Those skilled in the art will understand that Figure 2 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0042] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0043] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0044] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0045] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0046] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0047] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0048] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A multimodal skin cancer case retrieval method based on global-local alignment, characterized in that, The skin cancer multimodal case retrieval method based on global-local alignment includes: An image database is constructed and a retrieval query is defined. The image database contains multiple case samples, each case sample including a lesion image and its corresponding diagnostic label. The retrieval query includes: a reference lesion image and a clinical text description associated with the reference lesion image. Hierarchical visual encoding is performed on the reference lesion image to extract multi-level visual features of the reference lesion image; Text encoding is performed on the clinical text description to obtain text semantic features; The multi-level visual features and the text semantic features are input into a cross-modal Transformer for feature interaction and fusion, and the resulting image-text fusion query features are output. The target case image is obtained from the image database and its features are extracted to obtain the target image features; Based on the image-text fusion query features and the target image features, local similarity and global similarity are calculated respectively; The local similarity and the global similarity are weighted and fused to obtain the final similarity score between the search query and each target case sample; Based on the final similarity score, all case samples in the image database are sorted in descending order, and a predetermined number of cases with the highest ranking are recalled as search results; The accuracy of the retrieval is verified by comparing the consistency of the retrieval results with the diagnostic tags of the retrieval query.

2. The skin cancer multimodal case retrieval method based on global-local alignment according to claim 1, characterized in that, Hierarchical visual encoding of the reference lesion image, specifically extracting multi-level visual features of the reference lesion image, includes: A multi-level visual backbone network based on the Swin Transformer architecture is used to extract features from the reference lesion image. The multi-level visual features include low-level features, mid-level features and high-level features, which are extracted from the 10th, 11th and 12th layers of the visual backbone network, respectively. The dimension of each level feature is 768.

3. The skin cancer multimodal case retrieval method based on global-local alignment according to claim 1, characterized in that, The text encoding of the clinical text description to obtain text semantic features specifically includes: The clinical text description was encoded using a pre-trained BERT model to obtain a word sequence and corresponding text semantic features; The multi-level visual features and the text semantic features are input into a cross-modal Transformer for feature interaction and fusion, and the output image-text fusion query features specifically include: Construct a cross-modal Transformer containing three cross-attention blocks; The multi-level visual features are used as query vectors Q, and the text semantic features are used as key vectors K and value vectors V. The interaction of low, medium and high-level image and text features is realized through the cross attention block. The output is an image and text fusion query feature that is aligned with the dimensions of the multi-level visual features. The image and text fusion query feature includes low-level fusion features, medium-level fusion features and high-level fusion features.

4. The skin cancer multimodal case retrieval method based on global-local alignment according to claim 1, characterized in that, The process of calculating local similarity is as follows: For each level of the image-text fusion query features and the target image features, generate k spatial attention masks; The spatial attention mask is subjected to a Hadamard product with the features of the corresponding level, and a region descriptor is generated by mean pooling. The local feature similarity is calculated by substituting the region descriptor of the image-text fusion query feature and the region descriptor of the target image feature into the cosine similarity formula, and then aggregating the local feature similarities of k regions to obtain the local similarity.

5. The method for retrieving multimodal skin cancer cases based on global-local alignment according to claim 1, characterized in that, The process of calculating the global similarity is as follows: Global mean pooling is performed on the image-text fusion query features and the target image features respectively to obtain global feature representations of the query and the target; The global similarity is obtained by substituting the global feature representation of the query and the global feature representation of the target into the cosine similarity formula.

6. The method for retrieving multimodal skin cancer cases based on global-local alignment according to claim 1, characterized in that, The calculation process for the final similarity score is as follows: ； in, This indicates the final similarity score; Coefficients representing the trade-off between local and global signals; Indicates local similarity; Indicates global similarity.

7. A skin cancer multimodal case retrieval device based on global-local alignment, characterized in that, The skin cancer multimodal case retrieval device based on global-local alignment includes: The database construction module is used to construct an image database and define retrieval queries. The image database contains multiple case samples, each case sample including a lesion image and its corresponding diagnostic label. The retrieval query includes: a reference lesion image and a clinical text description associated with the reference lesion image. A visual encoding module is used to perform hierarchical visual encoding on the reference lesion image and extract multi-level visual features of the reference lesion image. The text encoding module is used to encode the clinical text description to obtain text semantic features; The feature interaction fusion module is used to input the multi-level visual features and the text semantic features into the cross-modal Transformer for feature interaction fusion and output the image-text fusion query features. The feature extraction module is used to obtain target case images from the image database and perform feature extraction to obtain target image features; The local similarity and global similarity calculation module is used to calculate local similarity and global similarity based on the image-text fusion query features and the target image features, respectively. The final similarity score calculation module is used to perform weighted fusion of the local similarity and the global similarity to obtain the final similarity score between the search query and each target case sample; The descending sorting module is used to sort all case samples in the image database in descending order according to the final similarity score, and recall a preset number of cases with the highest ranking as search results; The retrieval verification module is used to verify the accuracy of the retrieval by comparing the retrieval results with the diagnostic tags of the retrieval query.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the skin cancer multimodal case retrieval method based on global-local alignment as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the skin cancer multimodal case retrieval method based on global-local alignment as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the skin cancer multimodal case retrieval method based on global-local alignment as described in any one of claims 1-6.

Images