Medical image-text heterogeneous data retrieval method and system, and storage medium

Abstract

The application relates to a medical image and text heterogeneous data retrieval method and system and a storage medium, and relates to the technical field of medical data information processing. The method comprises the following steps: determining a fusion feature sequence of multiple modes in a medical sample; performing mask attention aggregation on the fusion feature sequence to obtain an enhanced feature sequence; learning a concept base coefficient feature and a global semantic feature of each sample through a star attention mechanism to construct a total loss function; iteratively updating a plurality of learnable parameters to obtain a plurality of target parameters, with the minimum total loss function as the target; updating model parameters of a preset model to the target parameters to obtain a target model; and using the target model to determine the similarity between the target concept base coefficient feature and the target global semantic feature of a query object and the concept base coefficient feature and the global semantic feature of all medical samples in a target data set associated with the target model, so as to obtain a target retrieval result matched with the query object.

Description

Technical Field

[0001] This application relates to the field of medical data information processing technology, and in particular to methods, systems and storage media for retrieving heterogeneous medical image and text data. Background Technology

[0002] In clinical practice, doctors often need to retrieve similar case images and their corresponding diagnostic reports based on existing medical images, or retrieve relevant medical images based on text descriptions. Radiologists, when reviewing images, hope to quickly retrieve past cases similar to the current patient to aid in diagnostic decisions. Medical researchers hope to retrieve images of specific lesion types and their corresponding reports from large-scale datasets to support clinical research. In medical education, students need to obtain typical cases for learning through text or image queries. Therefore, medical heterogeneous data retrieval (i.e., retrieving text from images or images from text) has broad application value in clinical diagnosis, medical education, disease research, and decision support.

[0003] Achieving heterogeneous data retrieval requires mapping heterogeneous data with completely different distributions and representations to a unified common semantic space to achieve semantic alignment. However, medical data is complex and specialized. Medical images typically contain rich structural information, with the imaging features of different tissues and organs intertwined. Lesions may appear in different locations, and the imaging manifestations of the same lesion may differ significantly in different locations. Furthermore, medical diagnostic reports contain a large number of technical terms, complex grammatical structures, and implicit semantic information. Current image-text heterogeneous retrieval methods fail to fully consider the multi-source complexity of medical data, resulting in low retrieval accuracy. Moreover, current medical heterogeneous data retrieval methods typically employ fully connected self-attention mechanisms, causing the representations between different locations to gradually converge, leading to over-smoothing and difficulty in preserving the specific information of each location, resulting in low accuracy in fine-grained differentiation. Summary of the Invention

[0004] This invention provides a method, system, and storage medium for retrieving heterogeneous medical image and text data, at least addressing the problem of poor retrieval accuracy caused by insufficient utilization of multimodal information in medical samples and the gradual convergence of representations of different modal features, resulting in the loss of specific information for each feature. The technical solution of this invention is as follows: According to a first aspect of the present invention, a method for retrieving heterogeneous medical image and text data is provided. The method includes: employing a target model to perform the following processing on a query object to obtain a target retrieval result: determining the target conceptual basis coefficient features and target global semantic features of the query object; obtaining a target retrieval result matching the query object based on the similarity between the target conceptual basis coefficient features and target global semantic features of the query object and the conceptual basis coefficient features and global semantic features of all medical samples in the retrieval dataset associated with the target model; the query object includes querying medical images or querying report text; wherein the target model is obtained by training a preset model using the following operations: extracting embedding sequences of multiple modalities from the medical samples; and processing the embedding sequences of the multiple modalities... The features are concatenated and fused to obtain a fused feature sequence. Multiple modalities include medical images, free report text, and medical label information. Based on the pre-defined information flow direction of the sequence features at each position in the fused feature sequence, masked attention aggregation is performed on the fused feature sequence to obtain an enhanced feature sequence. A star-shaped attention mechanism is used to determine the concept basis coefficient features and global semantic features based on the enhanced feature sequence and multiple learnable weight matrices. A total loss function is constructed based on the concept basis coefficient features and global semantic features. With the goal of minimizing the total loss function, multiple learnable parameters are iteratively updated to obtain multiple target parameters. These multiple learnable parameters include multiple learnable weight matrices and multiple learnable latent concept vectors. The model parameters of the pre-defined model are updated to the target parameters to obtain the target model.

[0005] As one implementation, the embedding sequence of multiple modalities of a medical sample is extracted, including extracting the embedding vector of local units in each modality of the medical sample according to a preset encoder to obtain a multimodal embedding sequence of the medical sample; the preset encoder includes a medical image encoder, a text report encoder, and a label encoder; the multimodal embedding sequence includes an image patch embedding sequence, a text embedding sequence, and a label embedding sequence; wherein, according to the medical image encoder, the embedding vector of each uniform image patch in the medical image of the medical sample in a preset dimension is extracted to obtain an image patch embedding sequence containing image spatial context information; and, according to the text report encoder, the embedding vector of each word in the free report text of the medical sample in a preset dimension is extracted to obtain a text embedding sequence containing text context background information; and, according to the label encoder, the embedding vector of medical label information of the medical sample in a preset dimension is extracted to obtain a label embedding sequence; the image patch embedding sequence, the text embedding sequence, and the label embedding sequence are concatenated and fused to obtain a fused feature sequence.

[0006] In this implementation, the local unit embedding sequences of different modalities (images, text, labels) are embedded by a preset encoder, preserving the local details of the original data and mapping them to the same embedding space. The embedding sequences of different modalities are fused into a feature sequence by direct splicing. The spliced ​​fused feature sequence can be naturally input into the subsequent attention layer, allowing interaction to be established between the positions of different modalities, thereby capturing fine-grained cross-modal correlations.

[0007] As one implementation, position representation is the embedding vector of the local unit of each modality in the fused feature sequence; a preset information flow direction is used to control each position to learn the semantic information of other positions within the allowed flow direction range; the enhanced feature sequence represents the feature sequence of each position that has fused cross-modal interaction information within the allowed range of the preset information flow direction; based on the preset information flow direction of the sequence features of each position in the fused feature sequence, mask attention aggregation is performed on the fused feature sequence to obtain the enhanced feature sequence, including: dividing the fused feature sequence into three continuous index intervals, including image index interval, text index interval, and label index interval; determining the preset information flow direction of the sequence features of each position in the fused feature sequence based on the preset allowed attention set, and constructing mask moments. The mask matrix allows the use of sequence features at target locations as query objects, restricting the index range of target key positions that are allowed to be followed by the query object. Positions in the mask matrix corresponding to the allowed attention set are determined as allowed values, while other non-corresponding positions are determined as disallowed values. Specifically, when the query object of the allowed attention set is a sequence feature within an image index range, the target key position is determined to be a sequence feature within both the image and text index ranges; when the query object of the allowed attention set is a sequence feature within a text index range, the target key position is determined to be a sequence feature within both the image and text index ranges; when the query object of the allowed attention set is a sequence feature within a tag index range, the target key position is determined to be a sequence feature within all three index ranges.

[0008] In this implementation, by pre-setting the allowed attention set for each position, the information flow is precisely controlled, clearly specifying which positions can follow each other and which are prohibited, avoiding unreasonable cross-modal or cross-positional information interference. Based on this, the allowed and disallowed values ​​in the mask matrix are clearly defined, avoiding invalid calculations and reducing the computational complexity of the attention mechanism. Thus, by restricting the information flow, the enhanced features of each position focus more on the relevant semantics within its allowed attention range, reducing noise from irrelevant modalities, thereby obtaining a clearer and more discriminative local representation.

[0009] As one implementation, a linear projection is performed on the fused feature sequence to obtain a first query vector, a first key vector, and a first value vector. Based on the first query vector and the first key vector, the original relevance score between each position in the fused feature sequence is determined. Based on the mask matrix, the weighted sum of the original relevance scores and the first value vector, a masked attention result is obtained. The masked attention result is then joined with the feature residuals of the previous layer and subjected to layer normalization to obtain an intermediate feature sequence. An independent nonlinear transformation is performed on the sequence features at each position in the intermediate feature sequence, and this transformation is then joined with the intermediate feature sequence using residuals and layer normalization to obtain an enhanced feature sequence.

[0010] In this implementation, multi-layer attention allows the model to progressively fuse cross-modal information from shallow to deep layers. Each layer extracts higher-order semantic features based on the previous layer, resulting in a more expressive enhanced feature sequence. During multi-layer processing, a mask matrix controls the information flow within and between modalities, preserving both the spatial structure of the image and the temporal structure of the text, while progressively strengthening the fine-grained semantic correspondence between image, text, and label. Simultaneously, the masking mechanism limits the scope of attention at each position within each layer, preventing the indiscriminate fusion of full-sequence information across all positions. This maintains the specificity of each position during multi-layer iterations and mitigates the oversmoothing problem common in fully connected self-attention.

[0011] As one implementation, multiple learnable weight matrices include a learnable first weight matrix, a learnable second weight matrix, and a learnable third weight matrix. A star-shaped attention mechanism is employed to determine concept basis coefficient features and global semantic features based on the enhanced feature sequence and multiple learnable weight matrices. This includes: initializing multiple learnable concept basis vectors; determining an initial query vector based on the enhanced feature sequence; performing multi-level iterative refinement on the initial query vector to obtain a refined query vector and concept basis coefficient features; and generating global semantic features based on the refined query vector and concept basis coefficient features. Each iteration of the multi-level iterative refinement of the initial query vector includes: determining the current layer's concept basis coefficients based on the query vector from the previous layer and multiple learnable concept basis vectors. The system first identifies the concept basis coefficient features. A linear projection of multiple learnable concept basis vectors onto the learnable first weight matrix yields the query vector. A linear projection of the enhanced feature sequence onto the learnable second weight matrix yields the key vector. A linear projection of the enhanced feature sequence onto the learnable third weight matrix yields the value vector. Based on the concept basis coefficient features of the current layer, a star-shaped attention aggregation is performed on the query vector, key vector, and value vector to obtain the query increment. The query increment is then residually connected to the query vector of the previous layer and subjected to layer normalization to obtain an intermediate normalized representation. This intermediate normalized representation is input into a feedforward network to obtain a feedforward output. Finally, the feedforward output is residually connected to the intermediate normalized representation and subjected to layer normalization to obtain the query vector of the current layer.

[0012] In this implementation, by constructing explicit concept basis vectors and calculating concept basis coefficient features, samples are abstracted into linear combinations of multiple learnable concepts, improving model interpretability and achieving decoupled semantic representation. A star-shaped attention mechanism is used, with concept basis vectors as central query nodes and the enhanced feature sequences of samples as key and value nodes. This topology allows each concept basis vector to directly extract the most semantically relevant information from all sample features, eliminating redundant computations of internal sample interactions. This achieves efficient and semantically focused global information aggregation indexed by concepts. An iterative refinement mechanism optimizes features from coarse to fine, resulting in global semantic features that not only include the refined sample's own query vector but also superimposed with concept basis vector combinations weighted by concept basis coefficients. This ensures that the query vector contains both the sample's unique semantic details and its shared attributes in the concept space, forming a more complete and robust global representation.

[0013] As one implementation, the total loss function includes multi-label classification loss and concept alignment loss. Based on concept basis coefficient features and global semantic features, the total loss function is constructed as follows: multi-label classification loss is constructed based on the maximum similarity between the global semantic features of the medical sample and multiple positive label embedding vectors, and the minimum similarity between the global semantic features of the medical sample and multiple negative label embedding vectors; concept alignment loss is constructed based on the similarity between each concept basis feature vector in the concept basis coefficient features of the medical sample and multiple positive label embedding vectors; the total loss function is obtained by weighting the multi-label classification loss and the concept alignment loss.

[0014] As one implementation method, with the goal of minimizing the total loss function, multiple learnable parameters are iteratively updated to obtain multiple target parameters. This includes: initializing multiple learnable parameters; determining the partial derivatives of the total loss function with respect to the initialized learnable parameters based on the initialized learnable parameters; and iteratively updating the learnable parameters based on the initialized learnable parameters and the partial derivatives until a preset condition is met, thereby obtaining multiple target parameters.

[0015] In this implementation, the learnable parameters in the attention mechanism are iteratively updated with the goal of minimizing the total loss function. This enables multiple learnable parameters to automatically learn better feature representations from multimodal data, thereby improving the accuracy and discriminative ability of subsequent local and global semantic features.

[0016] As one implementation method, the target retrieval result matching the query object is obtained by comparing the similarity between the target concept basis coefficient features and the target global semantic features of the query object and the concept basis coefficient features and global semantic features of all medical samples in the retrieval dataset associated with the target model. This includes: selecting multiple candidate samples from multiple medical samples whose coarse-grained scores meet a first preset threshold based on the coarse-grained scores of the target global semantic features of the query object and the global semantic features of multiple medical samples; obtaining fine-grained scores based on the similarity between the target concept basis coefficient features of the query object and the concept basis coefficient features of multiple candidate samples; obtaining the final score of each candidate sample by weighted fusion of the coarse-grained score and the fine-grained score; and determining the candidate sample with the highest final score as the target retrieval result. Specifically, if the query object is determined to be a query image, the target retrieval result is a text report matching the query image; if the query object is determined to be a query text report, the target retrieval result is a medical image matching the query text report.

[0017] In this implementation, by comparing the conceptual similarity between the query target and each medical sample, the target retrieval results are filtered out, realizing a semantic matching-based retrieval mechanism that can more accurately locate the sample most relevant to the query target.

[0018] According to a second aspect of the present invention, a medical image-text heterogeneous data retrieval system is provided, the medical image-text heterogeneous data retrieval system comprising: The feature sequence construction module is configured to extract the embedding sequences of multiple modalities in medical samples; and to concatenate and fuse the embedding sequences of multiple modalities to obtain a fused feature sequence; the multiple modalities include medical images, free report text, and medical label information.

[0019] The masked self-attention module is configured to perform masked attention aggregation on the fused feature sequence according to the preset information flow direction of the sequence features at each position in the fused feature sequence, so as to obtain the enhanced feature sequence.

[0020] The star-shaped attention module is configured to use a star-shaped attention mechanism to determine the concept basis coefficient features and global semantic features based on the enhanced feature sequence and multiple learnable weight matrices.

[0021] The target model determination module is configured to construct a total loss function based on concept basis coefficient features and global semantic features; with the goal of minimizing the total loss function, iteratively update multiple learnable parameters to obtain multiple target parameters; the multiple learnable parameters include multiple learnable weight matrices and multiple learnable latent concept vectors; and update the model parameters of the preset model to the target parameters to obtain the target model.

[0022] The retrieval module is configured to use a target model to perform the following processing on the query object to obtain the target retrieval results: determine the target conceptual basis coefficient features and target global semantic features of the query object; based on the similarity between the target conceptual basis coefficient features and target global semantic features of the query object and the conceptual basis coefficient features and global semantic features of all medical samples in the dataset to be retrieved associated with the target model, obtain the target retrieval results that match the query object; the query object includes querying medical images or querying report text.

[0023] The medical image and text heterogeneous data retrieval system is configured to perform a medical image and text heterogeneous data retrieval method as described in the first aspect and any of its possible implementations.

[0024] According to a third aspect of the present invention, a medical image-text heterogeneous data retrieval device is provided, the device being configured to perform a medical image-text heterogeneous data retrieval method as described in the first aspect and any possible implementation thereof.

[0025] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which instructions are stored, such that when the instructions in the computer-readable storage medium are executed by a processor of a medical image and text heterogeneous data retrieval device, the medical image and text heterogeneous data retrieval device is able to perform a medical image and text heterogeneous data retrieval method as described in the first aspect and any possible implementation thereof.

[0026] According to a fifth aspect of the present invention, a computer program product is provided, the computer program product including computer instructions, which, when executed on a medical image and text heterogeneous data retrieval device, cause the medical image and text heterogeneous data retrieval device to perform the medical image and text heterogeneous data retrieval method of the first aspect and any possible implementation thereof.

[0027] The technical solutions provided by the embodiments of the present invention bring at least the following beneficial effects: The present invention determines the embedding vector of local units of each modality extracted from medical samples, and concatenates and fuses the embedding vectors of multiple modalities to obtain a fused feature sequence. This can capture fine-grained correspondences across modalities, retain the fine-grained semantic information of each modality after multimodal fusion, and further employs an attention mechanism to interactively learn the fine-grained semantic associations between different modalities. By pre-setting the information flow direction, it controls each position in the fused feature sequence to learn the semantic information of other positions within the allowed flow direction range, explicitly constrains the information interaction range between different modalities, achieves high-level semantic alignment, and prevents the semantic representations of different modalities from gradually converging due to the free flow of information, effectively maintaining the specific information of each modality and ensuring the subsequent retrieval process. Accuracy is achieved by fully exploring complementary information between different modalities, avoiding retrieval bias caused by the lack of information in a single modality. Simultaneously, multiple learnable parameters in the multimodal interactive learning process are iteratively optimized by jointly optimizing multi-label classification loss and concept alignment loss, thereby training the target model and achieving semantic consistency in cross-modal learning. This ensures that, under the attention mechanism, while multimodal information achieves concept alignment in the shared semantic space, the unique structural information of each modality is strengthened. Based on this, the target model is used to extract the target concept basis coefficient features and target global semantic features of the query object. The similarity between these features and the concept basis coefficient features and global semantic features of the medical sample is used for cross-modal retrieval, enabling the capture of subtle differences while grasping the overall semantics, effectively improving the accuracy of the retrieval results.

[0028] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0030] Figure 1 This is a schematic diagram illustrating a medical image and text heterogeneous data retrieval system according to an exemplary embodiment; Figure 2 This is a flowchart illustrating a medical image-text heterogeneous data retrieval method according to an exemplary embodiment. Figure 1 ; Figure 3 This is a flowchart illustrating a medical image-text heterogeneous data retrieval method according to an exemplary embodiment. Figure 2 ; Figure 4 This is a schematic diagram of a medical image and text heterogeneous data retrieval device according to an exemplary embodiment. Detailed Implementation

[0031] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0032] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0033] Before providing a detailed introduction to the medical image and text heterogeneous data retrieval method provided in the embodiments of this application, let's first briefly introduce the application scenarios and implementation environment involved in the embodiments of this application.

[0034] With the rapid development of medical imaging technology and the widespread adoption of electronic medical record systems, medical institutions generate massive amounts of medical image data (such as X-rays, CT scans, MRI, and ultrasound images) and corresponding text-based diagnostic reports every day. This multimodal data provides rich and valuable data resources for disease-aided diagnosis, clinical decision support, precision medicine, medical education, and medical research.

[0035] In clinical practice, doctors often need to retrieve similar case images and their corresponding diagnostic reports based on existing medical images, or retrieve relevant medical images based on text descriptions. Radiologists, when reviewing images, hope to quickly retrieve past cases similar to the current patient to aid diagnostic decisions. Medical researchers hope to retrieve images of specific lesion types and their corresponding reports from large-scale datasets to support clinical research. In medical education, students need to obtain typical cases for learning through text or image queries. Therefore, medical heterogeneous data retrieval (i.e., retrieving text from images or images from text) has broad application value in clinical diagnosis, medical education, disease research, and decision support. However, medical images are visual data stored in pixel matrix form, reflecting the morphological characteristics of tissue structure and lesions, while diagnostic reports are text data stored in natural language form, describing radiological findings, clinical impressions, and diagnostic conclusions. The two modalities of data have completely different data distributions and representations, and their underlying feature spaces lack direct comparability, making it difficult to map heterogeneous medical image and text data to a unified common semantic space to achieve semantic alignment. Furthermore, the complexity and specialization of medical data increase the difficulty of understanding it. Medical images typically contain rich structural information, with the imaging features of different tissues and organs intertwined. Lesions may appear in different locations, and the imaging manifestations of the same lesion may differ significantly in different locations. At the same time, medical diagnostic reports contain a large number of technical terms, complex grammatical structures, and implicit semantic information. Therefore, retrieval of heterogeneous image and text data in the medical field presents the following two challenges.

[0036] First, current heterogeneous image-text retrieval methods fail to fully utilize the labeling and other information in medical data, resulting in low retrieval accuracy. In actual clinical practice, medical samples typically include not only images and text reports but also structured medical discovery labels extracted from diagnostic reports. These labels possess clear clinical semantics. Current heterogeneous medical data retrieval methods usually only utilize information from image and text modalities, failing to effectively leverage this rich structured labeling information. Even when labeling information is used, it is merely used as a classification supervision signal, failing to deeply integrate labeling information into the representation learning of heterogeneous data. Second, current heterogeneous medical data retrieval methods typically employ fully connected self-attention mechanisms, which suffer from problems such as oversmoothing and low computational efficiency. Each position in the sequence calculates attention weights with all other positions. This fully connected design leads to computational complexity proportional to the square of the sequence length, limiting the model's ability to process high-resolution medical images. Furthermore, multi-layered fully connected attention causes the representations between different positions to gradually converge, resulting in an "oversmoothing" problem. This makes it difficult for the model to retain the specific information of each position, reducing the accuracy of fine-grained distinctions.

[0037] To address the aforementioned issues, this application proposes a medical image-text heterogeneous data retrieval method. It employs an attention mechanism to interactively learn fine-grained semantic associations between different modalities. By pre-setting information flow directions, it controls each position in the fused feature sequence to learn semantic information from other positions within an allowed flow range. This explicitly constrains the scope of information interaction between different modalities, achieving high-level semantic alignment while preventing the semantic representations of different modalities from gradually converging due to free information flow. This effectively maintains the specific information of each modality, ensuring accuracy in subsequent retrieval processes. Furthermore, by jointly optimizing multi-label classification loss and concept alignment loss, iteratively optimizes multiple learnable parameters in the multimodal interactive learning process, achieving semantic consistency in cross-modal learning. This ensures that under the attention mechanism, while multimodal information achieves concept alignment in a shared semantic space, it also strengthens the unique structural information of each modality. This guarantees that the retrieval process can capture subtle differences while grasping the overall semantics, effectively improving the accuracy of retrieval results.

[0038] Secondly, the implementation architecture involved in this application will be briefly introduced below.

[0039] Figure 1 This is a schematic diagram of a medical image and text heterogeneous data retrieval system provided in this application. For example... Figure 1 As shown, the medical image and text heterogeneous data retrieval system includes a feature sequence construction module 11, a masked self-attention module 12, a star-shaped attention module 13, a training module 14, and a retrieval module 15.

[0040] The feature sequence construction module 11, mask self-attention module 12, star-shaped attention module 13, target model determination module 14 and retrieval module 15 are connected by communication.

[0041] The feature sequence construction module 11 is configured to extract the embedding sequences of multiple modalities in the medical sample; and to concatenate and fuse the embedding sequences of multiple modalities to obtain a fused feature sequence; the multiple modalities include medical images, free report text, and medical label information.

[0042] The masked self-attention module 12 is configured to perform masked attention aggregation on the fused feature sequence according to the preset information flow direction of the sequence features at each position in the fused feature sequence to obtain the enhanced feature sequence.

[0043] The star-shaped attention module 13 is configured to use a star-shaped attention mechanism to determine the concept basis coefficient features and global semantic features based on the enhanced feature sequence and multiple learnable weight matrices.

[0044] The target model determination module 14 is configured to construct a total loss function based on the concept basis coefficient features and global semantic features; to iteratively update multiple learnable parameters with the goal of minimizing the total loss function, thereby obtaining multiple target parameters; the multiple learnable parameters include multiple learnable weight matrices and multiple learnable latent concept vectors; and to update the model parameters of the preset model to the target parameters, thereby obtaining the target model.

[0045] The retrieval module 15 is configured to use a target model to perform the following processing on the query object to obtain the target retrieval result: determine the target conceptual basis coefficient features and target global semantic features of the query object, and obtain the target retrieval result matching the query object based on the similarity between the target conceptual basis coefficient features and target global semantic features of the query object and the conceptual basis coefficient features and global semantic features of all medical samples in the retrieval dataset associated with the target model; the query object includes querying medical images or querying report text.

[0046] As one implementation, the embedding sequence of multiple modalities of a medical sample is extracted, including extracting the embedding vector of local units in each modality of the medical sample according to a preset encoder to obtain a multimodal embedding sequence of the medical sample; the preset encoder includes a medical image encoder, a text report encoder, and a label encoder; the multimodal embedding sequence includes an image patch embedding sequence, a text embedding sequence, and a label embedding sequence; wherein, according to the medical image encoder, the embedding vector of each uniform image patch in the medical image of the medical sample in a preset dimension is extracted to obtain an image patch embedding sequence containing image spatial context information; and, according to the text report encoder, the embedding vector of each word in the free report text of the medical sample in a preset dimension is extracted to obtain a text embedding sequence containing text context background information; and, according to the label encoder, the embedding vector of medical label information of the medical sample in a preset dimension is extracted to obtain a label embedding sequence; the image patch embedding sequence, the text embedding sequence, and the label embedding sequence are concatenated and fused to obtain a fused feature sequence.

[0047] As one implementation, position representation is the embedding vector of the local unit of each modality in the fused feature sequence; a preset information flow direction is used to control each position to learn the semantic information of other positions within the allowed flow direction range; the enhanced feature sequence represents the feature sequence of each position that has fused cross-modal interaction information within the allowed range of the preset information flow direction; based on the preset information flow direction of the sequence features of each position in the fused feature sequence, mask attention aggregation is performed on the fused feature sequence to obtain the enhanced feature sequence, including: dividing the fused feature sequence into three continuous index intervals, including image index interval, text index interval, and label index interval; determining the preset information flow direction of the sequence features of each position in the fused feature sequence based on the preset allowed attention set, and constructing mask moments. The mask matrix allows the use of sequence features at target locations as query objects, restricting the index range of target key positions that are allowed to be followed by the query object. Positions in the mask matrix corresponding to the allowed attention set are determined as allowed values, while other non-corresponding positions are determined as disallowed values. Specifically, when the query object of the allowed attention set is a sequence feature within an image index range, the target key position is determined to be a sequence feature within both the image and text index ranges; when the query object of the allowed attention set is a sequence feature within a text index range, the target key position is determined to be a sequence feature within both the image and text index ranges; when the query object of the allowed attention set is a sequence feature within a tag index range, the target key position is determined to be a sequence feature within all three index ranges.

[0048] As one implementation, a linear projection is performed on the fused feature sequence to obtain a first query vector, a first key vector, and a first value vector. Based on the first query vector and the first key vector, the original relevance score between each position in the fused feature sequence is determined. Based on the mask matrix, the weighted sum of the original relevance scores and the first value vector, a masked attention result is obtained. The masked attention result is then joined with the feature residuals of the previous layer and subjected to layer normalization to obtain an intermediate feature sequence. An independent nonlinear transformation is performed on the sequence features at each position in the intermediate feature sequence, and this transformation is then joined with the intermediate feature sequence using residuals and layer normalization to obtain an enhanced feature sequence.

[0049] As one implementation, multiple learnable weight matrices include a learnable first weight matrix, a learnable second weight matrix, and a learnable third weight matrix. A star-shaped attention mechanism is employed to determine concept basis coefficient features and global semantic features based on the enhanced feature sequence and multiple learnable weight matrices. This includes: initializing multiple learnable concept basis vectors; determining an initial query vector based on the enhanced feature sequence; performing multi-level iterative refinement on the initial query vector to obtain a refined query vector and concept basis coefficient features; and generating global semantic features based on the refined query vector and concept basis coefficient features. Each iteration of the multi-level iterative refinement of the initial query vector includes: determining the current layer's concept basis coefficients based on the query vector from the previous layer and multiple learnable concept basis vectors. The system first identifies the concept basis coefficient features. A linear projection of multiple learnable concept basis vectors onto the learnable first weight matrix yields the query vector. A linear projection of the enhanced feature sequence onto the learnable second weight matrix yields the key vector. A linear projection of the enhanced feature sequence onto the learnable third weight matrix yields the value vector. Based on the concept basis coefficient features of the current layer, a star-shaped attention aggregation is performed on the query vector, key vector, and value vector to obtain the query increment. The query increment is then residually connected to the query vector of the previous layer and subjected to layer normalization to obtain an intermediate normalized representation. This intermediate normalized representation is input into a feedforward network to obtain a feedforward output. Finally, the feedforward output is residually connected to the intermediate normalized representation and subjected to layer normalization to obtain the query vector of the current layer.

[0050] As one implementation, the total loss function includes multi-label classification loss and concept alignment loss. Based on concept basis coefficient features and global semantic features, the total loss function is constructed as follows: multi-label classification loss is constructed based on the maximum similarity between the global semantic features of the medical sample and multiple positive label embedding vectors, and the minimum similarity between the global semantic features of the medical sample and multiple negative label embedding vectors; concept alignment loss is constructed based on the similarity between each concept basis feature vector in the concept basis coefficient features of the medical sample and multiple positive label embedding vectors; the total loss function is obtained by weighting the multi-label classification loss and the concept alignment loss.

[0051] As one implementation method, with the goal of minimizing the total loss function, multiple learnable parameters are iteratively updated to obtain multiple target parameters. This includes: initializing multiple learnable parameters; determining the partial derivatives of the total loss function with respect to the initialized learnable parameters based on the initialized learnable parameters; and iteratively updating the learnable parameters based on the initialized learnable parameters and the partial derivatives until a preset condition is met, thereby obtaining multiple target parameters.

[0052] As one implementation method, the target retrieval result matching the query object is obtained by comparing the similarity between the target concept basis coefficient features and the target global semantic features of the query object and the concept basis coefficient features and global semantic features of all medical samples in the retrieval dataset associated with the target model. This includes: selecting multiple candidate samples from multiple medical samples whose coarse-grained scores meet a first preset threshold based on the coarse-grained scores of the target global semantic features of the query object and the global semantic features of multiple medical samples; obtaining fine-grained scores based on the similarity between the target concept basis coefficient features of the query object and the concept basis coefficient features of multiple candidate samples; obtaining the final score of each candidate sample by weighted fusion of the coarse-grained score and the fine-grained score; and determining the candidate sample with the highest final score as the target retrieval result. Specifically, if the query object is determined to be a query image, the target retrieval result is a text report matching the query image; if the query object is determined to be a query text report, the target retrieval result is a medical image matching the query text report.

[0053] For ease of understanding, the medical image and text heterogeneous data retrieval method provided in this application will be described in detail below with reference to the accompanying drawings.

[0054] Figure 2 This is a flowchart illustrating a medical image-text heterogeneous data retrieval method according to an exemplary embodiment. Figure 1 ,like Figure 2 As shown, the medical image and text heterogeneous data retrieval method includes the following steps.

[0055] S21, extract the embedding sequences of multiple modalities from the medical sample; and concatenate and fuse the embedding sequences of multiple modalities to obtain the fused feature sequence.

[0056] These include multiple modalities such as medical images, free-report text, and medical label information.

[0057] Medical label information represents clinical judgment conclusions and is a structured medical discovery label with clear clinical semantics.

[0058] Understandably, before model training, in addition to extracting medical images and free report text from medical samples and establishing the relationship between images and text, in order to ensure the accuracy of image-text heterogeneous retrieval, medical label information from medical samples is further extracted as a supervised auxiliary condition to correct the alignment path between image features and text features, thereby narrowing the distance between images and text, making the retrieval results closer to the input medical images or free report text, and ensuring the accuracy of retrieval.

[0059] Therefore, a multimodal dataset of medical samples is constructed before extracting the embedding sequences of multiple modalities from the medical samples.

[0060] In one implementation, a dataset of medical images, free report text, medical label information and medical sample metadata for each medical sample is constructed, specifically represented by the following formula (1).

[0061] (1).

[0062] Where D is the multimodal dataset; The medical image of the i-th medical sample; The free report text for the i-th medical sample; This refers to the medical label information for the i-th medical sample.

[0063] Based on the above implementation method, the embedding sequences of multiple modalities in the medical sample are extracted, and the embedding sequences of multiple modalities are spliced ​​and fused to obtain the fused feature sequence, which specifically includes the following two steps.

[0064] First, based on a preset encoder, the embedding vectors of local units in each modality of the medical sample are extracted to obtain the multimodal embedding sequence of the medical sample.

[0065] The preset encoders include a medical image encoder, a text report encoder, and a label encoder.

[0066] Multimodal embedding sequences include image patch embedding sequences, text embedding sequences, and label embedding sequences.

[0067] In one implementation, an encoder is constructed from multi-source data such as medical images, text reports, structured labels, and metadata to extract the embedding vectors of local units of each modality in the multimodal dataset of medical samples, thereby obtaining a multimodal embedding sequence, which includes the following four aspects.

[0068] First, based on the medical image encoder, the embedding vector of each uniform image block in the medical image of the medical sample is extracted in a preset dimension to obtain the image block embedding sequence containing image spatial context information, specifically represented by the following formula (2).

[0069] A medical image encoder is constructed using the visual encoder of the MedCLIP medical vision-text pre-trained model.

[0070] (2).

[0071] in, For the i-th medical sample block image 3D embedding vector, and ; The number of patches that a medical image is uniformly divided into; For medical image encoders; Let be the embedding sequence of the i-th medical sample image.

[0072] Secondly, based on the text report encoder, the embedding vector of each word in the free report text of the medical sample in the preset dimension is extracted to obtain the text embedding sequence containing text context background information, specifically represented by the following formula (3).

[0073] A text report encoder is constructed using the text encoder of the MedCLIP medical vision-text pre-trained model.

[0074] (3).

[0075] in, The report text of the i-th medical sample Each word element 3D embedding vector, and ; The report text for the i-th medical sample The number of tokens; For text report encoder; This is the report text for the i-th medical sample; Let be the text embedding sequence of the report text for the i-th medical sample.

[0076] Third, according to the label encoder, the medical label information of the medical sample is extracted into the embedding vector in the preset dimension to obtain the label embedding sequence, which is represented by the following formula (4).

[0077] A label encoder is constructed using the text encoder of the MedCLIP medical vision-text pre-trained model.

[0078] (4).

[0079] in, Let i be the label embedding sequence of the i-th medical sample; For the i-th medical sample Each tag dimensional label embedding vector, i.e. , For tag encoders, For the i-th medical sample One tag; Let i be the set of all labels for the i-th sample. ; It is the number of labels for the i-th medical sample.

[0080] Finally, the image patch embedding sequence, text embedding sequence, and label embedding sequence are concatenated and fused to obtain the fused feature sequence.

[0081] It is understandable that images, text, and tags all exist in semantic space. For example, a pixel region in a medical image can correspond to a word in the report text or a medical tag. There is a strong point-to-point correspondence between these three in semantic space or vocabulary. Therefore, image patch embedding sequences, text embedding sequences, and tag embedding sequences can be spliced ​​and fused together.

[0082] In one implementation, a unified sequence of feature combinations from different modalities is spliced ​​together. This is achieved by splicing the first... The medical images, free text reports, and structured labels of each sample are combined into a unified sequence feature to obtain the fused feature sequence. The fused feature sequence is specifically represented by the following formula (5).

[0083] (5).

[0084] in, Let be the fusion feature sequence of the i-th medical sample; To fuse feature sequence length, ; The number of blocks into which a medical image is uniformly divided; Medical Freedom Report Text The number of lexical elements; It is the number of tags; Image embedding sequence; Embed a sequence of text; Embed the sequence for the label.

[0085] Based on this, the index ranges for images, text, and tags are respectively... , , .

[0086] In this implementation, the local unit embedding sequences of different modalities (images, text, labels) are embedded by a preset encoder, preserving the local details of the original data and mapping them to the same embedding space. The embedding sequences of different modalities are fused into a feature sequence by direct splicing. The spliced ​​fused feature sequence can be naturally input into the subsequent attention layer, allowing interaction to be established between the positions of different modalities, thereby capturing fine-grained cross-modal correlations.

[0087] S22, based on the preset information flow direction of the sequence features at each position in the fused feature sequence, perform masked attention aggregation on the fused feature sequence to obtain the enhanced feature sequence.

[0088] Each position represents the embedding vector of a local unit of each modality in the fused feature sequence.

[0089] The preset information flow direction is used to control each location to learn the semantic information of other locations within the allowed flow direction range.

[0090] The enhanced feature sequence representation incorporates cross-modal interaction information at each position within the range allowed by the preset information flow direction.

[0091] In one implementation, a mask matrix for the preset information flow direction of the sequence features at each position in the fused feature sequence is determined, and mask attention processing is performed within the constraints of the mask matrix to obtain the enhanced feature sequence. Specifically, this includes the following two steps.

[0092] Among them, the allowed focus set is used to use the sequence features of the target location as the query object, and the index range of the target key position that the query object can focus on is limited.

[0093] The attention set is allowed to forcibly define the scope of feature interactions through mandatory visual / textual / semantic privileges, thereby preventing semantic pollution and feature collapse of heterogeneous information at shallow levels.

[0094] First, a mask matrix is ​​constructed based on the preset set of allowed attention.

[0095] Based on step S21 above, the fused feature sequence is divided into three continuous index intervals, including the image index interval, the text index interval, and the label index interval.

[0096] In one implementation, first, the allowed concern set is initialized. and initially set the mask matrix. Each element is 0.

[0097] The permissible target objects for attention are determined separately for image features, text features, and label features, specifically including the following three cases.

[0098] When the value of a certain position in the allowed attention set is 1, it indicates that that position is a target key object that the query object is allowed to pay attention to.

[0099] When image features are used as the query object, it allows focusing on image features to preserve the spatial context information within the image, and allows focusing on text features to achieve heterogeneous data attention from image to text. If the first... The sequence feature is the query object, and the first sequence feature is the query object. If a sequence feature is the target key object, then the focus set is allowed. elements That is, the query object of the focus set is allowed to be the sequence features of the image index interval, and the target key position is determined as the sequence features in the image index interval and the text index interval.

[0100] When text features are used as the query object, it allows for attention to image features to achieve heterogeneous data attention from text to image, and allows for attention to text features to preserve the contextual dependencies within the text. If the first... The sequence feature is the query object, and the first sequence feature is the query object. If a sequence feature is the target key object, then the focus set is allowed. elements That is, the query object of the focus set is allowed to be the sequence feature of the text index interval, and the target key position is determined to be the sequence feature in the image index interval and the text index interval.

[0101] Using label features as query objects allows focusing on image features, text features, and label features to obtain contextual information. If the... The sequence feature is the query object, and the first sequence feature is the query object. If a sequence feature is the target key object, then the focus set is allowed. elements That is, the query object of the focus set is allowed to be the sequence feature of the tag index interval, and the target key position is determined to be the sequence feature in the image index interval, text index interval, and tag index interval.

[0102] Secondly, based on the preset allowed attention set, the preset information flow direction of the sequence features at each position of the fused feature sequence is determined, and a mask matrix is ​​constructed as shown in the following formula (6) for specific representation.

[0103] , ( (6).

[0104] in, It is a mask matrix; To allow for focused attention on specific locations.

[0105] In the mask matrix, the position corresponding to the allowed interest set is determined as an allowed value, i.e., 0, and other non-corresponding positions are determined as disallowed values, i.e., negative infinity.

[0106] Understandably, the mask matrix is ​​the same size as the allowed attention set matrix, and each position corresponds one-to-one. The values ​​at each position in the mask matrix are updated based on the values ​​at each position in the allowed attention set. Within the image patch embedding sequence, the mask matrix allows mutual attention between image patches to learn the spatial structure features within the image. Within the text tag embedding sequence, the mask matrix allows mutual attention between text tags to learn the contextual features within the text. Between the image patch embedding sequence and the text tag embedding sequence, the mask matrix allows mutual attention between heterogeneous image and text data to learn the semantic correspondences between medical image and text heterogeneous data. Between the label embedding sequence and the image patch embedding sequence and the text tag embedding sequence, the mask matrix allows labels to pay attention to image patches and text tags, enabling label-guided image and text feature learning.

[0107] In this implementation, by pre-setting the allowed attention set for each position, the information flow is precisely controlled, clearly specifying which positions can follow each other and which are prohibited, avoiding unreasonable cross-modal or cross-positional information interference. Based on this, the allowed and disallowed values ​​in the mask matrix are clearly defined, avoiding invalid calculations and reducing the computational complexity of the attention mechanism. Thus, by restricting the information flow, the enhanced features of each position focus more on the relevant semantics within its allowed attention range, reducing noise from irrelevant modalities, thereby obtaining a clearer and more discriminative local representation.

[0108] Secondly, the attention layer based on the mask matrix performs multi-layer mask attention processing on the fused feature sequence through self-learning and interactive learning. Within the allowed attention range, the sequence features at each position in the fused feature sequence learn the semantic information of other modalities, thus obtaining an enhanced feature sequence.

[0109] In one implementation, a multi-layer masked attention mechanism is used to learn fine-grained semantic relationships between different modalities such as images, text, and labels within the same sample, while simultaneously preserving the contextual structure within each modality. For the first... layer( The process involves performing linear projection, mask attention calculation, residual and normalization processing, and feedforward propagation to obtain the enhanced feature sequence. Specifically, it includes the following four steps.

[0110] First, linear projection is performed on the fused feature sequence to obtain the first query vector, the first key vector and the first value vector, which are represented by the following formulas (7) to (9).

[0111] (7).

[0112] (8).

[0113] (9).

[0114] in, For sample i in the th The first query vector of the layer; This is the first key vector; It is the first value vector; This is the fusion feature sequence of the previous layer; , , This is the weight matrix.

[0115] Secondly, based on the first query vector and the first key vector, the original relevance score between each position of the fused feature sequence is determined; based on the weighted sum of the mask matrix, the original relevance score and the first value vector, the mask attention result is obtained, as specifically represented by the following formula (10).

[0116] (10).

[0117] in, For sample i in the th Layer mask attention results; This is the first query vector; This is the first key vector; It is the first value vector; It is a mask matrix.

[0118] Next, the mask attention result and the feature residual of the previous layer are subjected to residual connection and layer normalization to obtain the intermediate feature sequence, which is specifically represented by the following formula (11).

[0119] (11).

[0120] in, The fused feature sequence of sample i in the previous layer; For layer normalization processing; For sample i in the th Intermediate feature sequences of the layer.

[0121] Finally, the sequence features at each position in the intermediate feature sequence are subjected to independent nonlinear transformations, and then residual connections and layer normalization are performed with the intermediate feature sequence to obtain the enhanced feature sequence, as specifically represented by the following formula (12).

[0122] (12).

[0123] in, For sample i in the th The layer outputs an enhanced feature sequence; It serves as a feedforward network layer to learn complex nonlinear relationships.

[0124] In this implementation, multi-layer attention allows the model to progressively fuse cross-modal information from shallow to deep layers. Each layer extracts higher-order semantic features based on the previous layer, resulting in a more expressive enhanced feature sequence. During multi-layer processing, a mask matrix controls the information flow within and between modalities, preserving both the spatial structure of the image and the temporal structure of the text, while progressively strengthening the fine-grained semantic correspondence between image, text, and label. Simultaneously, the masking mechanism limits the scope of attention at each position within each layer, preventing the indiscriminate fusion of full-sequence information across all positions. This maintains the specificity of each position during multi-layer iterations and mitigates the oversmoothing problem common in fully connected self-attention.

[0125] S23 employs a star-shaped attention mechanism to determine concept basis coefficient features and global semantic features based on enhanced feature sequences and multiple learnable weight matrices.

[0126] Among them, the multiple learnable weight matrices include a learnable first weight matrix, a learnable second weight matrix, and a learnable third weight matrix.

[0127] In one implementation, conceptual basis coefficient features and global semantic features of a sample are learned based on a star-shaped attention mechanism. Multiple conceptual basis vectors are learned from the entire training dataset, defining the fundamental dimensions of the semantic space. For each sample, coefficients on multiple conceptual basis vectors are computed by stacking multiple layers of star-shaped attention, thereby representing the sample as a linear combination of conceptual basis vectors. Figure 3 As shown, it specifically includes the following four steps.

[0128] S231, initialize multiple learnable concept basis vectors.

[0129] Assume all samples share A learnable concept basis vector, denoted as the concept basis matrix, is specifically represented by the following formula (13).

[0130] (13).

[0131] in, For the first ( There are 10 concept basis vectors. The concept basis vectors are initialized using a Xavier uniform distribution. .

[0132] S232, Determine the initial query vector based on the enhanced feature sequence.

[0133] For the Enhanced feature sequences from multimodal interaction fusion of individual samples The average along the sequence dimension is used as the initial query vector, as specifically represented by the following formula (14).

[0134] (14).

[0135] in, for The enhanced feature sequence Line, i.e., the first Embedding vectors of each feature unit; The length of the fused feature sequence; This is the initial query vector, serving as the initial representation of the sample in the current semantic space.

[0136] S233, perform multi-level iterative refinement on the initial query vector to obtain the refined query vector and concept basis coefficient features.

[0137] The topological relationship of the star-shaped structure layers is as follows: Each learnable concept basis vector serves as the central node, and the sample... of Each enhanced feature sequence unit serves as a surrounding node, and each central node is connected to all surrounding nodes. Surrounding nodes are not directly connected to each other, and central nodes are not directly connected to each other.

[0138] For the layer( The process involves calculating concept basis coefficient features, linear projection, star-shaped attention calculation, residual and normalization processing, and feedforward propagation to learn concept basis coefficient features and global semantic representation. Specifically, the following two steps represent the process.

[0139] S2331, calculate the conceptual basis coefficient characteristics of the sample.

[0140] Based on the query vector of the previous layer and multiple learnable concept basis vectors, determine the concept basis coefficient features of the current layer.

[0141] Specifically, the query vector at the previous layer As a query vector, with the concept basis matrix Given a key vector, calculate the sample query vector With concept basis matrix similarity The conceptual basis coefficient characteristics of the current layer are specifically represented by the following formula (15).

[0142] (15).

[0143] in, This is the query vector for the previous layer; It consists of multiple learnable concept basis vectors, i.e., the concept basis matrix; The basic coefficients are the characteristics of the concept; the Softmax(·) function is the normalized exponential function.

[0144] S2332 aggregates star-shaped attention information based on concept coefficient features.

[0145] The main steps involved are linear projection, star-shaped attention calculation, residual and normalization processing, and feedforward propagation, which are characterized by the following four steps.

[0146] Firstly, linear projection.

[0147] In one implementation, a query vector is obtained by linearly projecting multiple learnable concept basis vectors using a first weight matrix; a key vector is obtained by linearly projecting an enhanced feature sequence using a second weight matrix; and a value vector is obtained by linearly projecting an enhanced feature sequence using a third weight matrix.

[0148] Specifically, the concept basis matrix C and the sample The Layer fusion sequence features The query vector, key vector, and value vector are generated through linear projection. Specifically, they are represented by the following formulas (16) to (18).

[0149] (16).

[0150] (17).

[0151] (18).

[0152] in, For query vector; The key vector; It is a value vector; , and Here, are the learnable weight matrices; C is the concept basis matrix; For the sample The fusion sequence features.

[0153] Secondly, star-shaped attention calculation.

[0154] Based on the concept basis coefficients of the current layer, star-shaped attention aggregation is performed on the concept query matrix, key matrix, and value matrix to obtain the query increment.

[0155] Specifically, each concept base reads the information most relevant to its own semantics from the samples. Based on the samples... Activation coefficients of concept bases As a priori guide for attention weights, the weighted aggregation information of the current layer is calculated to obtain the query increment of the current layer. The following formula (19) specifically represents this.

[0156] (19).

[0157] in, For the sample In the Incremental query volume at each layer; For the sample In the The first layer coefficients on a concept basis And satisfy =1; For query vector; The key vector; It is a value vector.

[0158] Third, residual connection correction and layer normalization.

[0159] The query increment is joined with the query vector of the previous layer using a residual connection and then normalized to obtain an intermediate normalized representation.

[0160] Specifically, the incremental query information read by the current layer With the query vector of the previous layer By adding the residual connections, historical information is preserved, and after layer normalization, an intermediate normalized representation is obtained, as specifically represented by the following formula (20).

[0161] (20).

[0162] in, For the sample In the The intermediate normalized representation of the layer; For the sample In the Incremental query volume at each layer; For the sample The query vector at the previous level.

[0163] Fourth, feedforward networks.

[0164] The intermediate normalized representation is input into the feedforward network to obtain the feedforward output; the feedforward output is then residually connected to the intermediate query representation and layer normalized to obtain the query vector of the current layer.

[0165] Specifically, complex nonlinear relationships are learned through feedforward network layers, residual information is added, and this information is passed forward after normalization to obtain the first... The query vector output of the layer is specifically represented by the following formula (21).

[0166] (twenty one).

[0167] in, For the sample In the The query vector of the layer; For the sample In the The intermediate normalized representation of the layer; It is a feedforward network layer.

[0168] Thus, after After iterating through the star-shaped attention layer, the query vector is obtained. It has been gradually refined from the initial mean vector into a representation rich in sample semantic information, the coefficient vector. The process of refining the data layer by layer accurately reflects the activation intensity of each conceptual basis in the sample.

[0169] S234 generates global semantic features based on the refined query vector and concept basis coefficient features.

[0170] Specifically, from the sample Refined query vector The sample is obtained by superimposing a weighted linear combination of the conceptual bases. The global semantic features are specifically represented by the following formula (22).

[0171] (twenty two).

[0172] in, For the sample Global semantic features; For the sample In the The query vector of the layer; For the sample In the The first layer Coefficients on a conceptual basis; For the first A conceptual basis vector.

[0173] S24. Construct a total loss function based on the concept basis coefficient features and global semantic features; with the goal of minimizing the total loss function, iteratively update multiple learnable parameters to obtain multiple target parameters.

[0174] Among them, the learnable parameters include a pre-defined encoder, multiple learnable weight matrices, and multiple learnable concept basis vectors.

[0175] The total loss function includes multi-label classification loss and concept alignment loss.

[0176] In one implementation, the total loss function is constructed based on the conceptual basis coefficient features and global semantic features, as specifically represented by the following formulas (23) to (25).

[0177] The total loss function is obtained by weighting the multi-label classification loss and the concept alignment loss, as shown in the following formula (23).

[0178] (twenty three).

[0179] in, This is the total loss function; For multi-label classification loss function; For concept alignment loss function; It is the balance coefficient, and .

[0180] First, construct a multi-label classification loss.

[0181] A multi-label classification loss is constructed based on the maximum similarity between the global semantic features of the medical sample and multiple positive label embedding vectors, and the minimum similarity between the global semantic features of the medical sample and multiple negative label embedding sequences.

[0182] Among them, the positive label embedding vector represents the label embedding vector corresponding to the medical sample. The negative label embedding vector represents the label embedding vector that does not correspond to the medical sample.

[0183] To encourage a sample's concept vector to generally be close to all its positive labels while being far away from all other labels, a multi-label classification loss function is defined.

[0184] Specifically, the similarity between the global semantic features of the sample and each positive label embedding vector possessed by the sample is calculated. The similarity is divided by the temperature parameter and the exponent is taken. The sum of all concept vectors of the sample is then obtained to obtain the first matching score between the sample and the positive label. The similarity between the global semantic features of the sample and each label embedding vector of all possible labels is calculated. The similarity is divided by the temperature parameter and the exponent is taken. The sum of all concept vectors and all possible labels of the sample is then obtained to obtain the second matching score between the sample and all labels. The first matching score is divided by the second matching score to obtain the vote share of the label among all labels. The negative logarithm of the vote share of all positive labels of the sample is taken and then summed. The average is then taken over all samples in the entire batch to obtain the multi-label classification loss. The following formula (24) specifically represents this.

[0185] (twenty four).

[0186] in, For multi-label classification loss function; It is the collection of all tags; For the sample The collection of all tags; For tags The embedding vector; For tags Embedded vector, Here, is the temperature control parameter, and sim(.) is the cosine similarity.

[0187] Secondly, construct the concept alignment loss.

[0188] A concept alignment loss is constructed based on the similarity between each concept feature vector in the concept basis coefficient features of the medical sample and multiple positive label embedding vectors.

[0189] Specifically, for each sample in the training set, the model calculates the coefficient distribution of that sample across multiple concept bases. A larger coefficient indicates that the sample relies more heavily on the corresponding concept base to describe its semantic features. Simultaneously, for each concept base, the function finds the label most semantically similar to that concept base among all the positive labels the sample possesses, and calculates their cosine similarity. These two values ​​are then multiplied to obtain the "alignment contribution" of that concept base to the current sample. The concept alignment loss is obtained by summing the values ​​of each concept basis, averaging them over all samples, and finally adding a negative sign. The following formula (25) specifically represents this loss.

[0190] (25).

[0191] in, For concept alignment loss function; The number of feature vectors of a concept; It is a sample In the coefficients on a concept basis For the first A conceptual basis vector, To maximize the value, we select the label that is most similar to the concept base from all the positive labels of the sample.

[0192] In one implementation, multiple learnable parameters are iteratively updated with the goal of minimizing the total loss function to obtain multiple target parameters.

[0193] The parameters of the model are learned using gradient descent. Gradient descent updates multiple learnable parameters, including those of the encoder, conceptual basis, and attention mechanism, with the goal of minimizing the total loss function. This process involves the following three steps.

[0194] First, initialize multiple learnable parameters.

[0195] When the iteration number t=0, initialize all learnable parameters. .

[0196] Secondly, based on the initialized learnable parameters, determine the partial derivatives of the total loss function with respect to the initialized learnable parameters.

[0197] Specifically, the target total loss function is determined according to the above formula (23). and target total loss function For parameters The partial derivative, i.e. .

[0198] Third, based on the initialized learnable parameters and partial derivatives, the learnable parameters are iteratively updated until the preset conditions are met, resulting in multiple target parameters, specifically represented by the following formula (26).

[0199] (26).

[0200] in, This is the partial derivative of the target total loss function with respect to the parameters; These are learnable parameters; This is the learning rate.

[0201] The preset condition is that the objective function converges or the number of iterations t exceeds a threshold.

[0202] In this implementation, the learnable parameters in the attention mechanism are iteratively updated with the goal of minimizing the total loss function. This enables multiple learnable parameters to automatically learn better feature representations from multimodal data, thereby improving the accuracy and discriminative ability of subsequent local and global semantic features.

[0203] S25, update the model parameters of the preset model to multiple target parameters to obtain the target model.

[0204] Understandably, according to steps S21 to S24 above, the preset model is trained using medical images, free report text, and medical label information to obtain the target model. Among them, medical label information is used as a supervision constraint to guide the latent concepts to learn the correct pathological direction. At the same time, the concept basis coefficient features and global semantic features of positive labels in medical images and free report text are brought into the loss function, thereby iteratively updating the learnable parameters in the attention mechanism to obtain multiple target parameters and complete the training of the model.

[0205] In one implementation, a target model is used to perform the following processing on the query object to obtain the target retrieval results. Specifically, this includes the following five steps.

[0206] S251, Update the feature library of the dataset to be retrieved.

[0207] Using the model parameters trained in the target model, update the conceptual basis coefficient features and global semantic features of all medical samples in the dataset to be retrieved, and update the feature index library in the target model.

[0208] S252, determine the target concept base coefficient features and target global semantic features of the query object.

[0209] For the input query object, its conceptual basis coefficient features and global semantic features are calculated using the trained model and parameters. For image queries, where the input is only an image, the masked self-attention module degenerates into self-attention between image patches. For text queries, where the input is only text, the masked self-attention module degenerates into self-attention between text terms.

[0210] Specifically, if the query object is determined to be a query image, the target retrieval result is a text report that matches the query image; if the query object is determined to be a query text report, the target retrieval result is a medical image that matches the query text report.

[0211] S253, Coarse-grained retrieval. Based on the coarse-grained scores of the global semantic features of the query object and the global semantic features of multiple medical samples, multiple candidate samples whose coarse-grained scores meet a first preset threshold are selected from the multiple medical samples.

[0212] The cosine similarity between the global semantic features of the query image and the global semantic features of all medical samples in the feature index library of the target model is calculated to obtain a coarse-grained score, which is then used to select the top K candidate reports with the highest similarity.

[0213] S254, Fine-grained matching. A fine-grained score is obtained based on the similarity between the target concept basic coefficient features of the query object and the concept basic coefficient features of multiple candidate samples.

[0214] Specifically, for K candidate samples, fine-grained matching is performed. That is, the cosine similarity score between the concept basis coefficient features of the query object and the concept basis coefficient features of the candidate samples is further calculated.

[0215] S255. Based on the weighted fusion of the coarse-grained score and the fine-grained score of each candidate sample, the final score of each candidate sample is obtained; the candidate sample with the highest final score is determined as the target retrieval result.

[0216] The coarse search score and the fine-grained score are weighted and fused to obtain the final score, which is then reordered and returned as query results.

[0217] In this implementation, by comparing the conceptual similarity between the query target and the medical sample, the target retrieval results are filtered out, realizing a semantic matching-based retrieval mechanism that can more accurately locate the sample most relevant to the query target.

[0218] Figure 4 This is a schematic diagram of a medical image and text heterogeneous data retrieval device provided in this application. For example... Figure 4 The medical image and text heterogeneous data retrieval device 50 includes: a first processor 501, a communication bus 502, a memory 503, a communication interface 504, an output device 505, an input device 506, and a second processor 507.

[0219] The medical image and text heterogeneous data retrieval device 50 may include at least one first processor 501 and a memory 503 for storing processor-executable instructions. The first processor 501 is configured to execute the instructions in the memory 503 to implement the medical image and text heterogeneous data retrieval method in the following embodiments.

[0220] In addition, the medical image and text heterogeneous data retrieval device 50 may also include a communication bus 502, at least one communication interface 504, an input device 506, and an output device 505.

[0221] The first processor 501 may be a processor (central processing unit, CPU), a microprocessor unit, an ASIC, or one or more integrated circuits for controlling the execution of programs according to the present application.

[0222] The communication bus 502 may include a path for transmitting information between the aforementioned components.

[0223] Communication interface 504 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0224] Input device 506 is used to receive input signals and output device 505 is used to output signals.

[0225] Memory 503 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital universal discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory may exist independently and be connected to the processing unit via a bus. Memory may also be integrated with the processing unit.

[0226] The memory 503 stores instructions for executing the scheme of this application, and the execution is controlled by the first processor 501. The first processor 501 executes the instructions stored in the memory 503 to realize the functions of the method of this application.

[0227] In a specific implementation, as one example, the first processor 501 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 in the CPU.

[0228] In a specific implementation, as one example, the medical image and text heterogeneous data retrieval device 50 may include multiple processors, such as... Figure 4 The first processor 501 and the second processor 507 are described. Each of these processors can be a single-core processor or a multi-core processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0229] This medical image and text heterogeneous data retrieval device, such as Figure 4 The diagram includes a first processor 501 and a memory 503 for storing executable instructions of the first processor 501. The first processor 501 is configured to execute the executable instructions to implement the medical image-text heterogeneous data retrieval method as described in any of the possible embodiments above. Furthermore, it achieves the same technical effect, and to avoid repetition, it will not be described again here.

[0230] This application also provides a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by the processor of a medical image and text heterogeneous data retrieval device, the medical image and text heterogeneous data retrieval device is able to perform the medical image and text heterogeneous data retrieval method as described in any of the above possible implementations. And it can achieve the same technical effect; to avoid repetition, it will not be described again here.

[0231] This application also provides a computer program product, including a computer program or instructions, which are executed by a processor as a medical image and text heterogeneous data retrieval method according to any of the possible implementations described above. It achieves the same technical effects, and to avoid repetition, will not be described again here.

[0232] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0233] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for retrieving heterogeneous medical image and text data, characterized in that, The method includes: Using a target model, the following processing is performed on the query object to obtain the target retrieval results: The target conceptual basis coefficient features and target global semantic features of the query object are determined. Based on the similarity between the target conceptual basis coefficient features and the target global semantic features of the query object and the conceptual basis coefficient features and global semantic features of all medical samples in the dataset to be retrieved associated with the target model, respectively, the target retrieval results matching the query object are obtained; the query object includes querying medical images or querying report text. The target model is trained using a preset model by the following operations: extracting embedding sequences of multiple modalities from medical samples; concatenating and fusing the embedding sequences of the multiple modalities to obtain a fused feature sequence; the multiple modalities include medical images, free report text, and medical label information; and performing masked attention aggregation on the fused feature sequence according to the preset information flow direction of the sequence features at each position in the fused feature sequence to obtain an enhanced feature sequence. A star-shaped attention mechanism is used to determine the concept basis coefficient features and the global semantic features based on the enhanced feature sequence and multiple learnable weight matrices. Based on the concept basis coefficient features and the global semantic features, a total loss function is constructed; with the goal of minimizing the total loss function, the plurality of learnable parameters are iteratively updated to obtain the plurality of target parameters; the plurality of learnable parameters include the plurality of learnable weight matrices and the plurality of learnable latent concept vectors; The model parameters of the preset model are updated to the target parameters to obtain the target model.

2. The medical image and text heterogeneous data retrieval method according to claim 1, characterized in that, The extracted embedding sequences of multiple modalities from the medical sample include: Based on a preset encoder, the embedding vectors of local units in each modality of the medical sample are extracted to obtain the multimodal embedding sequence of the medical sample; the preset encoder includes a medical image encoder, a text report encoder, and a label encoder; the multimodal embedding sequence includes an image patch embedding sequence, a text embedding sequence, and a label embedding sequence; Specifically, according to the medical image encoder, the embedding vector of each uniform image patch in the medical image of the medical sample in a preset dimension is extracted to obtain the image patch embedding sequence containing image spatial context information; And, according to the text report encoder, the embedding vector of each word in the free report text of the medical sample in the preset dimension is extracted to obtain the text embedding sequence containing text context information; And, based on the label encoder, the embedding vector of the medical label information of the medical sample in the preset dimension is extracted to obtain the label embedding sequence; The image patch embedding sequence, the text embedding sequence, and the label embedding sequence are concatenated and fused to obtain the fused feature sequence.

3. The medical image and text heterogeneous data retrieval method according to claim 2, characterized in that, The location is represented by the embedding vector of the local unit of each modality in the fused feature sequence; the preset information flow direction is used to control each location to learn the semantic information of other locations within the allowed flow direction range; the enhanced feature sequence represents the feature sequence of each location that has fused cross-modal interaction information within the allowed range of the preset information flow direction. The step of performing masked attention aggregation on the fused feature sequence according to the preset information flow direction of the sequence features at each position in the fused feature sequence to obtain the enhanced feature sequence includes: The fused feature sequence is divided into three consecutive index intervals, including an image index interval, a text index interval, and a label index interval; Based on a preset set of allowed attention, the preset information flow direction of the sequence features at each position of the fused feature sequence is determined, and a mask matrix is ​​constructed. The set of allowed attention is used to limit the index range of the target key positions that the query object is allowed to focus on, with the sequence features at the target position as the query object. The positions in the mask matrix that correspond to the set of allowed attention are determined as allowed values, and other non-corresponding positions are determined as disallowed values. Wherein, when the query object of the allowed attention set is a sequence feature of the image index interval, the target key position is determined to be a sequence feature in the image index interval and the text index interval; When the query object of the allowed attention set is a sequence feature of the text index interval, the target key position is determined to be a sequence feature in the image index interval and the text index interval; When the query object of the allowed attention set is a sequence feature of the tag index interval, the target key position is determined to be a sequence feature in the image index interval, the text index interval, and the tag index interval.

4. The medical image and text heterogeneous data retrieval method according to claim 3, characterized in that, The method further includes: Linear projection is performed on the fused feature sequence to obtain a first query vector, a first key vector, and a first value vector; Based on the first query vector and the first key vector, determine the original relevance score between each position of the fused feature sequence; based on the mask matrix, the weighted sum of the original relevance scores and the first value vector, obtain the mask attention result; The mask attention result is coupled with the feature residual of the previous layer and subjected to residual concatenation and layer normalization to obtain the intermediate feature sequence; The sequence features at each position in the intermediate feature sequence are subjected to independent nonlinear transformations, and then residual connections and layer normalization are performed with the intermediate feature sequence to obtain the enhanced feature sequence.

5. The medical image and text heterogeneous data retrieval method according to any one of claims 1 to 4, characterized in that, The plurality of learnable weight matrices include a learnable first weight matrix, a learnable second weight matrix, and a learnable third weight matrix; The star-shaped attention mechanism is employed to determine the concept basis coefficient features and the global semantic features based on the enhanced feature sequence and multiple learnable weight matrices, including: Initialize multiple learnable concept basis vectors; and determine the initial query vector based on the enhanced feature sequence; The initial query vector is refined through multiple iterations to obtain a refined query vector and the concept basis coefficient features; The global semantic features are generated based on the refined query vector and the concept basis coefficient features; Each iteration of the multi-level iterative refinement of the initial query vector includes: Based on the query vector of the previous layer and the multiple learnable concept basis vectors, determine the concept basis coefficient features of the current layer; The query vector is obtained by linearly projecting the plurality of learnable concept basis vectors using the learnable first weight matrix; the key vector is obtained by linearly projecting the enhanced feature sequence using the learnable second weight matrix; and the value vector is obtained by linearly projecting the enhanced feature sequence using the learnable third weight matrix. Based on the concept basis coefficient features of the current layer, star-shaped attention aggregation is performed on the query vector, the key vector, and the value vector to obtain the query increment; The query increment is joined with the query vector of the previous layer by residual connection and layer normalization to obtain an intermediate normalized representation; The intermediate normalized representation is input into the feedforward network to obtain the feedforward output; the feedforward output and the intermediate normalized representation are then residually connected and layer normalized to obtain the query vector of the current layer.

6. The method for retrieving heterogeneous medical image and text data according to claim 5, characterized in that, The total loss function includes multi-label classification loss and concept alignment loss; The step of constructing the total loss function based on the concept basis coefficient features and the global semantic features includes: The multi-label classification loss is constructed based on the maximum similarity between the global semantic features of the medical sample and multiple positive label embedding vectors, and the minimum similarity between the global semantic features of the medical sample and multiple negative label embedding vectors. Furthermore, the concept alignment loss is constructed based on the similarity between each concept base feature vector in the concept base coefficient features of the medical sample and the plurality of positive label embedding vectors; The total loss function is obtained by weighting the multi-label classification loss and the concept alignment loss.

7. The medical image and text heterogeneous data retrieval method according to claim 6, characterized in that, The process of iteratively updating the plurality of learnable parameters with the objective of minimizing the total loss function yields the plurality of target parameters, including: Initialize the plurality of learnable parameters; determine the partial derivative of the total loss function with respect to the initialized learnable parameters based on the initialized learnable parameters; Based on the initialized learnable parameters and the partial derivatives, the learnable parameters are iteratively updated until a preset condition is met, thereby obtaining the multiple target parameters.

8. The method for retrieving heterogeneous medical image and text data according to claim 1, characterized in that, The step of obtaining target retrieval results matching the query object by comparing the similarity between the target concept basis coefficient features and the target global semantic features of the query object and the concept basis coefficient features and global semantic features of all medical samples in the retrieval dataset associated with the target model, respectively, includes: Based on the coarse-grained scores of the target global semantic features of the query object and the global semantic features of multiple medical samples, multiple candidate samples whose coarse-grained scores meet a first preset threshold are selected from the multiple medical samples. A fine-grained score is obtained based on the similarity between the target concept basic coefficient features of the query object and the concept basic coefficient features of the multiple candidate samples; The final score for each candidate sample is obtained by weighted fusion of the coarse-grained score and the fine-grained score; the candidate sample with the highest final score is determined as the target retrieval result. Specifically, the query object is determined to be a query image, and the target retrieval result is a text report that matches the query image; the query object is determined to be a query text report, and the target retrieval result is a medical image that matches the query text report.

9. A medical image-text heterogeneous data retrieval system, characterized in that, The system includes: The feature sequence construction module is configured to extract embedding sequences of multiple modalities from medical samples; and to concatenate and fuse the embedding sequences of the multiple modalities to obtain a fused feature sequence; the multiple modalities include medical images, free report text, and medical label information; The masked self-attention module is configured to perform masked attention aggregation on the fused feature sequence according to a preset information flow direction of the sequence features at each position in the fused feature sequence to obtain an enhanced feature sequence; The star-shaped attention module is configured to use a star-shaped attention mechanism to determine the concept basis coefficient features and the global semantic features based on the enhanced feature sequence and multiple learnable weight matrices. The target model determination module is configured to construct a total loss function based on the concept basis coefficient features and the global semantic features; to iteratively update the plurality of learnable parameters with the goal of minimizing the total loss function, thereby obtaining the plurality of target parameters; the plurality of learnable parameters include the plurality of learnable weight matrices and the plurality of learnable latent concept vectors; and to update the model parameters of the preset model to the target parameters, thereby obtaining the target model. The retrieval module is configured to use a target model to perform the following processing on the query object to obtain the target retrieval result: determine the target conceptual basis coefficient features and target global semantic features of the query object; based on the similarity between the target conceptual basis coefficient features and the target global semantic features of the query object and the conceptual basis coefficient features and global semantic features of all medical samples in the dataset to be retrieved associated with the target model, obtain the target retrieval result matching the query object; the query object includes querying medical images or querying report text; The system is configured to perform the medical image and text heterogeneous data retrieval method as described in any one of claims 1-8.

10. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the medical image and text heterogeneous data retrieval method as described in any one of claims 1-8.

Images