Medical image semantic extraction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a medical image semantic extraction method and device, electronic equipment and a storage medium, belongs to the technical field of artificial intelligence, and is suitable for the field of medical science and technology. The method comprises the following steps: acquiring an original medical image and an image annotation text of the original medical image; performing feature extraction on the original medical image to obtain medical image features; performing feature extraction on the image annotation text to obtain annotation text features; performing cross-modal fusion based on the medical image features and the annotation text features to obtain cross-modal fusion features; and performing semantic fusion on the medical image features and the annotation text features based on the cross-modal fusion features to obtain target semantic features. According to the embodiment of the invention, the accuracy of medical image semantic extraction can be improved.

Description

Medical image semantic extraction methods and devices, electronic devices and storage media Technical Field

[0001] This application relates to the field of artificial intelligence technology, and is applicable to the field of medical technology, particularly to a method and apparatus for semantic extraction of medical images, an electronic device, and a storage medium. Background Technology

[0002] Image semantic extraction is a computer vision technique that enables the identification and understanding of high-level semantic information from images, such as object categories, locations, outlines, and contextual relationships, to obtain descriptions that are intuitively perceptible to humans. Image semantic extraction technology is widely used in smart healthcare scenarios to extract information from various medical images, such as CT scans, MRI scans, X-rays, and pathological slides. For example, it allows for precise localization of lesions, helping medical staff quickly determine the location, size, and morphological characteristics of lesions; and it enables in-depth analysis of digital pathological slides, accurately identifying different cell types such as cancer cells and inflammatory cells, and detailing their distribution density and spatial relationships.

[0003] Currently, pre-trained deep learning models are mainly used to extract semantics from medical images, deriving the corresponding structured information. However, in real-world applications, the data is not single-modal; in addition to medical images, it often includes other modalities such as annotation text. Existing deep learning models typically employ pre-defined and fixed fusion strategies when processing multimodal data, such as simple feature concatenation or weighted averaging. This fails to dynamically extract semantics based on the quality, importance, and complex relationships between different modalities, thus affecting the accuracy of semantic extraction from medical images.

[0004] Therefore, improving the accuracy of semantic extraction from medical images has become an urgent technical problem to be solved. Summary of the Invention

[0005] The main objective of this application is to propose a method, apparatus, electronic device, and storage medium for semantic extraction of medical images, aiming to solve the technical problem of inaccurate semantic extraction caused by the inability to extract semantics based on the quality, importance, and complex relationships between various modal information, thereby improving the accuracy of semantic extraction of medical images.

[0006] To achieve the above objectives, a first aspect of this application proposes a method for semantic extraction of medical images. The method includes: acquiring an original medical image and image annotation text of the original medical image; performing feature extraction on the original medical image to obtain medical image features; performing feature extraction on the image annotation text to obtain annotation text features; performing cross-modal fusion based on the medical image features and the annotation text features to obtain cross-modal fusion features; and performing semantic fusion on the medical image features and the annotation text features based on the cross-modal fusion features to obtain target semantic features.

[0007] In some embodiments, the cross-modal fusion feature includes a first fusion feature and a second fusion feature; the step of semantically fusing the medical image feature and the annotation text feature based on the cross-modal fusion feature to obtain the target semantic feature includes: performing similarity calculation based on the first fusion feature and the second fusion feature to obtain fusion feature similarity data; and performing semantic fusion on the medical image feature and the annotation text feature based on the fusion feature similarity data to obtain the target semantic feature.

[0008] In some embodiments, the step of semantically fusing the medical image features and the annotation text features based on the fusion feature similarity data to obtain the target semantic features includes: if the fusion feature similarity data is in a first preset interval, obtaining the attention weight of the medical image features based on a preset weighted fusion strategy to obtain image attention weight, obtaining the attention weight of the annotation text features to obtain text attention weight; and performing weighted fusion of the medical image features and the annotation text features based on the image attention weight and the text attention weight to obtain the target semantic features.

[0009] In some embodiments, the step of semantically fusing the medical image features and the annotation text features based on the fusion feature similarity data to obtain the target semantic feature includes: if the fusion feature similarity data is in a second preset interval, performing intersection processing on the medical image features and the annotation text features based on a preset intersection fusion strategy to obtain intersection semantic features; and confirming the intersection semantic features as the target semantic feature.

[0010] In some embodiments, the step of semantically fusing the medical image features and the annotation text features based on the fusion feature similarity data to obtain the target semantic feature includes: if the fusion feature similarity data is within a third preset interval, performing union processing on the medical image features and the annotation text features based on a preset union fusion strategy to obtain a union semantic feature; and confirming the union semantic feature as the target semantic feature.

[0011] In some embodiments, the step of calculating similarity based on the first fusion feature and the second fusion feature to obtain fusion feature similarity data includes: calculating cosine similarity between the first fusion feature and the second fusion feature to obtain the fusion feature similarity data; or, calculating mutual information between the first fusion feature and the second fusion feature to obtain the fusion feature similarity data; or, performing similarity discrimination on the first fusion feature and the second fusion feature through a pre-trained similarity discrimination network to obtain the fusion feature similarity data.

[0012] In some embodiments, the cross-modal fusion based on the medical image features and the annotation text features to obtain cross-modal fused features includes: performing attention calculation using the medical image features as query features and the annotation text features as key and value features to obtain the first fused feature; and performing attention calculation using the annotation text features as query features and the medical image features as key and value features to obtain the second fused feature.

[0013] To achieve the above objectives, a second aspect of this application provides a medical image semantic extraction device, the device comprising: a data acquisition module for acquiring an original medical image and image annotation text of the original medical image; an image feature extraction module for extracting features from the original medical image to obtain medical image features; a text feature extraction module for extracting features from the image annotation text to obtain annotation text features; a cross-modal fusion module for performing cross-modal fusion based on the medical image features and the annotation text features to obtain cross-modal fusion features; and a semantic fusion module for performing semantic fusion based on the cross-modal fusion features on the medical image features and the annotation text features to obtain target semantic features.

[0014] To achieve the above objectives, a third aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect.

[0015] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0016] The medical image semantic extraction method, apparatus, electronic device, and storage medium proposed in this application acquire the original medical image and its accompanying text annotations. They then extract features from both the original medical image and the annotation text separately to obtain corresponding medical image features and annotation text features. This allows for accurate capture of key visual information in the image and extraction of the core semantics of the text. Next, cross-modal fusion is performed based on the medical image features and annotation text features to obtain cross-modal fusion features, breaking down the barriers between image and text modalities and integrating the advantages of different modalities. Finally, semantic fusion is performed on the medical image features and annotation text features based on the cross-modal fusion features to obtain target semantic features. This allows for deeper exploration of the inherent semantic relationships between the image and text, accurately and comprehensively reflecting medical image information. It solves the technical problem of inaccurate semantic extraction caused by the inability to extract semantics based on the quality, importance, and complex relationships between different modalities, thus improving the accuracy and reliability of medical image semantic extraction. Attached Figure Description

[0017] Figure 1 is a flowchart of the medical image semantic extraction method provided in the embodiment of this application; Figure 2 is a flowchart of step S104 in Figure 1; Figure 3 is a flowchart of step S105 in Figure 1; Figure 4 is a flowchart of step S302 in Figure 3; Figure 5 is another flowchart of step S302 in Figure 3; Figure 6 is another flowchart of step S302 in Figure 3; Figure 7 is a schematic diagram of the structure of the medical image semantic extraction device provided in the embodiment of this application; Figure 8 is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0021] First, let's clarify some terms used in this application: Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. AI also refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0022] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). It is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0023] Image semantic extraction is a computer vision technique that enables the identification and understanding of high-level semantic information from images, such as object categories, locations, outlines, and contextual relationships, to obtain descriptions that are intuitively perceptible to humans. Image semantic extraction technology is widely used in smart healthcare scenarios to extract information from various medical images, such as CT scans, MRI scans, X-rays, and pathological slides. For example, it allows for precise localization of lesions, helping medical staff quickly determine the location, size, and morphological characteristics of lesions; and it enables in-depth analysis of digital pathological slides, accurately identifying different cell types such as cancer cells and inflammatory cells, and detailing their distribution density and spatial relationships.

[0024] Currently, pre-trained deep learning models are mainly used to extract semantics from medical images, deriving the corresponding structured information. However, in real-world applications, the data is not single-modal; in addition to medical images, it often includes other modalities such as annotation text. Existing deep learning models typically employ pre-defined and fixed fusion strategies when processing multimodal data, such as simple feature concatenation or weighted averaging. This fails to dynamically extract semantics based on the quality, importance, and complex relationships between different modalities, thus affecting the accuracy of semantic extraction from medical images.

[0025] Based on this, embodiments of this application provide a method, apparatus, electronic device, and storage medium for semantic extraction of medical images, aiming to solve the technical problem of inaccurate semantic extraction caused by the inability to extract semantics based on the quality, importance, and complex relationships between various modal information, thereby improving the accuracy of semantic extraction of medical images.

[0026] The medical image semantic extraction method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the medical image semantic extraction method in this application embodiment is described.

[0027] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0028] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0029] The medical image semantic extraction method provided in this application relates to the field of artificial intelligence technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the medical image semantic extraction method, but is not limited to the above forms.

[0030] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0031] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0032] Figure 1 is an optional flowchart of a medical image semantic extraction method provided in an embodiment of this application. The method in Figure 1 may include, but is not limited to, steps S101 to S105.

[0033] Step S101: Obtain the original medical image and the image annotation text of the original medical image; Step S102: Extract features from the original medical image to obtain medical image features; Step S103: Extract features from the image annotation text to obtain annotation text features; Step S104: Perform cross-modal fusion based on the medical image features and annotation text features to obtain cross-modal fusion features; Step S105: Perform semantic fusion based on the cross-modal fusion features on the medical image features and annotation text features to obtain target semantic features.

[0034] Steps S101 to S105 of this embodiment involve acquiring the original medical image and its accompanying image annotation text, and extracting features from both to obtain corresponding medical image features and annotation text features. This process accurately captures key visual information in the image and extracts the core semantics of the text. Next, cross-modal fusion is performed based on the medical image features and annotation text features to obtain cross-modal fusion features, breaking down the barriers between image and text modalities and integrating the advantages of different modalities. Finally, semantic fusion is performed on the medical image features and annotation text features based on the cross-modal fusion features to obtain target semantic features. This process can more deeply explore the inherent semantic relationships between the image and text, accurately and comprehensively reflecting medical image information. It solves the technical problem of inaccurate semantic extraction caused by the inability to extract semantics based on the quality, importance, and complex relationships between different modalities, thus improving the accuracy and reliability of medical image semantic extraction.

[0035] In step S101 of some embodiments, the original medical image refers to image data (such as CT, MRI, X-ray, pathological slides, etc.) directly obtained from medical equipment (such as X-ray machines, CT scanners, MRI equipment, etc.) without any processing or analysis. The original medical image contains information such as the structure and morphology of the patient's body parts and is an important basis for medical diagnosis. The image annotation text is a text description added by medical professionals or pre-trained medical recognition models to the original medical image, which may include annotations of lesions in the image, descriptions of lesion features, diagnostic opinions, etc.

[0036] The original medical images provide intuitive visual information, while the image annotation text contains professional knowledge and experience summaries. The combination of the two provides a comprehensive and rich data foundation for subsequent analysis and processing, which helps to understand the image content more accurately and improve the accuracy and reliability of diagnosis.

[0037] For example, the original medical image is a brain MRI image, and the image annotation text is a text report stating "A space-occupying lesion is visible in the left frontal lobe of the patient, with clear borders and mild surrounding edema."

[0038] The original medical image was a chest CT scan, and the image annotation was a text report stating, "A nodule of about 2 cm is visible in the lower lobe of the left lung, with rough edges, which is highly likely to be a malignant tumor."

[0039] In step S102 of some embodiments, specific algorithms and models (such as deep learning models like convolutional neural networks (CNN), residual neural networks (ResNet), or traditional image processing algorithms like edge detection and texture analysis) are used to analyze and process the original medical image, extracting meaningful features to obtain medical image features. This can reduce data dimensionality, reduce computational load, and highlight key information in the image, facilitating subsequent analysis and diagnosis.

[0040] In addition, medical image features have image confidence scores, which are used to characterize the degree of certainty about medical image features.

[0041] For example, taking lung CT images as an example, a convolutional neural network (CNN) model is used to process the image. The CNN model automatically learns features in the image through operations such as convolutional layers and pooling layers. For instance, convolutional layers can detect local features such as edges and textures in the image, while pooling layers can downsample these features, reducing the amount of data. After multiple layers of processing, the model extracts feature vectors that describe the shape (e.g., round, irregular) and texture (e.g., uniform, rough) of lung nodules; these feature vectors are the medical image features.

[0042] In step S103 of some embodiments, natural language processing (NLP) techniques, such as word embedding models (Word2Vec, GloVe, etc.) and pre-trained language models (such as BERT, GPT, etc.), are used to analyze and process the image annotation text, converting words or sentences in the text into numerical vectors, thereby extracting the text features. The extracted text features can retain the semantic information of the text, which helps to understand the image content more accurately and assists in medical diagnosis.

[0043] In addition, annotation text features have text confidence scores, which are used to characterize the degree of certainty about the annotation text features.

[0044] For example, consider the image annotation text "A nodule of approximately 2cm is visible in the lower lobe of the left lung, with rough edges, highly suggestive of malignancy," processed using a BERT pre-trained model. The BERT model maps each word in the text to a high-dimensional word vector, then combines these word vectors to obtain a sentence vector that represents the semantics of the entire text. This sentence vector is the annotation text feature, which can be used for subsequent fusion and analysis with medical image features.

[0045] In step S104 of some embodiments, the cross-modal fusion feature includes a first fusion feature and a second fusion feature, which will be specifically described through the following embodiments.

[0046] Please refer to Figure 2. In some embodiments, step S104 may include, but is not limited to, steps S201 to S202: Step S201, performing attention calculation with medical image features as query features and annotation text features as key and value features to obtain a first fusion feature; Step S202, performing attention calculation with annotation text features as query features and medical image features as key and value features to obtain a second fusion feature.

[0047] Steps S201 to S202, as illustrated in this embodiment, firstly, attention is calculated using medical image features as query features and annotation text features as key and value features. This allows the focus on key text information based on image features, resulting in a first fusion feature that highlights the semantics of the image-related text. Next, attention is calculated using annotation text features as the query and medical image features as the key and value. This allows the focus on key regions of the image based on the text, resulting in a second fusion feature that highlights the semantics of the image-related text. The combination of these two features dynamically learns the semantic association mapping between different modal features, thereby generating the first and second fusion features within a shared semantic subspace.

[0048] In step S201 of some embodiments, medical image features are used as query features, and matching information is searched for in the key features (annotation text features) based on the characteristics of the medical image. By calculating the similarity between the query features and the key features, it is possible to determine which parts of the annotation text are most relevant to specific features of the medical image. Then, the value features (annotation text features) are weighted and summed according to these similarities, and the relevant image semantic information is fused into the medical image features to obtain the first fused feature, thereby softly aligning a certain region in the image with specific words in the text.

[0049] For example, in a lung CT image, only some blurry shadows may be visible from the image itself. However, after attention calculation fusion with the annotation text (such as "There is a ground-glass nodule with a diameter of about 1 cm in the upper lobe of the left lung"), the first fusion feature can clearly identify the lesion type, size and other detailed semantic information corresponding to the specific shadow in the image. This helps to understand and analyze the image more accurately and provides a more reliable basis for subsequent diagnosis and treatment.

[0050] In step S202 of some embodiments, the annotation text features are used as query features, and matching information is searched for in the key features (medical image features) based on the characteristics of the annotation text. By calculating the similarity between the query features and the key features, it is possible to determine which parts of the medical image are most relevant to the specific features of the annotation text. Then, the value features (medical image features) are weighted and summed according to these similarities, and the relevant textual semantic information is fused into the medical image features to obtain the second fused feature, thereby softly aligning a specific word in the text with a certain region in the image.

[0051] For example, a text annotation might describe "enlarged heart," but the degree and specific manifestations of this enlargement cannot be intuitively understood from the text alone. By performing attention-based fusion with medical image features, the second fusion feature can combine visual information such as the actual size and shape of the heart in the image, making the understanding of the description "enlarged heart" more accurate and intuitive, and helping doctors to more comprehensively assess the condition.

[0052] In step S105 of some embodiments, the target semantic features are the key semantic information that is finally obtained, representing the core content jointly expressed by the medical image and the annotation text, which can be used for tasks such as disease diagnosis, disease analysis, and information collection (for building a disease database).

[0053] Please refer to Figure 3. In some embodiments, step S105 may include, but is not limited to, steps S301 to S302: Step S301, performing similarity calculation based on the first fusion feature and the second fusion feature to obtain fusion feature similarity data; Step S302, performing semantic fusion on medical image features and annotation text features based on the fusion feature similarity data to obtain target semantic features.

[0054] Steps S301 to S302, as illustrated in this embodiment, involve calculating similarity based on the first and second fusion features to obtain fusion feature similarity data, which accurately measures the degree of correlation between different features. Next, semantic fusion is performed on the medical image features and annotation text features based on the fusion feature similarity data to obtain target semantic features. This allows the two features to complement each other, effectively integrating information and ultimately yielding target semantic features that more comprehensively and accurately reflect medical content, thus improving the accuracy and reliability of medical image semantic extraction.

[0055] In step S301 of some embodiments, the similarity calculation can be obtained by any of the following methods: Method 1: Perform cosine similarity calculation on the first fusion feature and the second fusion feature to obtain fusion feature similarity data; or Method 2: Perform mutual information calculation on the first fusion feature and the second fusion feature to obtain fusion feature similarity data; or Method 3: Perform similarity discrimination on the first fusion feature and the second fusion feature through a pre-trained similarity discrimination network to obtain fusion feature similarity data.

[0056] It should be noted that the similarity data of fusion features can provide an important reference for subsequent semantic fusion. The higher the value of the similarity data of fusion features, the more similar the first fusion feature and the second fusion feature are; the lower the value of the similarity data of fusion features, the less similar the first fusion feature and the second fusion feature are.

[0057] Specifically, for Method 1, a specific cosine similarity algorithm is used to measure the similarity between the first fused feature and the second fused feature to obtain fused feature similarity data.

[0058] For Method Two, mutual information is an indicator that measures the degree of interdependence between two random variables, reflecting the reduction in uncertainty about one variable after knowing information about the other. First, the probability distributions of the first and second fusion features are determined. Then, the mutual information values ​​are calculated for the first and second fusion features according to the mutual information formula, yielding fusion feature similarity data. This approach considers not only the surface similarity between features but also their probability distributions, providing a more comprehensive measure of the correlation between the two features.

[0059] For method three, the pre-trained similarity discriminant network is a neural network model trained on a large amount of data, capable of learning the similarity relationships between features of different modalities. The similarity discriminant network has multiple hidden layers, enabling it to perform complex nonlinear transformations on the input feature vectors, thereby more accurately determining the similarity between features.

[0060] Specifically, the first and second fusion features are input into a pre-trained similarity discrimination network. A series of nonlinear transformations and calculations are performed on the first and second fusion features, and finally a scalar value representing similarity is output, which is the fusion feature similarity data. For some similarities that are difficult to measure using simple mathematical methods, it can make more accurate judgments.

[0061] For example, in a medical imaging diagnostic scenario, the first fusion feature is a feature vector extracted from and fused with information such as image texture and lesion location from a lung CT image; the second fusion feature is a vector obtained by extracting and fusing features from the doctor's annotation text on the CT image (such as "a nodule in the upper lobe of the left lung with clear boundaries"). Calculating the similarity using either of these methods yields a similarity score of 0.85, indicating a high semantic correlation between the two features.

[0062] In other scenarios, assuming the original medical image is a picture of a "cat" and the image caption is "a cute kitten," the fused feature similarity data will be high. Conversely, assuming the original medical image is a picture of a "cat" and the image caption is "a speeding car," the fused feature similarity data will be low.

[0063] In step S302 of some embodiments, multiple preset intervals are established according to the numerical value of the fusion feature similarity data, namely a first preset interval, a second preset interval, a third preset interval and a fourth preset interval. Each preset interval corresponds to a fusion strategy, which is used to fuse medical image features and annotation text features to obtain target semantic features.

[0064] Specifically, assuming the range of values ​​for the fused feature similarity data is [0, 1], then the first preset interval can be set to (0.5, 0.75], the second preset interval can be set to (0.75, 1], the third preset interval can be set to (0.25, 0.5], and the fourth preset interval can be set to [0, 0.25].

[0065] For each preset interval, a corresponding fusion strategy is pre-set, as follows: For the first preset interval, the preset weighted fusion strategy is to perform weighted fusion of medical image features and annotation text features based on the attention weight of medical image features and the attention weight of annotation text features.

[0066] The second preset interval is the preset intersection fusion strategy, which is to perform intersection processing on medical image features and annotation text features.

[0067] The third preset interval is the preset union fusion strategy, which is to perform union processing on medical image features and annotation text features.

[0068] The fourth preset interval is a preset confidence screening strategy, which selects the data with the highest confidence from medical image features and annotation text features.

[0069] Please refer to Figure 4. In some embodiments, step S302 may include, but is not limited to, steps S401 to S402: Step S401: If the fusion feature similarity data is in the first preset interval, based on a preset weighted fusion strategy, the attention weight of the medical image features is obtained to obtain the image attention weight, and the attention weight of the annotation text features is obtained to obtain the text attention weight; Step S402: Based on the image attention weight and the text attention weight, the medical image features and the annotation text features are weighted and fused to obtain the target semantic features.

[0070] In steps S401 to S402 of this embodiment, when the fused feature similarity data is within a first preset interval, attention weights for medical image features are obtained based on a preset weighted fusion strategy to obtain image attention weights, and attention weights for annotation text features are obtained to obtain text attention weights, which can accurately measure the importance of different features. Then, the medical image features and annotation text features are weighted and fused based on the image attention weights and text attention weights to obtain target semantic features. This fully preserves the key information of the medical image and effectively incorporates the semantic content of the annotation text, thereby improving the rationality and accuracy of feature fusion and enhancing the accuracy and reliability of medical image semantic extraction.

[0071] In step S401 of some embodiments, the image attention weight represents the relative importance assigned to a medical image feature during the fusion process. A larger weight indicates a greater role the image feature plays in the final fusion result. The text attention weight represents the relative importance assigned to the annotation text feature during the fusion process. The weight magnitude reflects the degree of contribution of the text feature to the fusion result.

[0072] In some embodiments, the image attention weight can be the attention weight when the first fusion feature is obtained by attention calculation using medical image features as query features and annotation text features as key and value features, and the text attention weight can be the attention weight when the second fusion feature is obtained by attention calculation using annotation text features as query features and medical image features as key and value features.

[0073] In other embodiments, the weights of image attention weights and text attention weights can be set based on the numerical values ​​of the fused feature similarity data. For example, the numerical values ​​of the fused feature similarity data are the image attention weights, and (1 - the numerical values ​​of the fused feature similarity data) are the text attention weights.

[0074] In step S402 of some embodiments, the medical image features and annotation text features are combined according to their respective weights to obtain the target semantic features, i.e.: Target semantic features = Medical image features * Image attention weight + Annotation text features * Text attention weight. Weighted fusion can make full use of the feature information of images and texts, and reasonably combine them according to their importance, avoiding the inadequacy of single feature information and improving the richness and accuracy of features.

[0075] Please refer to Figure 5. In some embodiments, step S302 may include, but is not limited to, steps S501 to S502: Step S501: If the fusion feature similarity data is in the second preset interval, the medical image features and annotation text features are subjected to intersection processing based on a preset intersection fusion strategy to obtain intersection semantic features; Step S502: The intersection semantic features are confirmed as target semantic features.

[0076] In the embodiments of this application, steps S501 to S502, when the fusion feature similarity data is within a second preset range, are performed on the medical image features and annotation text features based on a preset intersection fusion strategy to obtain intersection semantic features. This can accurately extract the key information shared by both, obtain intersection semantic features, and confirm the intersection semantic features as target semantic features. This can ensure that the target semantic features are highly focused on the core content and improve semantic accuracy.

[0077] In step S501 of some embodiments, the medical image features and annotation text features are subjected to intersection processing to find the strong semantic information jointly supported by the two. This process eliminates irrelevant parts of the medical image features and annotation text features, reduces noise interference, and makes the obtained intersection semantic features purer and more accurate. For example, if the medical image features include the feature "lung nodule" and the annotation text features also mention "lung nodule," then "lung nodule" will be extracted as part of the intersection semantic features.

[0078] Furthermore, the intersection semantic features are identified as target semantic features, providing accurate semantic information for subsequent tasks such as disease diagnosis and disease analysis.

[0079] Please refer to Figure 6. In some embodiments, step S302 may include, but is not limited to, steps S601 to S602: Step S601, if the fusion feature similarity data is in a third preset interval, perform union processing on the medical image features and annotation text features based on a preset union fusion strategy to obtain union semantic features; Step S602, confirm the union semantic features as the target semantic features.

[0080] In the embodiments of this application, steps S601 to S602, when the fusion feature similarity data is in the third preset interval, perform union processing on medical image features and annotation text features based on a preset union fusion strategy. This can comprehensively integrate the information of both, cover more potential key content, obtain union semantic features, and confirm the union semantic features as target semantic features. This can ensure that the target semantic features are complete and rich, and improve the accuracy and reliability of medical image semantic extraction.

[0081] In step S601 of some embodiments, the medical image features and the annotation text features are merged to form a union semantic feature, integrating all information to avoid omitting key content and providing more complete semantic information for subsequent analysis. For example, if the medical image features include "brain tumor, right frontal lobe" and the annotation text features include "headache, brain tumor, 3cm", the union of these features yields "brain tumor, right frontal lobe, headache, 3cm".

[0082] Furthermore, the semantic features of the union are identified as the target semantic features, providing accurate semantic information for subsequent tasks such as disease diagnosis and disease analysis.

[0083] In some embodiments, step S302 may include, but is not limited to, the following steps: if the fused feature similarity data is within a fourth preset interval, based on a preset confidence screening strategy, obtain the image confidence of the medical image features and the text confidence of the annotation text features; select the data with the largest value from the text confidence and image confidence to obtain the target confidence; perform feature screening on the medical image features and annotation text features based on the target confidence to obtain the target semantic features.

[0084] Specifically, when the fused feature similarity data falls within the fourth preset interval, the image confidence score of the medical image features obtained in step S102 and the text confidence score of the annotation text features obtained in step S103 are acquired, which can accurately measure the reliability of different features. Next, the data with the highest value is selected from the text confidence score and image confidence score as the target confidence score, and the feature corresponding to the target confidence score (medical image feature or annotation text feature) is used as the target semantic feature, which helps improve the reliability of the final target semantic feature.

[0085] After step S105 in some embodiments, the medical image semantic extraction method may further include tasks such as disease diagnosis, disease analysis, and information collection (for building a disease database) based on target semantic features.

[0086] The medical image semantic extraction method of this application embodiment can be used not only for feature extraction of medical images, but also for semantic extraction of financial images, educational images, image classification tasks, RAG systems and other scenarios. The difference lies in the different data sources obtained in step S101.

[0087] In addition to extracting semantic information from images, semantic fusion can also be performed on speech and its corresponding annotation text, speech and its images, time-series data and its annotation text, and time-series data and images. The difference lies in the different feature extraction models used between steps S102 and S103 (e.g., using the Whisper model for feature extraction of speech and the LSTM model for feature extraction of time-series data). The appropriate model needs to be selected based on the actual application scenario, and is not limited to this.

[0088] Referring to Figure 7, this application embodiment also provides a medical image semantic extraction device, which can implement the above-mentioned medical image semantic extraction method. The device includes: a data acquisition module 701, used to acquire the original medical image and the image annotation text of the original medical image; an image feature extraction module 702, used to extract features from the original medical image to obtain medical image features; a text feature extraction module 703, used to extract features from the image annotation text to obtain annotation text features; a cross-modal fusion module 704, used to perform cross-modal fusion based on the medical image features and the annotation text features to obtain cross-modal fusion features; and a semantic fusion module 705, used to perform semantic fusion based on the cross-modal fusion features on the medical image features and the annotation text features to obtain target semantic features.

[0089] The specific implementation of this medical image semantic extraction device is basically the same as the specific implementation of the above-mentioned medical image semantic extraction method, and will not be repeated here.

[0090] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described medical image semantic extraction method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0091] Please refer to Figure 8, which illustrates the hardware structure of an electronic device according to another embodiment. The electronic device includes: a processor 801, which can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, for executing related programs to implement the technical solutions provided in the embodiments of this application; and a memory 802, which can be implemented using a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM), etc. The memory 802 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 802 and is called and executed by the processor 801 to implement the medical image semantic extraction method of the embodiments of this application. The input / output interface 803 is used to realize information input and output. The communication interface 804 is used to realize communication interaction between this device and other devices. Communication can be realized by wired means (such as USB, network cable, etc.) or by wireless means (such as mobile network, WIFI, Bluetooth, etc.). The bus 805 transmits information between the various components of the device (such as processor 801, memory 802, input / output interface 803 and communication interface 804). The processor 801, memory 802, input / output interface 803 and communication interface 804 realize communication connection between each other within the device through the bus 805.

[0092] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described medical image semantic extraction method.

[0093] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0094] The medical image semantic extraction method, apparatus, electronic device, and storage medium provided in this application acquire original medical images and image annotation text of the original medical images, and extract features from the original medical images and image annotation text respectively to obtain corresponding medical image features and annotation text features. This can accurately capture key visual information in the image and extract the core semantics of the text. Next, cross-modal fusion is performed based on the medical image features and annotation text features to obtain cross-modal fusion features, breaking down the modal barriers between image and text and integrating the advantages of different modalities. Finally, semantic fusion is performed on the medical image features and annotation text features based on the cross-modal fusion features to obtain target semantic features. This can more deeply explore the inherent semantic relationship between the image and text, accurately and comprehensively reflect medical image information, and solve the technical problem of inaccurate semantic extraction caused by the inability to extract semantics based on the quality, importance, and complex relationships between information of each modality. This improves the accuracy and reliability of medical image semantic extraction.

[0095] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0096] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0097] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0098] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0099] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0100] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0101] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0102] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0103] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0104] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0105] The software tools or components not belonging to our company that appear in the embodiments of this application are for illustrative purposes only and do not represent actual use.

[0106] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A method for semantic extraction from medical images, characterized in that, The method includes: acquiring an original medical image and image annotation text of the original medical image; extracting features from the original medical image to obtain medical image features; extracting features from the image annotation text to obtain annotation text features; performing cross-modal fusion based on the medical image features and the annotation text features to obtain cross-modal fusion features; and performing semantic fusion based on the cross-modal fusion features to obtain target semantic features.

2. The method according to claim 1, characterized in that, The cross-modal fusion feature includes a first fusion feature and a second fusion feature; the step of semantically fusing the medical image feature and the annotation text feature based on the cross-modal fusion feature to obtain the target semantic feature includes: performing similarity calculation based on the first fusion feature and the second fusion feature to obtain fusion feature similarity data; and performing semantic fusion on the medical image feature and the annotation text feature based on the fusion feature similarity data to obtain the target semantic feature.

3. The method according to claim 2, characterized in that, The step of semantically fusing the medical image features and the annotation text features based on the fusion feature similarity data to obtain the target semantic features includes: if the fusion feature similarity data is in a first preset interval, obtaining the attention weight of the medical image features based on a preset weighted fusion strategy to obtain the image attention weight, obtaining the attention weight of the annotation text features to obtain the text attention weight; and performing weighted fusion of the medical image features and the annotation text features based on the image attention weight and the text attention weight to obtain the target semantic features.

4. The method according to claim 2, characterized in that, The step of semantically fusing the medical image features and the annotation text features based on the fusion feature similarity data to obtain the target semantic feature includes: if the fusion feature similarity data is in a second preset interval, performing intersection processing on the medical image features and the annotation text features based on a preset intersection fusion strategy to obtain intersection semantic features; and confirming the intersection semantic features as the target semantic feature.

5. The method according to claim 2, characterized in that, The step of semantically fusing the medical image features and the annotation text features based on the fusion feature similarity data to obtain the target semantic feature includes: if the fusion feature similarity data is within a third preset interval, performing union processing on the medical image features and the annotation text features based on a preset union fusion strategy to obtain a union semantic feature; and confirming the union semantic feature as the target semantic feature.

6. The method according to claim 2, characterized in that, The step of calculating similarity based on the first fusion feature and the second fusion feature to obtain fusion feature similarity data includes: calculating cosine similarity between the first fusion feature and the second fusion feature to obtain the fusion feature similarity data; or, calculating mutual information between the first fusion feature and the second fusion feature to obtain the fusion feature similarity data; or, performing similarity discrimination on the first fusion feature and the second fusion feature through a pre-trained similarity discrimination network to obtain the fusion feature similarity data.

7. The method according to any one of claims 2 to 6, characterized in that, The method of performing cross-modal fusion based on the medical image features and the annotation text features to obtain cross-modal fusion features includes: performing attention calculation using the medical image features as query features, the annotation text features as key features and value features to obtain the first fusion feature; and performing attention calculation using the annotation text features as query features, the medical image features as key features and value features to obtain the second fusion feature.

8. A medical image semantic extraction device, characterized in that, The device includes: a data acquisition module for acquiring an original medical image and image annotation text of the original medical image; an image feature extraction module for extracting features from the original medical image to obtain medical image features; a text feature extraction module for extracting features from the image annotation text to obtain annotation text features; a cross-modal fusion module for performing cross-modal fusion based on the medical image features and the annotation text features to obtain cross-modal fusion features; and a semantic fusion module for performing semantic fusion based on the cross-modal fusion features on the medical image features and the annotation text features to obtain target semantic features.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images