Analysis method and device based on medical image, electronic equipment and storage medium

Abstract

The invention provides an analysis method and device based on a medical image, electronic equipment and a storage medium, and the method comprises the steps: extracting a target region feature of a target region of interest in a to-be-analyzed medical image; searching the first k image-text features with the highest similarity with the target area features from a reference feature database, k being a positive integer, reference region features extracted from a reference region of interest of a reference medical image and reference text features extracted from a reference text generated for the reference region of interest are stored in the reference feature database, and the image-text features are the reference region features or the reference text features. And generating an analysis result of the target region of interest based on each reference text corresponding to the k image-text features.

Description

Technical Field

[0001] The present specification relates to the field of medical imaging technology, and in particular to analysis methods, devices, electronic devices and storage media based on medical images. Background Art

[0002] Intelligent analysis based on medical images refers to the use of professional imaging equipment to generate high-resolution images of the human body, and the use of computers to analyze medical images (such as X-rays, CT, MRI, ultrasound, nuclear medicine images, etc.) to identify and evaluate lesions or abnormalities in human tissues and organs, and automatically give analysis results, thereby helping doctors to accurately locate lesions and quantify the condition of lesions, assisting doctors in disease analysis and treatment decision-making, and improving analysis efficiency and accuracy.

[0003] However, related intelligent analysis methods based on medical images are often based on training data of defined diseases and are unable to effectively identify or classify newly added categories of diseases. Summary of the invention

[0004] To overcome the problems existing in the related art, this specification provides an analysis method, device, electronic device and storage medium based on medical images.

[0005] According to a first aspect of an embodiment of this specification, a medical image-based analysis method is provided, the method comprising:

[0006] Extracting target region features of a target region of interest in a medical image to be analyzed;

[0007] Retrieving the first k graphic features with the highest similarity to the target region feature from a reference feature database, where k is a positive integer, and the reference feature database stores reference region features extracted from a reference region of interest of a reference medical image and reference text features extracted from a reference text generated for the reference region of interest, and the graphic features are the reference region features or the reference text features;

[0008] The analysis result of the target region of interest is generated based on the reference texts respectively corresponding to the k graphic features.

[0009] According to a second aspect of an embodiment of this specification, there is provided a medical image-based analysis device, the device comprising:

[0010] A region feature extraction module, used to extract target region features of a target region of interest in a medical image to be analyzed;

[0011] A graphic feature retrieval module is used to retrieve the top k graphic features with the highest similarity to the target region feature from a reference feature database, where k is a positive integer, and the reference feature database stores reference region features extracted from a reference region of interest of a reference medical image and reference text features extracted from a reference text generated for the reference region of interest, and the graphic feature is the reference region feature or the reference text feature;

[0012] The analysis result generating module is used to generate the analysis result of the target region of interest based on the reference texts corresponding to the k graphic features.

[0013] According to a third aspect of the embodiments of this specification, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the method described in the first aspect are implemented.

[0014] According to a fourth aspect of the embodiments of this specification, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method described in the first aspect.

[0015] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0016] In the embodiments of this specification, this scheme retrieves multiple graphic features with the highest similarity to the target region of interest features in the medical image to be analyzed from the reference feature database, and determines the analysis result of the target region of interest in the medical image to be analyzed based on the reference texts respectively associated with these graphic features. It can be seen that the reference analysis texts of the reference medical images similar to the medical image to be analyzed can be found through the above method, and the analysis result of the target region of interest in the medical image to be analyzed can be determined based on these reference analysis texts. In addition, by expanding the reference feature database with graphic feature data of new reference analysis cases, the reference analysis texts of these new analysis cases will also appear in the retrieval results accordingly, and the medical image to be analyzed can be analyzed based on the new reference analysis texts, so it can adapt to the analysis needs of new diseases.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0019] Figure 1The flowchart of a medical image-based analysis method is shown in this specification according to an exemplary embodiment.

[0020] Figure 2 It is a schematic diagram of a process of analyzing a medical image according to an exemplary embodiment of the present specification.

[0021] Figure 3 It is a structural diagram of a feature extraction model shown in this specification according to an exemplary embodiment.

[0022] Figure 4 It is a flow chart of a feature extraction method shown in this specification according to an exemplary embodiment.

[0023] Figure 5 It is a structural schematic diagram of an electronic device according to an exemplary embodiment of the present specification.

[0024] Figure 6 It is a block diagram of a medical image-based analysis device according to an exemplary embodiment of the present specification.

[0025] Figure 7 It is a block diagram of a feature extraction device shown in this specification according to an exemplary embodiment. DETAILED DESCRIPTION

[0026] Intelligent analysis of medical images refers to the use of professional imaging equipment to generate high-resolution images of the human body, and the use of computers to analyze medical images (such as X-rays, CT, MRI, ultrasound, nuclear medicine images, etc.) to identify and evaluate lesions or abnormalities in human tissues and organs, and automatically give analysis results, thereby helping doctors to accurately locate lesions and quantify the condition of lesions, assisting doctors in disease analysis and treatment decision-making, and improving analysis efficiency and accuracy.

[0027] Early anomaly recognition methods mainly relied on traditional computer vision techniques and hand-designed feature extraction algorithms. These methods perform well in dealing with specific types of anomalies. For example, threshold-based methods work well in detecting lesions with clear boundaries, while methods based on shape and texture features are also very effective in identifying certain specific structures. However, when faced with complex, diverse and interrelated anomalies, the limitations of traditional methods gradually become apparent. For example, they have difficulty handling lesions with blurred boundaries or irregular shapes, and are prone to false positives or false negatives when multiple types of anomalies coexist.

[0028] In recent years, with the development of deep learning technology, image classification models (especially convolutional neural networks and their variants) have been widely used in the intelligent analysis of medical images and have achieved remarkable results. For example, in X-ray screening for breast cancer, image classification models can distinguish between malignant and benign masses; for example, image classification models can also identify multiple pathological features in medical images. For example, in chest X-rays, image classification models can simultaneously classify multiple diseases such as pneumonia, tuberculosis, and pneumothorax.

[0029] However, image classification models can usually only identify a single or a small number of disease types, and cannot identify new categories that were not defined during training. As the disease spectrum continues to change (such as emerging infectious diseases, variant forms of lesions, etc.), image classification models cannot adapt dynamically and need to re-collect data and train models, which is time-consuming and labor-intensive and difficult to quickly respond to clinical needs.

[0030] In response to the above technical problems, this specification proposes an analysis method based on medical images to meet the analysis needs of new diseases.

[0031] Next, the embodiments of this specification are described in detail.

[0032] like Figure 1 As shown, Figure 1 1 is a flowchart of a medical image-based analysis method according to an exemplary embodiment of the present specification, including steps 101-103:

[0033] Step 101: extracting target region features of a target region of interest in a medical image to be analyzed.

[0034] Step 102: Retrieve the first k graphic features with the highest similarity to the target region features from the reference feature database, where k is a positive integer.

[0035] Step 103: Generate analysis results of the target region of interest based on the reference texts corresponding to the k graphic features.

[0036] The medical image to be analyzed may refer to a medical image obtained by medical imaging technology (such as X-ray, CT, MRI, ultrasound, pathological section, etc.) for which the analysis conclusion has not yet been determined. In general, the medical image to be analyzed may include multiple regions of interest, and the regions of interest may be divided according to the type of organ. For example, taking chest CT as the medical image to be analyzed, the medical image to be analyzed may be divided into regions of interest corresponding to different organs such as lungs, heart, mediastinum, aorta, pulmonary artery, esophagus, trachea, etc. In other cases, the medical image to be analyzed may be equivalent to the region of interest, that is, the region of interest covers the entire medical image to be analyzed, and the region of interest is the medical image to be analyzed itself.

[0037] The reference feature database stores reference region features extracted from the reference region of interest of the reference medical image and reference text features extracted from the reference text generated for the reference region of interest. The image and text features are reference region features and / or reference text features.

[0038] The reference region of interest of the reference medical image and the reference text generated for the reference region of interest may constitute a set of analysis cases, and the reference text may be an analysis conclusion drawn from the condition of the reference region of interest. For example, the reference medical image is a chest CT, the reference region of interest may be a heart region, and the doctor's analysis conclusion for the reference region of interest is "pericardial effusion", and "pericardial effusion" is the reference text. The reference region of interest of the reference medical image may be the reference medical image itself, and the reference text generated for the reference region of interest may be the reference text generated for the reference medical image. For example, the reference medical image is also a chest CT, and the doctor's analysis conclusion for the reference medical image is "flaky ground-glass shadows are seen in the right lower lung, and the left ventricle is slightly enlarged", and "flaky ground-glass shadows are seen in the right lower lung, and the left ventricle is slightly enlarged" is the reference text generated for the reference medical image.

[0039] Users can cooperate with hospitals and medical institutions to obtain analysis cases. Alternatively, they can use public medical imaging databases to obtain analysis cases. For each set of analysis cases, reference region features are extracted from the reference region of interest of the reference medical image, and reference text features are extracted from the reference text generated for the reference region of interest. Finally, the reference region features and reference text features are stored in the reference feature database. This reference feature database supports updating and expanding new analysis cases. When a new analysis case appears, the feature data corresponding to the new analysis case can be stored in the feature database.

[0040] It should be noted that those skilled in the art can design a feature extraction model based on the relevant knowledge of deep learning. The feature extraction model can be obtained by comparing and learning the regional features of the region of interest of the input image and the text features of the input text generated for the region of interest, so that the feature extraction model has the ability to align and extract the image features of the image and the text features of the text. The feature extraction model can be used to extract the target region features of the target region of interest in the medical image to be analyzed, and can also be used to extract reference region features from the reference region of interest of the reference medical image and to extract reference text features from the reference text generated for the reference region of interest. Of course, the feature extraction model provided in this specification (which will be introduced in the following embodiments) can also be used to achieve the above purpose, and this specification does not impose any restrictions on the feature extraction method.

[0041] like Figure 2 As shown, Figure 2 1 is a schematic diagram of a process of analyzing a medical image according to an exemplary embodiment of the present specification. First, assuming that the medical image 21 to be analyzed includes multiple regions of interest 211a, target regions of interest 211b, and regions of interest 211c, etc., a target region feature 22 of the target region of interest 211b in the medical image 21 to be analyzed can be extracted. If the complete medical image 21 to be analyzed is regarded as a target region of interest, extracting the target region feature 22 of the target region of interest 211b in the medical image 21 to be analyzed can be equivalent to extracting the image feature of the medical image to be analyzed.

[0042] Then, the first k graphic features 24a, graphic features 24b, and graphic features 24k having the highest similarity to the target region feature 22 are retrieved from the reference feature database 23. If the graphic feature is a reference region feature, the first k reference region features having the highest similarity to the target region feature 22 can be retrieved from the reference feature database 23. If the graphic feature is a reference text feature, the first k reference text features having the highest similarity to the target region feature 22 can be retrieved from the reference feature database 23. The similarity can be calculated using similarity calculation formulas such as cosine similarity and Euclidean distance, and this specification does not limit the method for calculating the similarity between the target region feature and the graphic feature.

[0043] Finally, the analysis result 26 of the target region of interest 211b is generated based on the k reference texts, such as reference text 25a, reference text 25b and reference text 25k, which correspond to the k graphic features. Regarding the method of determining the reference text based on the graphic feature, for example, if the graphic feature is a reference region feature, the reference region of interest represented by the reference region feature can be determined, and then the reference text for the reference region of interest can be determined. If the graphic feature is a reference text feature, the reference text represented by the reference text feature can be determined.

[0044] In this embodiment, this scheme retrieves multiple graphic features with the highest similarity to the target region of interest features in the medical image to be analyzed from the reference feature database, and determines the analysis result of the target region of interest in the medical image to be analyzed based on the reference texts respectively associated with these graphic features. It can be seen that the reference analysis texts of the reference medical images similar to the medical image to be analyzed can be found by the above method, and the analysis result of the target region of interest in the medical image to be analyzed can be determined based on these reference analysis texts. In addition, by expanding the reference feature database with graphic feature data of new reference analysis cases, the reference analysis texts of these new analysis cases will also appear in the retrieval results accordingly, and the medical image to be analyzed can be analyzed based on the new reference analysis texts, so it can adapt to the analysis needs of new diseases.

[0045] In one embodiment, the present specification also provides a preferred method for generating an analysis result 26 based on k reference texts. For each reference text, the disease recorded in the reference text can be identified and a corresponding symptom label can be generated. When the analysis result of the target region of interest of the medical image to be analyzed is generated based on the reference texts corresponding to the k graphic features, the symptom labels of the reference texts corresponding to the k graphic features can be obtained. If the ratio of the total number of occurrences of any symptom label to k is not less than the ratio threshold, the symptom label is used as the analysis result. For example, assuming k = 3, the first reference text is "right lung nodule, diameter 1.5cm, high possibility of malignancy", the symptom identified from the reference text is "right lung nodule", and the corresponding symptom label "lung nodule" is generated. The second reference text is "right lung nodule, rough edge, follow-up recommended", the symptom identified from the reference text is "right lung nodule", and the corresponding symptom label "lung nodule" is generated. The third reference text is "Right lung nodule suspected of early lung cancer, biopsy recommended", the symptoms identified from this reference text are "right lung nodule" and "early lung cancer", and the corresponding symptom labels are "lung nodule" and "lung cancer". The symptom label "lung nodule" generated for these three reference texts appears three times, while "lung cancer" appears once. The number of times "lung nodule" appears accounts for 100%, while the number of times "lung cancer" appears accounts for 33%. Assuming the ratio threshold is 50%, the final analysis result can be "lung nodule".

[0046] In this embodiment, for each reference text, the disease recorded in the reference text is identified and the corresponding symptom label is generated. When generating analysis results based on each reference text, the symptom labels in each reference text are counted and screened to synthesize the common symptom labels of all reference texts, which can effectively reduce the risk of bias and misdiagnosis caused by a single text. In addition, the symptom labels are screened based on the ratio threshold, which has a low implementation cost and is suitable for use in situations with limited resources.

[0047] In addition, natural language processing technology may be used to summarize all contents of the k reference texts to generate a summary analysis report as the analysis result 26. This specification does not limit the manner of generating the analysis result 26 based on the k reference texts.

[0048] In one embodiment, if Figure 3 As shown, Figure 3 It is a structural diagram of a feature extraction model shown in this specification according to an exemplary embodiment. The feature extraction model includes an image encoding module 34, a region encoding module 38 and a text encoding module 37. Among them, the image encoding module 34 is used to extract the global features 36 of the input image 32. The region encoding module 38 is used to extract the regional features 39 of the target region of interest 31 of the input image 32, and specifically can be to generate a target mask 35 of the target region of interest 31, and input the target mask 35 and the global features 36 to the region encoding module 38 to obtain the regional features 39. The text encoding module 37 is used to extract the text features 40 of the input text 33 generated for the target region of interest 31. The model can be obtained by comparative learning and training of the regional features 39 and the text features 40. The structure of the image encoding module 34 can be CNN (Convolutional Neural Network), ViT (Vision Transformer), Swin Transformer, etc. The region encoding module 38 may adopt the structure of a Transformer encoder, and the text encoding module 37 may adopt the structure of a Transformer encoder, or may adopt other encoders in natural language processing technology, such as a BERT (Bidirectional Encoder Representations from Transformers) encoder. This specification does not limit the specific structures of the image encoding module 34, the region encoding module 38, and the text encoding module 37.

[0049] The input image 32 may include at least one region of interest, and when the target region of interest 31 covers the entire input image 32, the input image 32 and the target region of interest 31 are equivalent. In the field of medical analysis, the type of the input image 32 may be a medical image, and the target region of interest 31 may be a region where a target organ is located in the medical image. In the field of face recognition, the type of the input image 32 may be a human face image, and the target region of interest 31 may be a region where a target sense organ is located on the face. In the field of medical analysis, it is assumed that the input image 32 is a chest CT, and the target region of interest 31 is the region where the heart is located in the chest CT. The input text 33 generated for the target region of interest 31 may be "pericardial effusion". Then, the feature extraction model may be used to extract the regional features 39 of the target region of interest 31 and / or extract the text features 40 of the input text 33. In the field of face recognition, it is assumed that the input image 32 is a human face image, and the target region of interest 31 may be the region where the nose is located. The input text 33 generated for the nose may be "high nose bridge and slightly downward curved nose tip". Then, the feature extraction model may also be used for feature extraction. The above is only an exemplary description, and this specification does not limit the application field of the feature extraction model.

[0050] In this embodiment, the present solution first uses the image encoding module 34 to extract the global features of the input image, and then uses the regional encoding module 38 to extract the regional features of the region of interest of the input image from the global features. The text encoding module 37 extracts the text features of the text in the region of interest. Therefore, when the feature extraction model performs feature alignment, it aligns the regional features of the region of interest of the input image and the text features of the text in the region of interest. The alignment granularity is smaller, which can improve the model's ability to extract image features, and can also achieve the purpose of feature extraction of any region of the input image.

[0051] In one embodiment, in the field of medical image analysis, the feature extraction model can be trained in the following manner:

[0052] A sample medical image and a sample report generated for the sample medical image are obtained, each sample region of interest is obtained from the sample medical image, and a sample text for each sample region of interest is obtained from the sample report.

[0053] The sample medical image is input into the image encoding module to obtain the sample global features.

[0054] For each sample region of interest, a sample mask of the sample region of interest is generated, the sample global feature and the sample mask are input into the region encoding module to obtain the sample region feature of the sample region of interest, and the sample text for the sample region of interest is input into the text encoding module to obtain the sample text feature of the sample text, and comparative learning is performed based on the sample region feature and the sample text feature.

[0055] Specifically, the relationship between the sample medical image and the sample report is that the sample report records the analysis text of the sample medical image. If the sample medical image contains multiple organ regions, the sample report records the sample text of the analysis results of the multiple organs accordingly. The sample regions of interest of each organ can be segmented from the sample medical image according to the organ type, and the sample text of the sample regions of interest for each organ can be obtained from the sample report according to the organ type. For example, the sample medical image is a chest CT, which contains organ regions such as the lungs and the heart, and the sample report records "no obvious calcification in the coronary artery, nodule in the right upper lobe of the lung". Then the region of interest where the heart is located can be segmented from the sample medical image, and the sample text "no obvious calcification in the coronary artery" can be split from the sample report; and the region of interest where the lungs are located can be segmented from the sample medical image, and the sample text "nodule in the right upper lobe of the lung" of the corresponding analysis conclusion can be split from the sample report. Of course, if the entire sample medical image is taken as the sample region of interest, the sample text for the sample region of interest is essentially a sample report for the sample medical image.

[0056] In the training phase, the sample medical image is input into the image encoding module 34 to obtain the sample global feature. For each sample region of interest in the sample medical image, a sample mask of the sample region of interest is generated. The sample mask can be a 0-1 matrix. For example, the pixel value of the sample region of interest in the sample medical image is 1, and the pixel value of the remaining region is 0. The size of the sample mask is processed to be the same as the size of the sample global feature, and the sample text for the region of interest is input into the text encoder 37 to obtain the sample text feature of the sample text, and comparative learning is performed based on the sample region feature and the sample text feature.

[0057] The loss function used in contrastive learning can be:

[0058]

[0059] Among them, B represents the batch size; x represents the sample region feature, y represents the sample text feature; z represents whether the sample region of interest and the sample text in a batch match. If they match, z is 1, otherwise it is -1; t and b are two learnable parameters.

[0060] This specification does not impose any restrictions on the contrastive learning technology used in the training model. By contrastive learning of sample region features and sample text features, cross-modal alignment of images and texts can be achieved.

[0061] Of course, the training method of the model in other application fields is similar, the difference is that the input data is different. Those skilled in the art can undoubtedly use data from other fields to train the feature extraction model so that it can be applied to other application fields.

[0062] The trained feature extraction model can be used to extract regional features of the region of interest of the input image and / or to extract text features of the input text for the region of interest. In one embodiment, it can be used to extract target region features of the target region of interest in the medical image to be analyzed:

[0063] Specifically, when extracting the target region features of the target region of interest in the medical image to be analyzed, the medical image to be analyzed can be input into the image encoding module 34 of the feature extraction model to obtain the global features to be analyzed. A target mask of the target region of interest in the medical image to be analyzed is generated, and the global features and the target mask are input into the region encoding module 38 to obtain the target region features of the target region of interest. The target mask of the target region of interest in the medical image to be analyzed can be generated by determining the target region of interest in the medical image to be analyzed, and setting the pixel values ​​of the region where the target region of interest in the medical image to be analyzed is located to 1, and setting the pixel values ​​of the region other than the target region of interest to 0, so as to obtain the target mask.

[0064] In one embodiment, the trained feature extraction model may also be used to extract reference region features of a reference region of interest of a reference medical image and reference text features of a reference text for the reference region of interest.

[0065] Specifically, a reference medical image and a reference report generated for the reference medical image are acquired.

[0066] The reference medical image is input into the image encoding module 34 to obtain reference global features, and a reference mask of any region of interest in the reference medical image is generated, and the reference global features and the reference mask are input into the region encoding module 38 to obtain reference region features.

[0067] The reference text for any region of interest is obtained from the reference report, and the reference text is input into the text encoding module 37 to obtain the reference text feature.

[0068] The reference region features and the reference text features are stored in a reference feature database.

[0069] In this embodiment, new analysis cases can be continuously acquired, and the reference region features of the reference region of interest of the reference medical image in the new analysis case and the reference text features of the reference region of interest in the reference medical image can be extracted through the feature extraction model, and these can be expanded into the reference feature database, so as to quickly adapt when new diseases appear, and further improve the accuracy and reliability of detection.

[0070] In an illustrated embodiment, a reference medical image and a reference report generated for the reference medical image can be obtained. The reference medical image is input into the image encoding module 34 to obtain a reference global feature, and a reference mask of any region of interest in the reference medical image is generated, and the reference global feature and the mask are input into the region encoding module 38 to obtain a reference region feature. When obtaining a reference text for any region of interest from a reference report, a large language model (which can be based on a prompt engineering method or a fine-tuning method) can be used to generate a symptom label corresponding to the reference text for the region of interest in the report, such as a symptom label of pulmonary nodules, fractures, coronary artery calcification, etc. So far, when constructing a reference feature database, each case of data can include: reference region features, reference text features, and symptom labels for the region of interest. Then, when using the feature database, for example, when obtaining the symptom labels of each reference text corresponding to k graphic features in the aforementioned embodiment, it can be directly obtained from the feature database without the need to temporarily generate them based on the reference text.

[0071] In addition, compared with the traditional image classification model, the feature extraction model used in this solution can avoid the difficulty of obtaining high-quality annotated data on a large scale in practical applications by aligning the region of interest of the medical image with the corresponding text generated for the region of interest, without the need for additional manual annotation. By combining the feature extraction model with the reference feature database, it is also possible to dynamically retrieve similar analysis cases in the reference feature database using feature similarity and obtain analysis results based on similar analysis cases.

[0072] In addition to the above uses, the feature extraction model used in this solution can also be used for retrieval, such as using images to retrieve similar reports, using reports to retrieve similar images, using images to retrieve similar images, or using keywords to retrieve similar images. For details, please refer to the following embodiments:

[0073] In one embodiment, reference texts and / or reference images corresponding to the k graphic features may also be obtained. For example, a user may obtain k reference images with similar features to the medical image to be analyzed or k reference analysis texts with similar features to the medical image to be analyzed.

[0074] In another embodiment, the symptom description text can be obtained and the symptom text features of the symptom description text can be extracted, the first k graphic features with the highest similarity to the symptom text features can be retrieved from the reference feature database, and the reference texts and / or reference images corresponding to the k graphic features can be obtained. For example, if a user obtains a symptom description text for heart disease, k reference images similar to the symptom description text or k reference description texts similar to the symptom description text can be obtained based on this method.

[0075] In an exemplary application scenario, doctors can find commonalities among rare lesions by comparing k reference images with consistent symptom descriptions. Doctors can also write analysis reports with reference to the obtained k similar reference texts, thereby reducing the writing burden.

[0076] like Figure 4 As shown, Figure 4 4 is a flowchart of a feature extraction method according to an exemplary embodiment of the present specification, including steps 401-403:

[0077] Step 401: Input an image into an image encoding module of a feature extraction model to obtain global features of the image, and generate a target mask of a target region of interest in the image.

[0078] Step 402: Input the global feature and the target mask into a region encoding module to obtain a target region feature of the target region of interest.

[0079] Step 403: Input the target description text for the target region of interest into a text encoding module to extract target text features of the target description text.

[0080] Wherein, the feature extraction model includes an image encoding module, a region encoding module and a text encoding module.

[0081] The feature extraction model in this method can be used Figure 3The model structure of the feature extraction model. The image may include at least one region of interest, and when the target region of interest covers the entire image, the image and the target region of interest are equivalent. In the field of medical analysis, the type of the image may be a medical image, and the target region of interest may be the region where a target organ is located in the medical image. In the field of face recognition, the type of the image may be a human facial image, and the target region of interest may be the region where a target sense organ is located on the face. In the field of medical analysis, assuming that the image is a chest CT, the target region of interest is the region where the heart is located in the chest CT. The target description text generated for the target region of interest may be "pericardial effusion". In the field of face recognition, assuming that the image is a human facial image, the target region of interest may be the region where the nose is located. The target description text generated for the nose may be "the bridge of the nose is high and the tip of the nose is slightly curved downward". The above is only an exemplary description, and this specification does not limit the application field of this method.

[0082] In this embodiment, when the feature extraction model performs feature alignment, what is aligned is the regional features of the region of interest of the input image and the text features of the text in the region of interest. The alignment granularity is smaller, which can improve the model's ability to extract image features, and can also achieve the purpose of feature extraction of any region of the input image.

[0083] The feature extraction model provided in this specification can also be applied to the fields of video-text conversation (VQA), report generation, etc. For example, when training video-text conversation, the input question may only be for a certain area of ​​interest in the input image. At this time, if the global features of the input image are extracted, it is not conducive to the model to generate targeted answers. The feature extraction model provided in this specification is used to extract the local features of the area of ​​interest, which can be aligned with the question text for the area of ​​interest.

[0084] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of a device and a terminal to which it is applied.

[0085] like Figure 5 As shown, Figure 5It is a structural diagram of an electronic device shown in this specification according to an exemplary embodiment. At the hardware level, the electronic device 500 includes a processor 502, an internal bus 504, a network interface 506, a memory 508 and a non-volatile memory 510, and of course may also include hardware required for other services. One or more embodiments of this specification can be implemented based on software, such as the processor 502 reading the corresponding computer program from the non-volatile memory 510 into the memory 508 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic module, but can also be hardware or logic devices.

[0086] like Figure 6 As shown, Figure 6 is a block diagram of a medical image analysis device according to an exemplary embodiment of the present specification. The device can be applied to Figure 5 In the electronic device 500 shown in the figure, the technical solution of this specification is implemented. The device includes:

[0087] The region feature extraction module 602 is used to extract target region features of a target region of interest in the medical image to be analyzed.

[0088] The graphic feature retrieval module 604 is used to retrieve the top k graphic features with the highest similarity to the target area features from a reference feature database, where k is a positive integer. The reference feature database stores reference area features extracted from a reference area of ​​interest of a reference medical image and reference text features extracted from a reference text generated for the reference area of ​​interest. The graphic features are the reference area features and / or the reference text features.

[0089] The analysis result generating module 606 is used to generate the analysis result of the target region of interest based on the reference texts corresponding to the k graphic features.

[0090] Optionally, the feature vector extraction module 602 is specifically used to input the medical image to be analyzed into the image encoding module of the feature extraction model to obtain the global feature to be analyzed; generate a target mask of the target region of interest in the medical image to be analyzed, and input the global feature and the target mask into the region encoding module to obtain the target region feature of the target region of interest; wherein the feature extraction model includes the image encoding module and the region encoding module.

[0091] Optionally, the feature extraction model also includes a text encoding module, which is used to extract text features of the input text. The feature extraction model is trained in the following manner: obtaining a sample medical image and a sample report generated for the sample medical image, obtaining each sample region of interest from the sample medical image and obtaining sample text for each sample region of interest from the sample report; inputting the sample medical image into the image encoding module to obtain a sample global feature; for each sample region of interest, generating a sample mask of the sample region of interest, inputting the sample global feature and the sample mask into the region encoding module to obtain a sample region feature of the sample region of interest, and inputting the sample text for the sample region of interest into the text encoding module to obtain a sample text feature of the sample text, and performing comparative learning based on the sample region feature and the sample text feature.

[0092] Optionally, the device also includes a reference feature storage module, which is used to obtain a reference medical image and a reference report generated for the reference medical image; input the reference medical image into the image encoding module to obtain a reference global feature, and generate a reference mask of any region of interest in the reference medical image, and input the reference global feature and the reference mask into the region encoding module to obtain a reference region feature; obtain a reference text for any region of interest from the reference report, and input the reference text into the text encoding module to obtain a reference text feature; and store the reference region feature and the reference text feature in the reference feature database.

[0093] Optionally, the device further includes a symptom label generation module, which is used to identify the symptoms recorded in each reference text and generate a corresponding symptom label. The analysis result generation module 606 is specifically used to obtain the symptom labels of each reference text corresponding to the k graphic features; if the ratio of the total number of occurrences of any symptom label to k is not less than a ratio threshold, the symptom label is used as the analysis result.

[0094] Optionally, the device further includes a first reference image-text acquisition module, which is used to acquire reference texts and / or reference images corresponding to the k image-text features. And / or, a second reference image-text acquisition module, which is used to acquire a symptom description text and extract symptom text features of the symptom description text, retrieve the first k image-text features with the highest similarity to the symptom text features from a reference feature database, and acquire reference texts and / or reference images corresponding to the k image-text features.

[0095] like Figure 7 As shown, Figure 7is a block diagram of a feature extraction device according to an exemplary embodiment of the present specification. The device can be applied to Figure 5 In the electronic device 500 shown in the figure, the technical solution of this specification is implemented. The device includes:

[0096] The target region of interest determination module 702 is used to input the image into the image encoding module of the feature extraction model to obtain the global features of the image and generate a target mask of the target region of interest in the image.

[0097] The target region feature extraction module 704 is used to input the global feature and the target mask into the region encoding module to obtain the target region feature of the target region of interest.

[0098] The target text feature extraction module 706 is used to input the target description text for the target region of interest into the text encoding module to extract the target text features of the target description text. The feature extraction model includes an image encoding module, a region encoding module and a text encoding module.

[0099] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0100] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. Ordinary technicians in this field can understand and implement it without paying creative work.

[0101] The present specification also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the steps of any of the aforementioned medical image-based analysis methods provided in the present application are implemented.

[0102] Specifically, computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including, for example, semiconductor memory devices (such as EPROM, EEPROM and flash memory devices), magnetic disks (such as internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks.

Claims

1. A medical image-based analysis method, characterized in that: The method comprises: Extracting target region features of a target region of interest in a medical image to be analyzed; Retrieving the first k graphic features with the highest similarity to the target region features from a reference feature database, where k is a positive integer, and the reference feature database stores reference region features extracted from a reference region of interest of a reference medical image and reference text features extracted from a reference text generated for the reference region of interest, and the graphic features are the reference region features and / or the reference text features; The analysis result of the target region of interest is generated based on the reference texts respectively corresponding to the k graphic features.

2. The method according to claim 1, characterized in that: The step of extracting target region features of a target region of interest in a medical image to be analyzed includes: Inputting the medical image to be analyzed into an image encoding module of a feature extraction model to obtain a global feature to be analyzed; Generating a target mask of a target region of interest in the medical image to be analyzed, and inputting the global feature and the target mask into a region encoding module to obtain a target region feature of the target region of interest; Wherein, the feature extraction model includes the image encoding module and the region encoding module.

3. The method according to claim 2, characterized in that The feature extraction model further includes a text encoding module, which is used to extract text features of the input text. The feature extraction model is trained in the following manner: Acquire a sample medical image and a sample report generated for the sample medical image, acquire each sample region of interest from the sample medical image and acquire sample texts for each sample region of interest from the sample report; Inputting the sample medical image into the image encoding module to obtain sample global features; For each sample region of interest, a sample mask of the sample region of interest is generated, the sample global feature and the sample mask are input into the region encoding module to obtain the sample region feature of the sample region of interest, and the sample text for the sample region of interest is input into the text encoding module to obtain the sample text feature of the sample text, and comparative learning is performed based on the sample region feature and the sample text feature.

4. The method according to claim 3, characterized in that: The method further comprises: Acquiring a reference medical image and a reference report generated for the reference medical image; Inputting the reference medical image into the image encoding module to obtain a reference global feature, and generating a reference mask of any region of interest in the reference medical image, and inputting the reference global feature and the reference mask into the region encoding module to obtain a reference region feature; Acquire reference text for any region of interest from the reference report, and input the reference text into the text encoding module to obtain reference text features; The reference region feature and the reference text feature are stored in the reference feature database.

5. The method according to claim 1, characterized in that: The method further comprises: for each reference text, identifying the disease recorded in the reference text and generating a corresponding symptom label; The method of generating the analysis result of the target area of ​​interest of the medical image to be analyzed based on the reference texts corresponding to the k graphic features includes: obtaining the symptom labels of the reference texts corresponding to the k graphic features; if the ratio of the total number of occurrences of any symptom label to k is not less than a ratio threshold, then using the symptom label as the analysis result.

6. The method according to claim 1, characterized in that The method further comprises: Obtain reference texts and / or reference images corresponding to the k graphic features.

7. The method according to claim 1, characterized in that The method further comprises: Acquire symptom description text and extract symptom text features of the symptom description text, retrieve top k graphic features with the highest similarity to the symptom text features from a reference feature database, and acquire reference texts and / or reference images corresponding to the k graphic features.

8. A medical image-based analysis device, characterized in that: The device comprises: A region feature extraction module, used to extract target region features of a target region of interest in a medical image to be analyzed; A graphic feature retrieval module is used to retrieve the top k graphic features with the highest similarity to the target region feature from a reference feature database, where k is a positive integer, and the reference feature database stores reference region features extracted from a reference region of interest of a reference medical image and reference text features extracted from a reference text generated for the reference region of interest, and the graphic feature is the reference region feature or the reference text feature; The analysis result generating module is used to generate the analysis result of the target region of interest based on the reference texts corresponding to the k graphic features.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images