Medical image-based answer generation method, device, equipment and storage medium

Abstract

The present application provides a method, apparatus, device and storage medium for generating an answer based on medical images, the method comprising: obtaining a target question, obtaining a first period medical image and a second period medical image according to the target question; performing feature extraction on the medical image through a lesion extraction model to obtain a first lesion feature and a second lesion feature; performing feature matching on the first lesion feature and the second lesion feature to obtain lesion matching information; updating the first lesion feature, the second lesion feature and the text embedding feature based on a first attention mechanism to obtain a lesion map of the first period and a lesion map of the second period; performing a difference comparison between the lesion map of the first period and the lesion map of the second period based on the lesion matching information to obtain a lesion difference map; performing feature extraction on the lesion map of the first period, the lesion map of the second period and the lesion difference map based on a second attention mechanism to obtain visual features; and obtaining a corresponding target answer according to the visual features.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for generating answers based on medical images. Background Art

[0002] With the development of artificial intelligence, a large number of algorithms for intelligent detection and analysis of ultrasound lesions based on ultrasound images have emerged, achieving the purpose of using computers to assist in lesion detection in medical images. Currently, almost all intelligent image analysis algorithms are based on ultrasound images and ultrasound videos to analyze the current video to achieve the purpose of analyzing the current video, and the data dimension is relatively single.

[0003] However, during ultrasound review, doctors are still required to review the patient's previous ultrasound images, assess changes in lesions and evaluate the diagnosis and treatment process. This needs to be handled by experienced doctors, which is time-consuming and labor-intensive and cannot be fully automated. Summary of the invention

[0004] The present application provides a method, device, equipment and storage medium for generating answers based on medical images to at least solve the above technical problems existing in the prior art.

[0005] According to a first aspect of the present application, a method for generating an answer based on medical images is provided, the method comprising: obtaining a target question corresponding to the target object, and obtaining a first period medical image and a second period medical image corresponding to the target object according to the target question; performing feature extraction on the first period medical image and the second period medical image through a lesion extraction model to obtain a first lesion feature and a second lesion feature; performing feature matching on the first lesion feature and the second lesion feature to obtain lesion matching information; performing text embedding processing on the target question to obtain a text embedding feature; updating the first lesion feature, the second lesion feature and the text embedding feature based on a first attention mechanism to obtain a first period lesion map and a second period lesion map; performing a difference comparison between the first period lesion map and the second period lesion map based on the lesion matching information to obtain a lesion difference map; performing feature extraction on the first period lesion map, the second period lesion map and the lesion difference map based on a second attention mechanism to obtain visual features; generating an answer based on the visual features to obtain a target answer corresponding to the target question.

[0006] In one possible implementation, the step of extracting features from the medical images of the first period and the medical images of the second period using a lesion extraction model to obtain a first lesion feature and a second lesion feature includes: extracting features from the medical images of the first period using the lesion extraction model to obtain a first original feature, the lesion extraction model being obtained by training the feature extraction model using training samples; ranking the first original feature by confidence, and determining the first original feature whose ranking meets the confidence standard as the first lesion feature; extracting features from the medical images of the second period using the lesion extraction model to obtain a second original feature; and ranking the second original feature by confidence, and determining the second original feature whose ranking meets the confidence standard as the second lesion feature.

[0007] In one possible implementation manner, performing feature matching on the first lesion feature and the second lesion feature to obtain lesion matching information includes: performing lesion feature matching on the first lesion feature and the second lesion feature by using a scale-invariant feature transformation algorithm to obtain lesion matching information.

[0008] In one possible implementation, the first lesion feature, the second lesion feature and the text embedding feature are updated based on the first attention mechanism to obtain a first period lesion map and a second period lesion map, including: splicing the first lesion feature and the text embedding feature to obtain a first period splicing map, the first period splicing map includes multiple first graph nodes and first connecting edges; determining the first lesion weight and the first lesion frame weight corresponding to the first graph node; determining the first node weight corresponding to the first graph node according to the first lesion weight and the first lesion frame weight; updating the first graph node by the first node weight to obtain the first period lesion map; splicing the second lesion feature and the text embedding feature to obtain a second period splicing map, the second period splicing map includes multiple second graph nodes and second connecting edges; determining the second lesion weight and the second lesion frame weight corresponding to the second graph node; determining the second node weight corresponding to the second graph node according to the second lesion weight and the first lesion frame weight; updating the second graph node by the second node weight to obtain the second period lesion map.

[0009] In one possible implementation, the difference comparison of the first period lesion map and the second period lesion map based on the lesion matching information to obtain a lesion difference map includes: based on the lesion matching information, the difference comparison of the first update node and the corresponding second update node, the difference comparison of the first connecting edge and the corresponding second connecting edge to obtain a lesion difference map.

[0010] In one possible implementation, the feature extraction of the first period lesion map, the second period lesion map and the lesion difference map based on the second attention mechanism to obtain visual features includes: determining question-related weights based on text embedding features; updating the first period lesion map, the second period lesion map and the lesion difference map based on the question-related weights to obtain corresponding visual features.

[0011] In one possible implementation, generating an answer based on the visual features to obtain a target answer corresponding to the target question includes: determining a word class tag corresponding to the target answer; dynamically adjusting the visual features based on the word class tags to obtain adjusted features; and generating an answer based on the adjusted features by an answer generator to obtain a target answer corresponding to the target question.

[0012] According to a second aspect of the present application, a device for generating an answer based on medical images is provided, the device comprising: an acquisition module, used to acquire a target question corresponding to the target object, and acquire a first period medical image and a second period medical image corresponding to the target object according to the target question; an extraction module, used to perform feature extraction on the first period medical image and the second period medical image through a lesion extraction model to obtain a first lesion feature and a second lesion feature; a matching module, used to perform feature matching on the first lesion feature and the second lesion feature to obtain lesion matching information; an embedding module, used to perform text embedding processing on the target question to obtain a text embedding feature; an updating module, used to update the first lesion feature, the second lesion feature and the text embedding feature based on a first attention mechanism to obtain a first period lesion map and a second period lesion map; a comparison module, used to perform difference comparison on the first period lesion map and the second period lesion map based on the lesion matching information to obtain a lesion difference map; the extraction module is also used to perform feature extraction on the first period lesion map, the second period lesion map and the lesion difference map based on a second attention mechanism to obtain visual features; a generation module, used to generate an answer according to the visual features to obtain a target answer corresponding to the target question.

[0013] In one possible implementation, the extraction module includes: an extraction submodule, used to perform feature extraction on the medical images of the first period through the lesion extraction model to obtain a first original feature, wherein the lesion extraction model is obtained by training the feature extraction model through training samples; a ranking submodule, used to perform confidence ranking on the first original feature, and determine the first original feature whose ranking meets the confidence standard as the first lesion feature; the extraction submodule, also used to perform feature extraction on the medical images of the second period through the lesion extraction model to obtain a second original feature; the ranking submodule, also used to perform confidence ranking on the second original feature, and determine the second original feature whose ranking meets the confidence standard as the second lesion feature.

[0014] In one possible implementation, the matching module includes: performing lesion feature matching on the first lesion feature and the second lesion feature by using a scale-invariant feature transformation algorithm to obtain lesion matching information.

[0015] In one embodiment, the updating module includes: a splicing submodule, which is used to splice the first lesion feature and the text embedding feature to obtain a first period splicing graph, wherein the first period splicing graph includes multiple first graph nodes and first connecting edges; a determining submodule, which is used to determine the first lesion weight and the first lesion frame weight corresponding to the first graph node; the determining submodule is also used to determine the first node weight corresponding to the first graph node according to the first lesion weight and the first lesion frame weight; an updating submodule, which is used to update the first graph node by the first node weight to obtain a first period lesion graph; the splicing submodule is also used to splice the second lesion feature and the text embedding feature to obtain a second period splicing graph, wherein the second period splicing graph includes multiple second graph nodes and second connecting edges; the determining submodule is also used to determine the second lesion weight and the second lesion frame weight corresponding to the second graph node; the determining submodule is also used to determine the second node weight corresponding to the second graph node according to the second lesion weight and the second lesion frame weight; the updating submodule is also used to update the second graph node by the second node weight to obtain a second period lesion graph.

[0016] In one possible implementation, the comparison module includes: based on the lesion matching information, performing a difference comparison between the first update node and the corresponding second update node, and performing a difference comparison between the first connection edge and the corresponding second connection edge to obtain a lesion difference map.

[0017] In one possible implementation, the extraction module includes: determining a question-related weight based on text embedding features; and updating the first period lesion map, the second period lesion map, and the lesion difference map based on the question-related weight to obtain corresponding visual features.

[0018] In one possible implementation, the generation module includes: determining the word class tag corresponding to the target answer; dynamically adjusting the visual feature according to the word class tag to obtain an adjusted feature; and generating an answer according to the adjusted feature by an answer generator to obtain a target answer corresponding to the target question.

[0019] According to the third aspect of the present application, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the present application.

[0020] According to a fourth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the present application.

[0021] The embodiments of the present application provide a method, device, equipment and storage medium for generating answers based on medical images. The method extracts lesion features from the first period medical images and the second period medical images corresponding to the target object, fuses the corresponding first lesion features and the second lesion features with the text embedding features of the corresponding target question, and explores the relationship between the lesion features and the target question through the attention mechanism, thereby focusing on the features related to the target question, so as to accurately generate the target answer corresponding to the target question. It should be added that the target answer generated by the answer generation method of the present application is particularly suitable for answering questions related to comparing the differences in medical images of different periods.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become readily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:

[0024] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0025] Figure 1 The following is a schematic diagram showing the implementation process of a method for generating answers based on medical images in an embodiment of the present application. Figure 1 ;

[0026] Figure 2 A schematic diagram of a scenario flow of a method for generating an answer based on medical images according to an embodiment of the present application is shown;

[0027] Figure 3 The following is a schematic diagram showing the implementation process of a method for generating answers based on medical images in an embodiment of the present application. Figure 2 ;

[0028] Figure 4 A schematic diagram of implementation modules of a device for generating answers based on medical images according to an embodiment of the present application is shown;

[0029] Figure 5 A schematic diagram of the structure of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] In order to make the purpose, features, and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0031] Figure 1 The following is a schematic diagram showing the implementation process of a method for generating answers based on medical images in an embodiment of the present application. Figure 1 . Figure 2 A scenario flow diagram of a method for generating answers based on medical images according to an embodiment of the present application is shown.

[0032] See also Figure 1 and Figure 2According to the first aspect of the present application, a method for generating an answer based on medical images is provided, the method comprising: operation 101, obtaining a target question corresponding to a target object, and obtaining a first period medical image and a second period medical image corresponding to the target object according to the target question; operation 102, performing feature extraction on the first period medical image and the second period medical image through a lesion extraction model to obtain a first lesion feature and a second lesion feature; operation 103, performing feature matching on the first lesion feature and the second lesion feature to obtain lesion matching information; operation 104, performing text embedding processing on the target question to obtain a text embedding feature; operation 105, updating the first lesion feature, the second lesion feature and the text embedding feature based on a first attention mechanism to obtain a first period lesion map and a second period lesion map; operation 106, performing a difference comparison on the first period lesion map and the second period lesion map based on the lesion matching information to obtain a lesion difference map; operation 107, performing feature extraction on the first period lesion map, the second period lesion map and the lesion difference map based on a second attention mechanism to obtain a visual feature; operation 108, generating an answer based on the visual feature to obtain a target answer corresponding to the target question.

[0033] The medical image-based answer generation method provided in the embodiment of the present application extracts lesion features from the first period medical images and the second period medical images corresponding to the target object, fuses the corresponding first lesion features and the second lesion features with the text embedding features of the corresponding target question, and explores the relationship between the lesion features and the target question through the attention mechanism, thereby focusing on the visual features related to the target question, and accurately generating the target answer corresponding to the target question. It should be added that the target answer generated by the embodiment of the present application is particularly suitable for answering questions related to comparing the differences in medical images of different periods.

[0034] In operation 101 of the method, the method is applied to an electronic device with data processing capabilities, the target object can be a user, a patient or a tester, and the target question can be any image-related question related to medical imaging, such as "What is the echo type of the lesion in the image?" and other questions. Further, the target question of the present application can be a question related to the difference between specific tissues in different periods, and can be a question related to the change of specific tissues in images in different periods, such as "Compared with the past, are there any new or enlarged lesions at present?", "Compared with the past, has the structure (density, morphology, etc.) of the current breast tissue changed?", "Compared with the past, has the current lymph node area abnormally enlarged or changed in morphology?" and other questions. Correspondingly, according to the period specified by the target question, it is determined to obtain medical images of the target object in the first period and the second period. Among them, the second period can be used to characterize the current period, and the first period can be used to characterize the reference period, and the reference period is located before the current period.

[0035] For example, when the target question is "Compared to three months ago, is there any abnormal enlargement or morphological change in the current lymph node area?", the medical image of the first period is selected as the medical image of the lymph node area taken by the target object three months ago, and the medical image of the second period is selected as the medical image of the lymph node area taken by the target object at present. That is, the first period and the second period are usually used to characterize the medical image taken currently and the medical image taken earlier than the current one, respectively. It is understandable that the present application can preset the default interval between the first period and the second period based on prior knowledge, such as when the target question is "Compared to before, is there any abnormal enlargement or morphological change in the current lymph node area?" The device can default to the difference question related to the lymph node area, and the second period is selected as the medical image three months before the first period. The medical image is selected as an ultrasound image, and specifically can be selected as an ultrasound video.

[0036] In operation 102 of the present method, the lesion extraction model can be obtained by training a convolutional neural network model with training samples corresponding to the target lesions, and the convolutional neural network model can be selected as Faster-RCNN. The training samples include video samples corresponding to the same training object at different times. For example, when the target problem is a breast-related target problem, the training samples can be breast ultrasound videos of multiple objects before and after different periods, such as 500 pairs of breast ultrasound videos of 500 patients before and after different periods. Different problem contents can be covered in the ultrasound video, including but not limited to: lesion problems, structural problems, lymph node problems, cystic and solid lesions, blood flow problems, echo problems and lesions. In this way, the trained lesion extraction model can extract the first lesion features corresponding to the medical images of the first period and the second lesion features corresponding to the medical images of the second period.

[0037] In operations 103 and 104 of the method, the first lesion feature and the second lesion feature are matched one-to-one through a feature matching algorithm to obtain lesion matching information, which helps to construct a graph network with lesions as nodes in the next step to track the evolution of lesions in different periods.

[0038] By embedding the target question, the target question is encoded and the text embedding feature is obtained. Specifically, the question encoder can be used to learn the text embedding representation of the input target question. The question encoder can use the RNN Encoder-Decoder model. The question encoder is trained by using multiple questions corresponding to the ultrasound video as training samples. The training samples cover different question types, including but not limited to: lesion change question type, structural change question type, lymph node change question type, cystic and solid lesion question type, blood flow condition question type, echo question type and lesion progression question type.

[0039] In operation 105 of the method, each lesion feature is concatenated with the text embedding feature to be represented as a graph node, and the graph node is updated through the first attention mechanism to capture the relationship between the lesion feature and the target problem, and to construct a first period lesion map corresponding to the first period and a second period lesion map corresponding to the second period, respectively.

[0040] In operations 106 and 107 of the present method, the lesion matching information is used to compare the differences between the lesion map of the first period and the lesion map of the second period to obtain a lesion difference map, and then based on the second attention mechanism, the model focuses on the features related to the target problem in the lesion map of the first period, the lesion map of the second period, and the lesion difference map to obtain a visual feature vector.

[0041] In operation 108 of the present method, the visual feature vector is input into the answer generator for prediction and generation, and a target answer corresponding to the target question is obtained. In this way, comparative analysis of medical images of different periods can be achieved, and the target answer can be accurately output.

[0042] In one possible implementation, operation 102, performing feature extraction on the medical images of the first period and the medical images of the second period through a lesion extraction model to obtain a first lesion feature and a second lesion feature, includes: first, performing feature extraction on the medical images of the first period through the lesion extraction model to obtain a first original feature, and the lesion extraction model is obtained by training the feature extraction model through training samples; then, performing confidence ranking on the first original feature, and determining the first original feature whose ranking meets the confidence standard as the first lesion feature; then, performing feature extraction on the medical images of the second period through the lesion extraction model to obtain a second original feature; thereafter, performing confidence ranking on the second original feature, and determining the second original feature whose ranking meets the confidence standard as the second lesion feature.

[0043] This method uses 1600 breast ultrasound videos collected on Faster RCNN to train a lesion detection model as the backbone. The pre-trained Faster RCNN is used to extract lesion features from the breast ultrasound videos of the target object in the current period and the reference period, and the first original features corresponding to the first period and the first confidence value corresponding to each first original feature are obtained. Then the first confidence values ​​are sorted from large to small, and the top k first original features are determined as the first lesion features. The first lesion features are used to characterize the candidate lesions in the medical image. Similarly, the second original features corresponding to the second period and the second confidence values ​​corresponding to each second original feature are also obtained. The second confidence values ​​are sorted from large to small, and the top k second original features are determined as the second lesion features. The number of second lesion features is consistent with that of the first lesion features, so that the first lesion features and the second lesion features can be one-to-one corresponded.

[0044] In one possible implementation, operation 103, performing feature matching on the first lesion feature and the second lesion feature to obtain lesion matching information, includes: performing lesion feature matching on the first lesion feature and the second lesion feature by using a scale-invariant feature transformation algorithm to obtain lesion matching information.

[0045] The present application adopts the Scale Invariant Feature Transform (SIFT) algorithm as a feature matching algorithm to perform one-to-one matching between the first lesion feature and the second lesion feature, which helps to construct a graph network with lesions as nodes and track the evolution of the same lesion.

[0046] Figure 3 A schematic diagram of the implementation flow of a method for generating an answer based on medical images according to an embodiment of the present application is shown.

[0047] refer to Figure 3In one possible implementation, operation 105 is to update the first lesion feature, the second lesion feature and the text embedding feature based on the first attention mechanism to obtain a first period lesion map and a second period lesion map, including: operation 1051, splicing the first lesion feature and the text embedding feature to obtain a first period splicing map, the first period splicing map including a plurality of first graph nodes and first connecting edges; operation 1052, determining a first lesion weight and a first lesion frame weight corresponding to the first graph node; operation 1053, determining a first node weight corresponding to the first graph node according to the first lesion weight and the first lesion frame weight; Operation 1054, updating the first graph node by the first node weight to obtain the lesion map of the first period; operation 1055, splicing the second lesion feature and the text embedding feature to obtain the spliced ​​graph of the second period, the spliced ​​graph of the second period includes multiple second graph nodes and second connecting edges; operation 1056, determining the second lesion weight and the second lesion frame weight corresponding to the second graph node; operation 1057, determining the second node weight corresponding to the second graph node according to the second lesion weight and the second lesion frame weight; operation 1058, updating the second graph node by the second node weight to obtain the lesion map of the second period.

[0048] The embodiment of the present application represents the concatenation of each lesion feature and the text embedding feature as a graph node, and determines the corresponding connection edge. The first period splicing graph is formed by the first graph node and the first connection edge corresponding to the first period, and there are k first graph nodes. The second period splicing graph is formed by the second graph node and the second connection edge corresponding to the second period, and there are k second graph nodes. The first attention mechanism is specifically a multi-head graph attention mechanism, which captures the relationship between the lesion feature and the target problem through the multi-head graph attention mechanism. Each graph node v i Update according to the following formula:

[0049]

[0050] in,

[0051] v i Used to represent the updated i-th update node, v i ∈R d , v i is a vector belonging to d-dimensional space;

[0052] v j Used to represent the jth updated node after the update;

[0053] N i Used to represent a set of graph nodes;

[0054] W oIt is used to represent the projection matrix of dimension d×Md, where M is used to represent the number of heads of multi-head attention, that is, to execute M attention mechanisms;

[0055] W m Used to represent the dimension d×(d n +d q ), where

[0056] d n Used to characterize the lesion feature dimension, d q Used to characterize text embedding feature dimensions;

[0057] (d n +d q ) is used to characterize the splicing of lesion features and embedded sign dimensions.

[0058] α ij Used to represent the node weight, indicating the association between graph node i and graph node j, α ij The specific formula is as follows:

[0059]

[0060] in,

[0061] Used to characterize the lesion weight, It is used to characterize the weight of the lesion box, and K is used to characterize the number of graph nodes;

[0062] Lesion weight The specific formula is as follows:

[0063]

[0064] in,

[0065] U is a d×(d n +d q ) is the projection matrix, V is a matrix with dimensions d×(d n +d q )’s projection matrix;

[0066] Lesion box weight The specific formula is as follows:

[0067]

[0068] in,

[0069] w∈R d , w is used to represent the weight of a d-dimensional vector, which is used to transform f b (b ij ) into a scalar value;

[0070] f b Used to characterize the transformation of 4-dimensional coordinates into d dimensions;

[0071] b ij is the relative geometric feature of nodes i and j. The specific formula is as follows:

[0072]

[0073] (x i ,y i ,) is used to represent the coordinates of the lesion box corresponding to node i, w i It is used to represent the width of the lesion box corresponding to node i, h i Used to represent the height of the lesion box corresponding to node i;

[0074] (x j ,y j ,) is used to represent the coordinates of the lesion box corresponding to node j, w j It is used to represent the width of the lesion box corresponding to node j, h j Used to characterize the height of the lesion box corresponding to node j.

[0075] Thus, the node weight α ij Not only depends on the lesion weight corresponding to the lesion feature It also depends on the lesion box weight corresponding to the lesion feature The node weight is obtained by integrating the lesion weight and the lesion frame weight, and each graph node is updated by the node weight to obtain an updated update node.

[0076] By processing each first graph node of the first period splicing graph by the above method, an updated first period lesion graph V can be obtained. cur ∈R k×d By processing each second graph node of the second period splicing graph in the above method, an updated second period lesion graph V can be obtained. bef ∈R k×d Among them, R k×d To characterize V bef is a graph with k nodes and d edges.

[0077] The operations of obtaining the first period lesion map and the second period lesion map in this method are independent of each other and do not affect each other, so there is no specific order of operation 1051 to operation 1054 and operation 1055 to operation 1058, and they can be performed simultaneously or in steps.

[0078] In one possible implementation, operation 106 performs a difference comparison between the lesion map of the first period and the lesion map of the second period based on the lesion matching information to obtain a lesion difference map, including: based on the lesion matching information, performing a difference comparison between the first update node and the corresponding second update node, and performing a difference comparison between the first connecting edge and the corresponding second connecting edge to obtain a lesion difference map.

[0079] Through the lesion matching information, the first update node and the corresponding second update node can be matched one by one in the lesion map of the first period and the lesion map of the second period, and the lesion map of the second period is subtracted from the lesion map of the first period, that is, the lesion map of the reference period is subtracted from the lesion map of the current period, specifically, the nodes in the lesion map are subtracted, and the edges are subtracted, so as to obtain the lesion difference map V cur , the specific node subtraction formula is as follows:

[0080]

[0081] Used to represent the i-th difference node in the lesion difference graph.

[0082] The formula for subtracting edges is the same as the formula for subtracting nodes, and will not be repeated below.

[0083] In one possible implementation, operation 107 performs feature extraction on the first period lesion map, the second period lesion map, and the lesion difference map based on the second attention mechanism to obtain visual features, including: first, determining question-related weights based on text embedding features; then, updating the first period lesion map, the second period lesion map, and the lesion difference map based on the question-related weights to obtain corresponding visual features.

[0084] Next, the generated first-period lesion map, second-period lesion map, and lesion difference map are sent to the graph feature attention module (Graph Attention), so that the model focuses on the features related to the target problem.

[0085] Taking the second period lesion image input graph feature attention module as an example, the second visual attention weight a of the graph feature attention module cur The processing formula is as follows:

[0086]

[0087] Among them, C cur ∈R k×d ;

[0088] V cur Used to characterize the lesion map of the second period;

[0089] middle, represents the weight matrix, and c represents the calculation C cur The weight at the time of , q is the q in the definition of the attention mechanism.

[0090] In the above example, v is used to represent v in the attention mechanism definition; b c ∈R d , b c is the bias, the purpose of which is to adjust the position of the function on the x-axis.

[0091]

[0092] Among them, A cur ∈R k×d ;

[0093] b a ∈R d , b a is the bias, the purpose of which is to adjust the position of the function on the x-axis.

[0094] By C cur and A cur Integrated calculation of the second visual attention weight a cur , the calculation formula is as follows:

[0095] C′ cur =A cur ⊙C cur

[0096] a cur =sigmoid(FC2(ReLU(FC1([V cur ||V diff ||C′ cur ]))))

[0097] Among them, || represents splicing, a cur ∈R k .

[0098] Correspondingly, through the second visual attention weight a cur The second visual feature vector obtained by processing the lesion image of the second period is represented as:

[0099]

[0100] Similarly, the first visual attention weight a can be calculated using the same formula bef , through the first visual attention weight a bef The visual feature vector of the first period corresponding to the first lesion period map is obtained, which is represented as:

[0101]

[0102] The difference visual feature vector corresponding to the lesion difference map is represented as:

[0103] G diff =G cur -G bef

[0104] The visual features referred to in the embodiments of the present application include a first period visual feature vector, a second period visual feature vector, and a difference visual feature vector.

[0105] In one possible implementation, operation 108 generates an answer based on the visual features to obtain a target answer corresponding to the target question, including: first, determining the word class tag corresponding to the target answer; then, dynamically adjusting the visual features according to the word class tag to obtain an adjusted feature; and then, generating an answer based on the adjusted feature by an answer generator to obtain the target answer corresponding to the target question.

[0106] The generated first-period visual feature vector, second-period visual feature vector and difference visual feature vector are input into the answer generator (Answers Predictor), and the target answer corresponding to the target question is predicted based on the lesions and their differences.

[0107] First, based on the previous hidden state of the answer generator Dynamically predict the part-of-speech tag (POStag) of the target answer. The POS tags of the words in the ground truth can be obtained according to existing methods, such as using the Stanford Log-linear Part-Of-Speech Tagger method to obtain the POStags of the words in the ground truth. The predicted POS tags can help the answer generator dynamically adjust the visual features. Specifically, at time t, First input to a hidden layer:

[0108]

[0109]

[0110] in, are learnable parameters.

[0111] Therefore, we can get the POS tag of the word in the target answer:

[0112]

[0113] Among them, E p is the learnable POS embedding matrix.

[0114] Next, we use the visual attention LSTM v From G cur ,G bef ,G diff Select a candidate feature that is relevant to the target answer:

[0115]

[0116] i∈(cur,bef,diff), is the visual attention weight at the current time t, which is calculated according to the following steps:

[0117]

[0118]

[0119]

[0120]

[0121] in, is a learnable parameter, It is LSTM v The output at time step t is, is the answer generator LSTM a Output at time step t-1.

[0122] Next, design a switch β t , based on the calculated pt, decide whether to rely on visual features to predict the next word in the answer.

[0123] When at time t,

[0124] in, Used to represent the weight matrix.

[0125] So the final adjusted features are calculated as follows:

[0126]

[0127] where β t The value range of is [0,1], which indicates the degree to which visual features are used when predicting the target word.

[0128] After the above steps, the appropriate visual features are obtained, and finally LSTM is used a To generate answers word by word. For time step t, and the word w at step t-1 t-1 , labels during training, predicted words during inference testing, input into LSTM together aTo predict the next word:

[0129]

[0130]

[0131]

[0132] Where E is the word embedding matrix, W c ,b c is a learnable parameter.

[0133] The above is the overall process of generating the target answer through the answer generator, by maximizing the word sequence of the predicted answer and part-of-speech sequences The likelihood of is used to train the model end-to-end, so the training goal is to learn the parameters by minimizing its negative log-likelihood loss, so the loss function is designed as follows:

[0134]

[0135] In the loss function, m represents the length of the words in the answer, θ p ,θ c They represent the parameters of the part-of-speech predictor and the word predictor respectively, and α and β are the parameters obtained through training.

[0136] Through the above method, we can automate visual question answering based on ultrasound videos of two periods. Given a pair of ultrasound lesion videos of the same target object, this method can answer questions about lesion properties, progression, etc. by comparing the differences through model calculation and analysis. This greatly facilitates the doctor's diagnosis process and improves the efficiency and accuracy of image comparison and answer output.

[0137] Figure 4 A schematic diagram of implementation modules of a medical image-based answer generation device according to an embodiment of the present application is shown.

[0138] See also Figure 4According to the second aspect of the present application, a device for generating an answer based on medical images is provided, and the device includes: an acquisition module 401, which is used to acquire a target question corresponding to a target object, and acquire a first period medical image and a second period medical image corresponding to the target object according to the target question; an extraction module 402, which is used to perform feature extraction on the first period medical image and the second period medical image through a lesion extraction model to obtain a first lesion feature and a second lesion feature; a matching module 403, which is used to perform feature matching on the first lesion feature and the second lesion feature to obtain lesion matching information; an embedding module 404, which is used to perform text embedding processing on the target question , and obtain text embedding features; an updating module 405 is used to update the first lesion feature, the second lesion feature and the text embedding feature based on the first attention mechanism to obtain a first period lesion map and a second period lesion map; a comparing module 406 is used to compare the first period lesion map and the second period lesion map based on the lesion matching information to obtain a lesion difference map; the extracting module 402 is also used to extract features from the first period lesion map, the second period lesion map and the lesion difference map based on the second attention mechanism to obtain visual features; a generating module 407 is used to generate answers according to the visual features to obtain target answers to corresponding target questions.

[0139] In one embodiment, the extraction module 402 includes: an extraction submodule 4021, which is used to perform feature extraction on medical images of the first period through a lesion extraction model to obtain a first original feature, and the lesion extraction model is obtained by training the feature extraction model through training samples; a ranking submodule 4022, which is used to perform confidence ranking on the first original feature, and determine the first original feature whose ranking meets the confidence standard as the first lesion feature; the extraction submodule 4021 is also used to perform feature extraction on medical images of the second period through the lesion extraction model to obtain a second original feature; the ranking submodule 4022 is also used to perform confidence ranking on the second original feature, and determine the second original feature whose ranking meets the confidence standard as the second lesion feature.

[0140] In one possible implementation, the matching module 403 includes: performing lesion feature matching on the first lesion feature and the second lesion feature by using a scale-invariant feature transformation algorithm to obtain lesion matching information.

[0141] In one embodiment, the updating module 405 includes: a splicing submodule 4051, which is used to splice the first lesion feature and the text embedding feature to obtain a first period splicing graph, wherein the first period splicing graph includes multiple first graph nodes and first connecting edges; a determining submodule 4052, which is used to determine the first lesion weight and the first lesion frame weight corresponding to the first graph node; the determining submodule 4052 is also used to determine the first node weight corresponding to the first graph node according to the first lesion weight and the first lesion frame weight; an updating submodule 4053, which is used to update the first graph node by the first node weight to obtain to the lesion map of the first period; the splicing submodule 4051 is also used to splice the second lesion feature and the text embedding feature to obtain the splicing map of the second period, and the second period splicing map includes multiple second graph nodes and second connecting edges; the determination submodule 4052 is also used to determine the second lesion weight and the second lesion frame weight corresponding to the second graph node; the determination submodule 4052 is also used to determine the second node weight corresponding to the second graph node according to the second lesion weight and the second lesion frame weight; the updating submodule 4053 is also used to update the second graph node by the second node weight to obtain the lesion map of the second period.

[0142] In one possible implementation, the comparison module 406 includes: based on the lesion matching information, performing a difference comparison between the first updated node and the corresponding second updated node, and performing a difference comparison between the first connecting edge and the corresponding second connecting edge to obtain a lesion difference map.

[0143] In one possible implementation, the extraction module 402 includes: determining a question-related weight based on text embedding features; and updating the first period lesion map, the second period lesion map, and the lesion difference map based on the question-related weight to obtain corresponding visual features.

[0144] In one possible implementation, generation module 407 includes: determining a word class tag corresponding to a target answer; dynamically adjusting visual features according to the word class tag to obtain an adjusted feature; and generating an answer according to the adjusted feature by an answer generator to obtain a target answer to the target question.

[0145] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.

[0146] Figure 5A schematic block diagram of an example electronic device 500 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0147] like Figure 5 As shown, the device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0148] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0149] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above, such as a method for generating answers based on medical images. For example, in some embodiments, a method for generating answers based on medical images may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method for generating answers based on medical images described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute a medical image-based answer generation method in any other appropriate manner (eg, by means of firmware).

[0150] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0151] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0152] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0154] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0155] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0156] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0157] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0158] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for generating answers based on medical images, characterized in that: The method comprises: Acquire a target question corresponding to a target object, and acquire a first-period medical image and a second-period medical image corresponding to the target object according to the target question; Extracting features from the medical images of the first period and the medical images of the second period using a lesion extraction model to obtain first lesion features and second lesion features; Performing feature matching on the first lesion feature and the second lesion feature to obtain lesion matching information; Performing text embedding processing on the target question to obtain text embedding features; Based on the first attention mechanism, the first lesion feature, the second lesion feature and the text embedding feature are updated to obtain a lesion map for the first period and a lesion map for the second period; Comparing the lesion image of the first period with the lesion image of the second period based on the lesion matching information, obtaining a lesion difference image; Based on the second attention mechanism, feature extraction is performed on the lesion map of the first period, the lesion map of the second period, and the lesion difference map to obtain visual features; An answer is generated according to the visual features to obtain a target answer corresponding to the target question.

2. The method according to claim 1, characterized in that The extracting features of the medical images of the first period and the medical images of the second period by using the lesion extraction model to obtain the first lesion feature and the second lesion feature includes: Extracting features from the medical images of the first period by using the lesion extraction model to obtain first original features, wherein the lesion extraction model is obtained by training the feature extraction model with training samples; Rank the first original features according to their confidence level, and determine the first original features whose rankings meet the confidence level standard as the first lesion features; Extracting features from the medical images in the second period using the lesion extraction model to obtain second original features; The second original features are ranked according to confidence, and the second original features whose rankings meet the confidence standard are determined as the second lesion features.

3. The method according to claim 1, characterized in that The performing feature matching on the first lesion feature and the second lesion feature to obtain lesion matching information includes: Lesion feature matching is performed on the first lesion feature and the second lesion feature using a scale-invariant feature transformation algorithm to obtain lesion matching information.

4. The method according to claim 1, characterized in that The updating of the first lesion feature, the second lesion feature and the text embedding feature based on the first attention mechanism to obtain a lesion map for the first period and a lesion map for the second period includes: Splicing the first lesion feature and the text embedding feature to obtain a first period splicing graph, wherein the first period splicing graph includes a plurality of first graph nodes and first connecting edges; Determine a first lesion weight and a first lesion frame weight corresponding to the first graph node; Determine a first node weight corresponding to the first graph node according to the first lesion weight and the first lesion frame weight; The first graph nodes are updated by using the first node weights to obtain a lesion graph for the first period; Splicing the second lesion feature and the text embedding feature to obtain a second period splicing graph, wherein the second period splicing graph includes a plurality of second graph nodes and second connecting edges; Determine a second lesion weight and a second lesion frame weight corresponding to the second graph node; Determine a second node weight corresponding to the second graph node according to the second lesion weight and the second lesion frame weight; The second graph nodes are updated by the second node weights to obtain a lesion map for the second period.

5. The method according to claim 4, characterized in that The step of comparing the difference between the first period lesion map and the second period lesion map based on the lesion matching information to obtain a lesion difference map comprises: Based on the lesion matching information, a first update node and a corresponding second update node are compared for difference, and a first connection edge and a corresponding second connection edge are compared for difference to obtain a lesion difference map.

6. The method according to claim 1, characterized in that The extracting features of the lesion map of the first period, the lesion map of the second period and the lesion difference map based on the second attention mechanism to obtain visual features includes: Determine question-related weights based on text embedding features; The first period lesion map, the second period lesion map and the lesion difference map are updated based on the question-related weights to obtain corresponding visual features.

7. The method according to claim 1, characterized in that Generating an answer according to the visual feature to obtain a target answer corresponding to the target question includes: Determine a part-of-speech tag corresponding to the target answer; Dynamically adjusting the visual feature according to the word class label to obtain an adjusted feature; The answer generator generates an answer according to the adjustment feature to obtain a target answer corresponding to the target question.

8. A device for generating answers based on medical images, characterized in that: The device comprises: A question acquisition module, used to acquire a target question corresponding to a target object, and acquire a first-period medical image and a second-period medical image corresponding to the target object according to the target question; A feature extraction module, used to extract features from the medical images of the first period and the medical images of the second period by using a lesion extraction model to obtain first lesion features and second lesion features; A feature matching module, used for performing feature matching on the first lesion feature and the second lesion feature to obtain lesion matching information; A text embedding module, used to perform text embedding processing on the target question to obtain text embedding features; A feature updating module, configured to update the first lesion feature, the second lesion feature and the text embedding feature based on a first attention mechanism to obtain a lesion map for a first period and a lesion map for a second period; A difference comparison module is used to compare the lesion image of the first period and the lesion image of the second period based on the lesion matching information to obtain a lesion difference image; The feature extraction module is further used to extract features from the lesion image of the first period, the lesion image of the second period, and the lesion difference image based on a second attention mechanism to obtain visual features; The answer generation module is used to generate answers according to the visual features to obtain a target answer corresponding to the target question.

9. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-7.

Images