Explanation information generation method, apparatus, device, medium, and program product

Abstract

The application relates to an interpretation information generation method, device, equipment, medium and program product. The method comprises the following steps: acquiring image semantic information corresponding to a to-be-tested image; the image semantic information is used for representing feature information related to a semantic description of a region of interest in the to-be-tested image; determining an image processing result corresponding to the to-be-tested image by using a preset image processing network; generating interpretation information corresponding to the to-be-tested image according to the image semantic information, the image processing result and a preset language model; the language model is trained based on reference interpretation information corresponding to each training image; and the interpretation information is used for representing related information on which the image processing network depends when determining the image processing result. The method can provide auxiliary information for doctors, so that the doctors can accurately perform a subsequent image analysis process.

Description

Technical Field

[0001] This application relates to the field of medical technology, and in particular to a method, apparatus, device, medium, and program product for generating interpretable information. Background Technology

[0002] With the continuous development of artificial intelligence technology, it has been widely applied in the processing of medical images. For example, artificial intelligence technology is added to the processing of lesion segmentation, lesion detection, and lesion classification in medical images, so that relatively accurate lesion segmentation results, lesion detection results, and lesion classification results can be obtained quickly.

[0003] Taking the lesion classification process in medical image processing as an example, a classification model is generally used to classify lesions in medical images. Specifically, the model can be trained in advance using training images with labeled lesions. After training is completed, the trained classification model is used to perform intensive calculations on the test images to classify the lesions and obtain the classification results.

[0004] However, for inexperienced doctors, relying solely on the classification results output by the aforementioned classification model for subsequent image analysis may affect the accuracy of the subsequent image analysis process. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, device, medium, and program product that can provide doctors with auxiliary information to accurately perform subsequent image analysis.

[0006] Firstly, this application provides a method for generating explanatory information, the method comprising:

[0007] Obtain the image semantic information corresponding to the image to be tested; the above image semantic information is used to characterize the feature information in the image to be tested that is related to the semantic description of the region of interest.

[0008] A preset image processing network is used to determine the image processing result corresponding to the image under test.

[0009] Based on the aforementioned image semantic information, image processing results, and a pre-defined language model, explanatory information corresponding to the image to be tested is generated. The aforementioned language model is trained based on the reference explanatory information corresponding to each training image. The aforementioned explanatory information is used to characterize the relevant information that the image processing network uses to determine the image processing results.

[0010] In one embodiment, the above-mentioned generation of explanatory information corresponding to the image under test based on image semantic information, image processing results, and a preset language model includes:

[0011] Based on the image processing results, determine the guidance question information related to the image processing results;

[0012] Based on the semantic information of the image, the guiding question information, and the preset language model, the explanatory information corresponding to the image to be tested is generated; the aforementioned guiding question information is used to guide the language model to generate relevant information for the image processing results.

[0013] In one embodiment, the above-mentioned generation of explanatory information corresponding to the image to be tested based on image semantic information, guiding question information, and a preset language model includes:

[0014] Based on the guidance question information and the language model, determine the guidance question embedding vector corresponding to the guidance question information;

[0015] The semantic information of the image is mapped to the text information space to determine the image feature embedding vector corresponding to the image under test.

[0016] Image feature embedding vectors and guiding question embedding vectors are input into a language model for text generation processing to obtain explanatory information.

[0017] In one embodiment, the above-mentioned mapping of image semantic information to text information space to determine the image feature embedding vector corresponding to the image to be tested includes:

[0018] Image semantic information and guidance question embedding vectors are input into a pre-defined fusion model for fusion processing to determine image feature embedding vectors;

[0019] The fusion model described above is trained based on the reference interpretation information corresponding to each training image.

[0020] In one embodiment, the training method of the above-mentioned fusion model includes:

[0021] Determine the semantic information of the predicted image and the embedding vector of the prediction guidance question for each training image;

[0022] Determine the embedding vector of the interpretation information corresponding to each reference interpretation information;

[0023] Each reference interpretation information, each predicted image semantic information and each prediction guidance question embedding vector are input into the initial fusion model for fusion processing to determine the prediction image feature embedding vector corresponding to each training image.

[0024] The initial fusion model is trained based on the feature embedding vectors of each predicted image and the corresponding explanatory information embedding vectors to determine the fusion model.

[0025] In one embodiment, the above-mentioned inputting each reference interpretation information, each predicted image semantic information, and each prediction guidance question embedding vector into an initial fusion model for fusion processing to determine the predicted image feature embedding vector corresponding to each training image includes:

[0026] Each reference interpretation information and each prediction guidance question embedding vector are input into the initial fusion model, and self-attention fusion processing is performed on each reference interpretation information and each prediction guidance question embedding vector to obtain each prediction fusion information;

[0027] The semantic information of each predicted image is input into the initial fusion model, and interactive attention fusion processing is performed on the semantic information of each predicted image and the fusion information of each prediction to obtain the feature embedding vector of each predicted image.

[0028] In one embodiment, training the initial fusion model based on each predicted image feature embedding vector and the corresponding explanatory information embedding vector includes:

[0029] Calculate the matching loss between the corresponding training image and the reference interpretation information based on the feature embedding vector of each predicted image;

[0030] Calculate the feature similarity loss between the feature embedding vectors of each predicted image and the corresponding explanatory information embedding vectors;

[0031] The initial fusion model is trained based on matching loss and feature similarity loss.

[0032] In one embodiment, the language model includes an additional weight layer; the additional weight layer is used to instruct the language model on relevant information for generating image processing results.

[0033] In one embodiment, obtaining the image semantic information corresponding to the image to be tested includes:

[0034] Acquire the image to be tested;

[0035] The image to be tested is input into a preset image encoder to extract the semantic information of the image.

[0036] Secondly, this application also provides an explanatory information generation apparatus, the apparatus comprising:

[0037] The semantic information determination module is used to obtain the image semantic information corresponding to the image to be tested; the aforementioned image semantic information is used to characterize the feature information in the image to be tested that is related to the semantic description of the region of interest.

[0038] The processing result determination module is used to determine the image processing result corresponding to the image under test using a preset image processing network.

[0039] The explanation information generation module is used to generate explanation information corresponding to the image under test based on the above image semantic information, image processing results and preset language model; the above language model is trained based on the reference explanation information corresponding to each training image, and the above explanation information is used to characterize the relevant information used by the image processing network to determine the image processing results.

[0040] Thirdly, this application also provides a computer device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0041] Obtain the image semantic information corresponding to the image to be tested; the above image semantic information is used to characterize the feature information in the image to be tested that is related to the semantic description of the region of interest.

[0042] A preset image processing network is used to determine the image processing result corresponding to the image under test.

[0043] Based on the aforementioned image semantic information, image processing results, and a pre-defined language model, explanatory information corresponding to the image to be tested is generated. The aforementioned language model is trained based on the reference explanatory information corresponding to each training image. The aforementioned explanatory information is used to characterize the relevant information that the image processing network uses to determine the image processing results.

[0044] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0045] Obtain the image semantic information corresponding to the image to be tested; the above image semantic information is used to characterize the feature information in the image to be tested that is related to the semantic description of the region of interest.

[0046] A preset image processing network is used to determine the image processing result corresponding to the image under test.

[0047] Based on the aforementioned image semantic information, image processing results, and a pre-defined language model, explanatory information corresponding to the image to be tested is generated. The aforementioned language model is trained based on the reference explanatory information corresponding to each training image. The aforementioned explanatory information is used to characterize the relevant information that the image processing network uses to determine the image processing results.

[0048] Fifthly, this application also provides a computer program product, which includes a computer program that, when executed by a processor, performs the following steps:

[0049] Obtain the image semantic information corresponding to the image to be tested; the above image semantic information is used to characterize the feature information in the image to be tested that is related to the semantic description of the region of interest.

[0050] A preset image processing network is used to determine the image processing result corresponding to the image under test.

[0051] Based on the aforementioned image semantic information, image processing results, and a pre-defined language model, explanatory information corresponding to the image to be tested is generated. The aforementioned language model is trained based on the reference explanatory information corresponding to each training image. The aforementioned explanatory information is used to characterize the relevant information that the image processing network uses to determine the image processing results.

[0052] The aforementioned method, apparatus, device, medium, and program products for generating explanatory information acquire image semantic information corresponding to the image under test and determine the image processing result corresponding to the image under test using a preset image processing network. Then, explanatory information corresponding to the image under test is generated based on the image semantic information, the image processing result, and a preset language model. The image semantic information characterizes the features related to the semantic description of the region of interest in the image under test. The language model is trained based on the reference explanatory information corresponding to each training image, and the generated explanatory information characterizes the relevant information used by the image processing network to determine the image processing result. In this method, since explanatory information for the image under test can be generated from the image semantic information, the image processing result, and the language model, and this explanatory information is the relevant information used by the image processing network to obtain the image processing result, less experienced doctors can combine this explanatory information with the image under test for comprehensive analysis when referring to the image processing result. This is more intuitive and can assist doctors in the image analysis process, ensuring the accuracy of the image analysis process. Furthermore, since the generated explanatory information is produced by a language model trained with reference explanatory information, combined with image processing results and information that can characterize the semantic descriptive features of the region of interest, the generated explanatory information is closely related to the region of interest on the image under test and the image processing results. It is more in line with the actual situation and more accurate. This also ensures that the information referenced by doctors when performing image analysis is very accurate, further ensuring the accuracy of the image analysis process. Attached Figure Description

[0053] Figure 1 This is an internal structural diagram of a computer device in one embodiment;

[0054] Figure 2 This is a flowchart illustrating the information generation method in one embodiment;

[0055] Figure 3 This is a flowchart illustrating the information generation method in another embodiment;

[0056] Figure 4 This is a flowchart illustrating the information generation method in another embodiment;

[0057] Figure 5 This is a flowchart illustrating the information generation method in another embodiment;

[0058] Figure 6 This is a detailed flowchart illustrating the information generation process in another embodiment;

[0059] Figure 7 This is a structural block diagram illustrating the information generation device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] Artificial intelligence (AI) technology is now widely used in medical image processing. However, in these applications, results based on AI models are often directly presented. These results are simply judgments made by the AI ​​model after learning from labeled data, lacking an intuitive explanation of the model's assessment. In clinical medical image analysis, analyzing a lesion requires a comprehensive assessment of multiple factors, such as the lesion's morphology, density, and location. A single different indicator can lead to completely different classification conclusions. For example, a calcified nodule is benign, even if its location and morphology are similar to malignant nodules of other densities. Because of its calcification, it can be classified as a benign lesion. While the training data for classification models implicitly includes relevant classification criteria, directly presenting classification conclusions from the model can still confuse users. For instance, inexperienced doctors relying solely on the classification results from these models for subsequent image analysis may compromise the accuracy of the analysis. Based on this, embodiments of this application provide an explanation information generation method, apparatus, device, medium, and program product that can solve the above-mentioned technical problems.

[0062] The explanation information generation method provided in this application embodiment can be applied to computer devices, which can be terminals or servers. Taking a terminal as an example, its internal structure diagram can be as follows: Figure 1As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an information generation method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0063] Those skilled in the art will understand that Figure 1 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0064] In one embodiment, such as Figure 2 As shown, an explanatory information generation method is provided, which can be applied to... Figure 1 Taking a computer device as an example, the method may include the following steps:

[0065] S202, Obtain the image semantic information corresponding to the image to be tested; the above image semantic information is used to characterize the feature information related to the semantic description of the region of interest in the image to be tested.

[0066] The image to be tested can be a two-dimensional image or a three-dimensional image; the image to be tested can be a medical image of the part of the object to be tested, and the image to be tested can generally include a region of interest, which can be a lesion area or abnormal area in the part of the object to be tested corresponding to the image to be tested, and there can be one or more regions of interest in the image to be tested.

[0067] In this step, when obtaining the image semantic information corresponding to the image to be tested, it is optional to obtain the image to be tested; input the image to be tested into a preset image encoder to extract the semantic information in the image to be tested.

[0068] Specifically, before acquiring semantic information from an image, the image to be tested can be acquired in advance. The acquisition method for the image to be tested can be to scan the part of the object to be tested and reconstruct the image from the scan data to obtain the image to be tested, or it can be to obtain the image to be tested that is pre-stored in the cloud or server, or it can be other acquisition methods, which are not specifically limited here.

[0069] The image encoder here can be a pre-trained and initialized model, such as a Transformer model or a CNN (Convolutional Neural Networks) model. This image encoder primarily extracts semantic information from the input image; specifically, it encodes the input image into a fixed-dimensional vector [v0, v1, v2, ... v...]. n ].

[0070] After obtaining the image to be tested, it can be input into a pre-trained image encoder for semantic information extraction. The extracted semantic information consists of feature information on the image, which may include feature information from one or more regions of interest (ROIs), or feature information from other regions. Ultimately, the semantic information corresponding to the image to be tested is obtained. The feature information related to the semantic description of the ROI mainly refers to the image semantic information describing the ROI. For example, if the semantic description of the ROI is a ground-glass appearance, then the extracted feature information will primarily be the ground-glass appearance feature information. Extracting the feature information related to the semantic description of the ROI in the image to be tested is mainly to facilitate a better description of the ROI in the image later.

[0071] S204, use a preset image processing network to determine the image processing result corresponding to the image to be tested.

[0072] In this step, the image processing network can be a neural network, and the specific network type and architecture are not specifically limited here. This image processing network is mainly used to process the image to be tested, and may include image detection networks, image segmentation networks, image classification networks, etc. The image detection network is mainly used to detect target regions in the image and obtain detection results, such as detecting lung nodules in an image and obtaining the detected lung nodule regions; the image segmentation network is mainly used to segment target regions in the image and obtain segmentation results, such as segmenting lung nodule regions in an image and obtaining the segmented lung nodule regions; the image classification network is mainly used to classify target regions in the image and obtain classification results, such as classifying various regions in the image and determining whether they belong to lung nodule regions, etc.

[0073] Through the above explanation and examples, we can understand that image processing networks are mainly used to process images and obtain corresponding image processing results. Therefore, after obtaining the image to be tested, an image processing network of the specific type of processing required can be used to process the region of interest (ROI) within the image, and then obtain the corresponding image processing result for the ROI. For example, the ROI can be classified to obtain the classification result.

[0074] It should be noted that there is no sequential relationship between S202 and S204. S202 and S204 can be executed in parallel, or S204 can be executed first and then S202, or S202 can be executed first and then S204.

[0075] S206. Based on the above image semantic information, image processing results, and preset language model, generate explanation information corresponding to the image to be tested. The above language model is trained based on the reference explanation information corresponding to each training image. The above explanation information is used to characterize the relevant information that the image processing network uses to determine the image processing results.

[0076] The language model can be a natural language model, which may include, for example, a text generation model. This language model can be a neural network model; the specific network type and architecture are not limited here, but could be, for example, the GPT3 model. This language model is primarily used for text generation, such as generating natural language text or understanding the meaning of language text.

[0077] Before using a language model, it can generally be pre-trained. This training can be done using various types of text data. Because training a language model is quite difficult, its initial parameters are usually frozen after training with a large amount of text data. In this embodiment, before actually applying the language model, it can also be fine-tuned by incorporating reference explanation information corresponding to each training image to obtain the final trained language model.

[0078] The reference explanation information here can be the explanation information corresponding to the actual training image. It can be obtained by experienced doctors after analyzing and annotating the training image. Taking the region of interest as the lesion region and the image processing network as a classification network as an example, the reference explanation information here can be the text description information corresponding to the classification category of the region of interest output by the image and the lesion classification model (or lesion classification network).

[0079] After obtaining the trained language model, the image semantic information and image processing results of the test image obtained above can be preprocessed and then input into the trained language model for text generation to obtain the explanatory information corresponding to the test image. This explanatory information can be text-based, primarily consisting of relevant information used by the image processing network to obtain the image processing results. Specifically, it can be the image processing network obtaining the image processing results for the region of interest (ROI) based on this relevant information. This relevant information can be related to the ROI, such as its shape, density, location, major and minor axes, etc.

[0080] In addition, by inputting feature information related to the semantic description of the region of interest and image processing results into the language model, the language model can be guided to better learn the relevant image semantic information and processing results during training, thereby improving the accuracy of the generated explanation information when used.

[0081] The explanatory information, consisting of text, generated by the language model described above, combined with the image under test and the image processing results, provides doctors with more and richer information for reference. This allows doctors to perform subsequent image analysis more accurately. For example, if the image processing network or language model has a bias in its understanding of one or more attributes of the current region of interest, the explanatory information generated here can be used to adjust the judgment in a timely manner, ensuring the accuracy of the subsequent image analysis process.

[0082] In the aforementioned method for generating explanatory information, the semantic information of the image to be tested is obtained, and the image processing result corresponding to the image to be tested is determined using a preset image processing network. Then, explanatory information corresponding to the image to be tested is generated based on the semantic information, the image processing result, and a preset language model. The semantic information is used to characterize the features related to the semantic description of the region of interest in the image to be tested. The language model is trained based on the reference explanatory information corresponding to each training image, and the generated explanatory information is used to characterize the relevant information used by the image processing network to determine the image processing result. This method generates explanatory information for the image to be tested from the semantic information, the image processing result, and the language model. This explanatory information is the relevant information used by the image processing network to obtain the image processing result. Therefore, when doctors with less experience perform image analysis by referring to the image processing result, they can combine this explanatory information with the image to be tested for comprehensive analysis, which is more intuitive and can assist doctors in the image analysis process, ensuring the accuracy of the image analysis process. Furthermore, since the generated explanatory information is produced by a language model trained with reference explanatory information, combined with image processing results and information that can characterize the semantic descriptive features of the region of interest, the generated explanatory information is closely related to the region of interest on the image under test and the image processing results. It is more in line with the actual situation and more accurate. This also ensures that the information referenced by doctors when performing image analysis is very accurate, further ensuring the accuracy of the image analysis process.

[0083] The above embodiments mention that explanatory information of the image to be tested can be generated by combining the image semantic information and image processing results of the image to be tested with a language model. The following embodiments illustrate a possible implementation of this process.

[0084] In another embodiment, another method for generating explanatory information is provided, based on the above embodiments, such as... Figure 3 As shown, the above S206 may include the following steps:

[0085] S302, Based on the image processing results, determine the guidance question information related to the image processing results.

[0086] In this step, after obtaining the image processing result by processing the image to be tested through the image processing network, relevant guiding questions can be set according to the specific content of the image processing result. For example, if the image processing result is a classification result, such as the A parameter of the region of interest being category B, then the guiding questions can be set as: Why is the classification result category B? Why is the classification result not another category? Why is the A parameter category B? etc.

[0087] S304, Based on the image semantic information, the guiding question information, and the preset language model, generate the explanation information corresponding to the image to be tested; the aforementioned guiding question information is used to guide the language model to generate relevant information for the image processing results.

[0088] In this step, after setting the associated guiding question information through the image processing results, the guiding question information can be obtained. Since the guiding question information is set for the image processing results, it is also strongly related to the image processing results. In this way, in the subsequent process of the language model generating text, it can guide the language model to better generate explanatory information associated with the image processing results.

[0089] Specifically, after obtaining the image semantic information of the image to be tested and the guiding question information associated with the image processing result, the image semantic information and the guiding question information can be processed into the data format required by the language model, and then input into the language model for text generation processing to obtain the explanatory information in the image to be tested that is strongly related to the image processing result.

[0090] In this embodiment, the guidance question information associated with the image processing result is determined, and the explanation information is generated based on the guidance question information, image semantic information, and language model. The guidance question information is mainly used to guide the language model to generate explanation information that is strongly related to the image result. This makes the explanation information finally generated by the language model strongly correlated with the image processing result and the image semantic information, and more accurate, thereby providing doctors with more accurate reference information.

[0091] The above embodiments mention that the image semantic information and guiding question information of the image under test can be converted into the content required by the language model. The following embodiments will describe the process in detail.

[0092] In another embodiment, another method for generating explanatory information is provided, based on the above embodiments, such as... Figure 4 As shown, the above S304 may include the following steps:

[0093] S402, Based on the guidance question information and the language model, determine the guidance question embedding vector corresponding to the guidance question information.

[0094] In this step, the guidance question information is essentially text. Here, the guidance question information can be input into the embedding layer (i.e., the embedding layer) in the language model to project the guidance question information into a high-dimensional space and obtain the embedding vector corresponding to the guidance question information, which is denoted as the guidance question embedding vector. This guidance question embedding vector is also the unique encoding vector corresponding to the guidance question information.

[0095] S404: Map the semantic information of the image to the text information space to determine the image feature embedding vector corresponding to the image to be tested.

[0096] In this step, image semantic information is the feature information in the image to be tested that is related to the semantic description of the region of interest. In essence, it still belongs to image feature information. In order to use the language model to perform text generation processing on the image semantic information, it is necessary to align the image semantic information to the embedding space of the language model input. As an optional embodiment, the image semantic information and the guidance question embedding vector can be input into a preset fusion model for fusion processing to determine the image feature embedding vector.

[0097] The fusion model can be a neural network model; the specific network type and structure are not limited here, but could be a Transformer model. This fusion model also needs to be trained before use. The training process can be based on the reference interpretation information corresponding to each training image. Specifically, it can be trained using the loss between the reference interpretation information and the prediction information output by the initial fusion model, resulting in a well-trained fusion model.

[0098] After obtaining the trained fusion model, the image semantic information of the image to be tested and the guidance question embedding vector transformed from the guidance question information can be input into the trained fusion model. In the fusion training model, the image semantic information is first mapped from the image feature space to the text information space, that is, the image features are aligned to the text information space. Then, the key information in the image semantic information can be extracted in the trained fusion model according to the guidance question embedding vector to obtain the image feature embedding vector corresponding to the image semantic information, that is, the image feature embedding vector corresponding to the image to be tested.

[0099] S406, input the image feature embedding vector and the guided question embedding vector into the language model for text generation processing to obtain explanatory information.

[0100] In this step, after obtaining the image feature embedding vector and the guiding question embedding vector corresponding to the image to be tested, these two embedding vectors have been aligned to the input embedding space of the language model. Therefore, these two embedding vectors can be directly input into the language model at the same time, and text generation processing is performed in the language model to generate the explanatory information corresponding to the image to be tested.

[0101] In this embodiment, a guiding question embedding vector is determined based on the guiding question information and the language model. Image semantic information is then mapped to the text information space to determine image feature embedding vectors. These image feature embedding vectors and guiding question embedding vectors are then input into the language model for text generation processing to obtain explanatory information. This process transforms both the guiding question information and image semantic information into embedding vectors within the language model's input embedding space, enabling the language model to generate text more quickly and accurately based on these embedding vectors, thus improving the efficiency and accuracy of the final explanatory information generation. Furthermore, when mapping image semantic information to the text information space, a fusion model trained with positive and negative samples and reference explanatory information is used to obtain image feature embedding vectors, further enhancing the efficiency and accuracy of obtaining these vectors.

[0102] The training process of the fusion model was briefly mentioned in the above embodiments. The following embodiments will describe the specific training process of the fusion model in detail.

[0103] In another embodiment, another method for generating explanatory information is provided, based on the above embodiments, such as... Figure 5 As shown, the training method for the above fusion model may include the following steps:

[0104] S502, determine the semantic information of the predicted image and the embedding vector of the prediction guidance question corresponding to each training image.

[0105] In this step, after obtaining each training image, each training image can be input into the image encoder to extract image semantic information, thereby obtaining the image semantic information corresponding to each training image, which is denoted as the predicted image semantic information.

[0106] Simultaneously, each training image can be processed through an image processing network to obtain the image processing result corresponding to each training image. Then, based on the image processing result of each training image, a corresponding guidance question is set to obtain the guidance question information of each training image. Then, the guidance question information of each training image is input into the embedding layer of the language model for vector transformation processing to obtain the guidance question embedding vector corresponding to each training image, which is denoted as the predicted guidance question embedding vector.

[0107] S504, determine the embedding vector of the interpretation information corresponding to each reference interpretation information.

[0108] In this step, while obtaining each training image, reference interpretation information corresponding to each training image can also be obtained. This reference interpretation information can be the explanation information annotated by doctors in practical applications by combining various information and experience. This reference interpretation information generally corresponds to both the training image and its image processing result, and is strongly correlated. After obtaining the reference interpretation information corresponding to each training image, the reference interpretation information can also be mapped to the input embedding space of the language model. Specifically, the embedding layer of the language model or the embedding layer of the initial fusion model can be used to perform vector transformation processing on the reference interpretation information to obtain the interpretation information embedding vector corresponding to each reference interpretation information.

[0109] S506, input the reference interpretation information, the semantic information of each predicted image, and the embedding vector of each prediction guidance question into the initial fusion model for fusion processing, and determine the prediction image feature embedding vector corresponding to each training image.

[0110] In this step, after obtaining the reference interpretation information, image semantic information, and guidance question embedding vector for each training image, these three can be input into the initial fusion model for fusion processing. As an optional embodiment, the fusion processing can involve inputting the reference interpretation information and the prediction guidance question embedding vector into the initial fusion model, performing self-attention fusion processing on the reference interpretation information and the prediction guidance question embedding vector to obtain prediction fusion information; and inputting the semantic information of each predicted image into the initial fusion model, performing interactive attention fusion processing on the semantic information of each predicted image and the prediction fusion information to obtain feature embedding vectors for each predicted image.

[0111] Furthermore, the attention mechanism refers to allowing the model to focus on important regions over a wider range. For the self-attention mechanism within the attention mechanism, it primarily relies on the correlation between input vectors to output each vector, ensuring that each output vector contains the contextual relationships of the input vectors. Therefore, in the initial fusion model, the explanatory information embedding vectors and the corresponding prediction guidance question embedding vectors of each training image can be processed separately using the self-attention mechanism. The self-attention processing of the explanatory information embedding vectors also includes self-attention of image semantic information. Ultimately, the output vectors of the explanatory information embedding vectors and the corresponding prediction guidance question embedding vectors can be obtained. These two output vectors are then fused to obtain the reference explanatory information and the prediction fusion information of the guidance question for each training image. The self-attention mechanism allows for the rapid and accurate acquisition of the respective output vectors of the reference explanatory information and the guidance question information.

[0112] Furthermore, after obtaining the reference explanation information and the prediction fusion information of the guiding question information for each training image, the predicted image semantic information and the corresponding prediction fusion information of each training image can be fused using an interactive attention mechanism. The interactive attention mechanism is mainly used to guide the initial fusion model to pay attention to the image semantic information in the reference explanation information corresponding to the training image through the guiding question information. This includes the mutual attention interaction between image semantic information and text information, so that the fusion model can learn more reference explanation information and guiding question information. The generated image semantic information corresponds to the prediction image feature embedding vector containing more reference explanation information and guiding question information, so that the subsequent language model can generate more accurate explanation information.

[0113] S508, the initial fusion model is trained based on the feature embedding vectors of each predicted image and the corresponding explanatory information embedding vectors to determine the fusion model.

[0114] In this step, after obtaining the predicted image feature embedding vectors corresponding to each training image, the loss can be calculated to train the initial fusion model. Specifically, as an optional embodiment, the matching loss between the corresponding training image and the reference explanation information can be calculated based on the predicted image feature embedding vectors; the feature similarity loss between each predicted image feature embedding vector and the corresponding explanation information embedding vector can be calculated; and the initial fusion model can be trained based on the matching loss and the feature similarity loss.

[0115] Specifically, after obtaining the feature embedding vectors of each predicted image, a classification field can be obtained from the predicted image feature embedding vectors. This classification field represents the prediction information of whether the training image corresponding to the predicted image feature embedding vector is paired with its corresponding reference interpretation information. Simultaneously, when the training image and its corresponding reference interpretation information are pre-set, the pairing status (i.e., the pairing label) can also be obtained. Here, the matching degree between the pairing label and the aforementioned pairing prediction information can be calculated to obtain the matching loss. Alternatively, contrastive learning can be used to calculate the feature similarity between each predicted image feature embedding vector and its corresponding interpretation information embedding vector to obtain the feature similarity loss. Then, the matching loss and feature similarity loss of each training image can be summed or weighted to obtain the total loss. The initial fusion model is then trained using backpropagation of the total loss. When the total loss remains constant, tends to stabilize, or reaches a certain set value, the parameters of the fusion model can be fixed to obtain the trained fusion model.

[0116] Alternatively, during the training of the fusion model, the reference explanation information corresponding to each training image can be used to fine-tune both the fusion model and the language model. For example, the image feature embedding vector and the prediction guidance question embedding vector obtained from the initial fusion model for each training image can be calculated and input into the language model to obtain the prediction explanation information. The loss between the prediction explanation information and the corresponding reference explanation information can then be calculated. This loss, along with the matching loss and feature similarity loss mentioned above, can be used to fine-tune both the fusion model and the language model, ultimately resulting in a well-trained fusion model and a well-trained language model.

[0117] In this embodiment, the reference explanation information, image semantic information, and guiding question embedding vector of each training image are input into the initial fusion model. Simultaneously, the initial fusion model is trained using the explanation information embedding vector corresponding to the reference explanation information to obtain a trained fusion model. This allows for relatively accurate and rapid training of the fusion model using multiple training images, improving its performance. Furthermore, the initial fusion model first performs attention fusion on the guiding question embedding vector and reference explanation information of each training image, and then performs interactive attention fusion on the fusion information and image semantic information of each training image. This allows the initial fusion model to focus more on learning the guiding question information and reference explanation information, making the generated image feature embedding vector closer to the guiding question information and reference explanation information, thus improving the accuracy of the explanation information generated by the subsequent language model. Further, the initial fusion model is trained using various losses such as matching loss and feature similarity loss for each training image, which improves the generalization performance of the trained fusion model, resulting in a better-performing fusion model.

[0118] Furthermore, in the process of text generation of the test image by the language model, in order to make the explanatory information generated by the language model closer to the user's explanatory information of the region of interest of the test image, the above-mentioned language model may also include an additional weight layer. This additional weight layer can be one or more layers, and can generally be set between the layers (e.g., fully connected layers) in the Transformer model. It is mainly used to indicate the relevant information of the image processing results generated by the language model.

[0119] The language model itself includes attention layers, fully connected layers, etc. To make the explanatory information generated by the language model closer to the user's explanation of the region of interest in the image being tested, one or more additional weight layers can be set in the language model, referred to as additional weight layers. These additional weight layers are used to instruct the language model to generate relevant information about the image processing results. In other words, these additional weight layers can guide the language model to generate explanatory information that is closer to the user's expectations during the text generation process, that is, information that is closer to the user's expectations and more relevant to the image processing results. At the same time, when the language model needs to be adjusted in upstream tasks, such as during the training of the fusion model, only the parameters in the additional weight layers of the language model can be adjusted. This allows the language model to converge quickly during fine-tuning in upstream tasks, improving the training speed and performance of the model in upstream tasks.

[0120] Specifically, after obtaining the image feature embedding vector corresponding to the image to be tested, the image feature embedding vector can be input into the language model for processing. The language model processing includes the processing of additional weight layers in the language model. In the language model, multiple fully connected layers, one or more weight layers, and multiple pieces of information are combined to perform text generation processing, and finally, the explanatory information of the image to be tested is generated.

[0121] In this embodiment, the language model also includes an additional weight layer, which is used to instruct the language model to generate relevant information about the image processing results. By combining multiple fully connected layers, one or more additional weight layers, and multiple pieces of information in the language model for text generation processing, the final generated explanation information can better match the user's interpretation of the region of interest, that is, better match the actual situation, and thus truly provide effective data reference for doctors' subsequent image analysis.

[0122] The following detailed embodiment illustrates the technical solution of this application. Based on the above embodiment, and in conjunction with... Figure 6 The detailed flowchart shown illustrates that this method may include the following steps:

[0123] I. The training process of the fusion model.

[0124] D1, obtain each training sample and the corresponding reference explanation information for each training sample;

[0125] D2, each training image is input into the image encoder to extract image semantic information and obtain the predicted image semantic information corresponding to each training image;

[0126] D3 uses the embedding layer of the language model to process the reference explanation information of each training image and determine the explanation information embedding vector corresponding to each reference explanation information.

[0127] D4: Input each training image into a preset image processing network for image processing to obtain the image processing result corresponding to each training image;

[0128] D5, based on the image processing results of each training image, determines the guidance question information corresponding to each training image;

[0129] D6 uses the embedding layer of the language model to process the guidance question information of each training image and determines the predicted guidance question embedding vector corresponding to each guidance question information.

[0130] D7. Input the embedding vectors of each reference interpretation information and each prediction guidance question into the initial fusion model, and perform self-attention fusion processing on each reference interpretation information and each prediction guidance question embedding vector to obtain each prediction fusion information.

[0131] D8 inputs the semantic information of each predicted image into the initial fusion model, performs interactive attention fusion processing on the semantic information of each predicted image and the fusion information of each prediction, and obtains the feature embedding vector of each predicted image.

[0132] D9, determine the embedding vector of the interpretation information corresponding to each reference interpretation information;

[0133] D10, calculates the matching loss between the corresponding training image and the reference interpretation information based on the feature embedding vector of each predicted image;

[0134] D11, calculate the feature similarity loss between each predicted image feature embedding vector and the corresponding explanation information embedding vector;

[0135] D12: The initial fusion model is trained based on matching loss and feature similarity loss to obtain a trained fusion model.

[0136] II. Application Process of Fusion Model and Language Model

[0137] S1, acquire the image to be tested;

[0138] S2, input the image to be tested into the preset image encoder to extract the image semantic information in the image to be tested;

[0139] S3, input the image to be tested into a preset image processing network for image processing, and obtain the image processing result corresponding to the image to be tested;

[0140] S4, Determine guidance question information related to the image processing results based on the image processing results of the image to be tested;

[0141] S5, the embedding layer of the language model is used to process the guidance question information of the image under test, and the guidance question embedding vector corresponding to the guidance question information is determined;

[0142] S6. Input the image semantic information and the guidance question embedding vector into the fusion model for fusion processing to determine the image feature embedding vector corresponding to the image to be tested.

[0143] S7. The image feature embedding vector and the guided question embedding vector are input into the language model for text generation processing to determine the explanatory information corresponding to the image to be tested. The language model is trained based on the reference explanatory information corresponding to each training image. The explanatory information is used to characterize the relevant information that the image processing network uses to determine the image processing result.

[0144] It should be noted that there is no timing relationship between S2 and S3. S2 and S3 can be executed in parallel, or S2 can be executed first and then S3, or S3 can be executed first and then S2.

[0145] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0146] Based on the same inventive concept, this application also provides an explanatory information generation apparatus for implementing the explanatory information generation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more explanatory information generation apparatus embodiments provided below can be found in the limitations of the explanatory information generation method described above, and will not be repeated here.

[0147] In one embodiment, such as Figure 7 As shown, an explanatory information generation device is provided, comprising: an image semantic information determination module, a processing result determination module, and an explanatory information generation module, wherein:

[0148] The semantic information determination module is used to obtain the image semantic information corresponding to the image to be tested; the aforementioned image semantic information is used to characterize the feature information in the image to be tested that is related to the semantic description of the region of interest.

[0149] The processing result determination module is used to determine the image processing result corresponding to the image under test using a preset image processing network.

[0150] The explanation information generation module is used to generate explanation information corresponding to the image under test based on the above image semantic information, image processing results and preset language model; the above language model is trained based on the reference explanation information corresponding to each training image, and the above explanation information is used to characterize the relevant information used by the image processing network to determine the image processing results.

[0151] Optionally, the above image semantic information determination module may include:

[0152] An image acquisition unit is used to acquire the image to be tested.

[0153] The encoding unit is used to input the image to be tested into a preset image encoder to extract the semantic information of the image to be tested.

[0154] In another embodiment, another explanatory information generation apparatus is provided. Based on the above embodiments, the explanatory information generation module may include:

[0155] The guidance information determination unit is used to determine guidance question information related to the image processing results based on the image processing results.

[0156] The explanation information generation unit is used to generate explanation information corresponding to the image under test based on the image semantic information, the guiding question information and the preset language model; the aforementioned guiding question information is used to guide the language model to generate relevant information for the image processing results.

[0157] In another embodiment, another explanatory information generation apparatus is provided. Based on the above embodiments, the explanatory information generation unit may include:

[0158] The guidance vector determination subunit is used to determine the guidance question embedding vector corresponding to the guidance question information based on the guidance question information and the language model;

[0159] The image vector determination subunit is used to map the semantic information of the image to the text information space and determine the image feature embedding vector corresponding to the image under test.

[0160] The explanatory information generation subunit is used to input the image feature embedding vector and the guiding question embedding vector into the language model for text generation processing to obtain explanatory information.

[0161] Optionally, the aforementioned image vector determination subunit is specifically used to input the image semantic information and the guidance question embedding vector into a preset fusion model for fusion processing to determine the image feature embedding vector; the aforementioned fusion model is trained based on the reference interpretation information corresponding to each training image.

[0162] In another embodiment, another explanatory information generation apparatus is provided. Based on the above embodiments, the apparatus may further include a training module, which may include:

[0163] The prediction guidance vector determination unit is used to determine the semantic information of the prediction image and the embedding vector of the prediction guidance question corresponding to each training image;

[0164] An interpretation information vector determination unit is used to determine the interpretation information embedding vector corresponding to each reference interpretation information.

[0165] The prediction image vector determination unit is used to input the embedding vectors of each reference interpretation information, each prediction image semantic information and each prediction guidance question into the initial fusion model for fusion processing, and determine the prediction image feature embedding vectors corresponding to each training image.

[0166] The training unit is used to train the initial fusion model based on the feature embedding vectors of each predicted image and the corresponding explanatory information embedding vectors, and to determine the fusion model.

[0167] Optionally, the above-mentioned predicted image vector determination unit may include:

[0168] The first fusion subunit is used to input the embedding vectors of each reference interpretation information and each prediction guidance question into the initial fusion model, and to perform self-attention fusion processing on each reference interpretation information and each prediction guidance question embedding vector to obtain each prediction fusion information.

[0169] The second fusion subunit is used to input the semantic information of each predicted image into the initial fusion model, and to perform interactive attention fusion processing on the semantic information of each predicted image and the fusion information of each prediction to obtain the feature embedding vector of each predicted image.

[0170] Optionally, the above training unit may include:

[0171] The first loss calculation subunit is used to calculate the matching loss between the corresponding training image and the reference interpretation information based on the feature embedding vector of each predicted image.

[0172] The second loss calculation subunit is used to calculate the feature similarity loss between each predicted image feature embedding vector and the corresponding explanation information embedding vector.

[0173] The training subunit is used to train the initial fusion model based on the matching loss and feature similarity loss.

[0174] In another embodiment, another explanatory information generation apparatus is provided, which, based on the above embodiments, includes an additional weight layer in the language model; the additional weight layer is used to instruct the language model to generate relevant information about the image processing results.

[0175] Each module in the aforementioned explanatory information generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0176] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0177] Obtain the image semantic information corresponding to the image to be tested; the image semantic information is used to characterize the feature information related to the semantic description of the region of interest in the image to be tested; use a preset image processing network to determine the image processing result corresponding to the image to be tested; generate the explanatory information corresponding to the image to be tested based on the image semantic information, the image processing result and the preset language model; the language model is trained based on the reference explanatory information corresponding to each training image, and the explanatory information is used to characterize the relevant information on which the image processing network determines the image processing result.

[0178] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0179] Based on the image processing results, guide question information related to the image processing results is determined; based on the image semantic information, guide question information, and a preset language model, explanatory information corresponding to the image to be tested is generated; the above guide question information is used to guide the language model to generate relevant information for the image processing results.

[0180] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0181] Based on the guidance question information and the language model, the guidance question embedding vector corresponding to the guidance question information is determined; the image semantic information is mapped to the text information space to determine the image feature embedding vector corresponding to the image to be tested; the image feature embedding vector and the guidance question embedding vector are input into the language model for text generation processing to obtain the explanation information.

[0182] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0183] Image semantic information and guidance question embedding vectors are input into a pre-defined fusion model for fusion processing to determine image feature embedding vectors; the fusion model is trained based on the reference interpretation information corresponding to each training image.

[0184] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0185] Determine the semantic information of the predicted image and the embedding vector of the prediction guidance question corresponding to each training image; determine the embedding vector of the interpretation information corresponding to each reference interpretation information; input each reference interpretation information, each predicted image semantic information and each prediction guidance question embedding vector into the initial fusion model for fusion processing to determine the feature embedding vector of the predicted image corresponding to each training image; train the initial fusion model based on each predicted image feature embedding vector and the corresponding interpretation information embedding vector to determine the fusion model.

[0186] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0187] Each reference interpretation information and each prediction guidance question embedding vector are input into the initial fusion model. Self-attention fusion processing is performed on each reference interpretation information and each prediction guidance question embedding vector to obtain each prediction fusion information. Each prediction image semantic information is input into the initial fusion model. Interactive attention fusion processing is performed on each prediction image semantic information and each prediction fusion information to obtain each prediction image feature embedding vector.

[0188] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0189] Calculate the matching loss between the corresponding training image and the reference interpretation information based on the feature embedding vector of each predicted image; calculate the feature similarity loss between the feature embedding vector of each predicted image and the corresponding interpretation information embedding vector; train the initial fusion model based on the matching loss and the feature similarity loss.

[0190] In one embodiment, the language model includes an additional weight layer; the additional weight layer is used to instruct the language model on relevant information for generating image processing results.

[0191] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0192] Acquire the image to be tested; input the image to be tested into a preset image encoder to extract the semantic information of the image to be tested.

[0193] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0194] Obtain the image semantic information corresponding to the image to be tested; the image semantic information is used to characterize the feature information related to the semantic description of the region of interest in the image to be tested; use a preset image processing network to determine the image processing result corresponding to the image to be tested; generate the explanatory information corresponding to the image to be tested based on the image semantic information, the image processing result and the preset language model; the language model is trained based on the reference explanatory information corresponding to each training image, and the explanatory information is used to characterize the relevant information on which the image processing network determines the image processing result.

[0195] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0196] Based on the image processing results, guide question information related to the image processing results is determined; based on the image semantic information, guide question information, and a preset language model, explanatory information corresponding to the image to be tested is generated; the above guide question information is used to guide the language model to generate relevant information for the image processing results.

[0197] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0198] Based on the guidance question information and the language model, the guidance question embedding vector corresponding to the guidance question information is determined; the image semantic information is mapped to the text information space to determine the image feature embedding vector corresponding to the image to be tested; the image feature embedding vector and the guidance question embedding vector are input into the language model for text generation processing to obtain the explanation information.

[0199] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0200] Image semantic information and guidance question embedding vectors are input into a pre-defined fusion model for fusion processing to determine image feature embedding vectors; the fusion model is trained based on the reference interpretation information corresponding to each training image.

[0201] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0202] Determine the semantic information of the predicted image and the embedding vector of the prediction guidance question corresponding to each training image; determine the embedding vector of the interpretation information corresponding to each reference interpretation information; input each reference interpretation information, each predicted image semantic information and each prediction guidance question embedding vector into the initial fusion model for fusion processing to determine the feature embedding vector of the predicted image corresponding to each training image; train the initial fusion model based on each predicted image feature embedding vector and the corresponding interpretation information embedding vector to determine the fusion model.

[0203] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0204] Each reference interpretation information and each prediction guidance question embedding vector are input into the initial fusion model. Self-attention fusion processing is performed on each reference interpretation information and each prediction guidance question embedding vector to obtain each prediction fusion information. Each prediction image semantic information is input into the initial fusion model. Interactive attention fusion processing is performed on each prediction image semantic information and each prediction fusion information to obtain each prediction image feature embedding vector.

[0205] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0206] Calculate the matching loss between the corresponding training image and the reference interpretation information based on the feature embedding vector of each predicted image; calculate the feature similarity loss between the feature embedding vector of each predicted image and the corresponding interpretation information embedding vector; train the initial fusion model based on the matching loss and the feature similarity loss.

[0207] In one embodiment, the language model includes an additional weight layer; the additional weight layer is used to instruct the language model on relevant information for generating image processing results.

[0208] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0209] Acquire the image to be tested; input the image to be tested into a preset image encoder to extract the semantic information of the image to be tested.

[0210] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps:

[0211] Obtain the image semantic information corresponding to the image to be tested; the image semantic information is used to characterize the feature information related to the semantic description of the region of interest in the image to be tested; use a preset image processing network to determine the image processing result corresponding to the image to be tested; generate the explanatory information corresponding to the image to be tested based on the image semantic information, the image processing result and the preset language model; the language model is trained based on the reference explanatory information corresponding to each training image, and the explanatory information is used to characterize the relevant information on which the image processing network determines the image processing result.

[0212] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0213] Based on the image processing results, guide question information related to the image processing results is determined; based on the image semantic information, guide question information, and a preset language model, explanatory information corresponding to the image to be tested is generated; the above guide question information is used to guide the language model to generate relevant information for the image processing results.

[0214] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0215] Based on the guidance question information and the language model, the guidance question embedding vector corresponding to the guidance question information is determined; the image semantic information is mapped to the text information space to determine the image feature embedding vector corresponding to the image to be tested; the image feature embedding vector and the guidance question embedding vector are input into the language model for text generation processing to obtain the explanation information.

[0216] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0217] Image semantic information and guidance question embedding vectors are input into a pre-defined fusion model for fusion processing to determine image feature embedding vectors; the fusion model is trained based on the reference interpretation information corresponding to each training image.

[0218] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0219] Determine the semantic information of the predicted image and the embedding vector of the prediction guidance question corresponding to each training image; determine the embedding vector of the interpretation information corresponding to each reference interpretation information; input each reference interpretation information, each predicted image semantic information and each prediction guidance question embedding vector into the initial fusion model for fusion processing to determine the feature embedding vector of the predicted image corresponding to each training image; train the initial fusion model based on each predicted image feature embedding vector and the corresponding interpretation information embedding vector to determine the fusion model.

[0220] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0221] Each reference interpretation information and each prediction guidance question embedding vector are input into the initial fusion model. Self-attention fusion processing is performed on each reference interpretation information and each prediction guidance question embedding vector to obtain each prediction fusion information. Each prediction image semantic information is input into the initial fusion model. Interactive attention fusion processing is performed on each prediction image semantic information and each prediction fusion information to obtain each prediction image feature embedding vector.

[0222] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0223] Calculate the matching loss between the corresponding training image and the reference interpretation information based on the feature embedding vector of each predicted image; calculate the feature similarity loss between the feature embedding vector of each predicted image and the corresponding interpretation information embedding vector; train the initial fusion model based on the matching loss and the feature similarity loss.

[0224] In one embodiment, the language model includes an additional weight layer; the additional weight layer is used to instruct the language model on relevant information for generating image processing results.

[0225] In one embodiment, when the computer program is executed by a processor, it also performs the following steps:

[0226] Acquire the image to be tested; input the image to be tested into a preset image encoder to extract the semantic information of the image to be tested.

[0227] It should be noted that all data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are fully authorized by all parties.

[0228] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0229] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0230] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for generating explanatory information, characterized in that, The method includes: Obtain the image semantic information corresponding to the image to be tested; the image semantic information is used to characterize the feature information related to the semantic description of the region of interest in the image to be tested, and the region of interest is used to characterize the lesion area in the test site corresponding to the image to be tested; The image processing result corresponding to the image under test is determined using a preset image processing network. Based on the image processing result, guiding question information related to the image processing result is determined. Based on the image semantic information, the guiding question information, and a preset language model, explanatory information corresponding to the image to be tested is generated. The language model is trained based on the reference explanatory information corresponding to each training image. The explanatory information is used to characterize the relevant information that the image processing network uses to determine the image processing result. The guiding question information is used to guide the language model to generate relevant information for the image processing result.

2. The method according to claim 1, characterized in that, The step of generating explanatory information corresponding to the image under test based on the image semantic information, the guiding question information, and the preset language model includes: Based on the guidance question information and the language model, determine the guidance question embedding vector corresponding to the guidance question information; The image semantic information is mapped to the text information space to determine the image feature embedding vector corresponding to the image to be tested; The image feature embedding vector and the guiding question embedding vector are input into the language model for text generation processing to obtain the explanation information.

3. The method according to claim 2, characterized in that, The step of mapping the image semantic information to the text information space and determining the image feature embedding vector corresponding to the image to be tested includes: The image semantic information and the guidance question embedding vector are input into a preset fusion model for fusion processing to determine the image feature embedding vector; The fusion model is trained based on the reference interpretation information corresponding to each training image.

4. The method according to claim 3, characterized in that, The training methods for the fusion model include: Determine the semantic information of the predicted image and the embedding vector of the prediction guidance question corresponding to each of the training images; Determine the embedding vector of the interpretation information corresponding to each of the aforementioned reference interpretation information; The reference interpretation information, the semantic information of the predicted image, and the embedding vector of the prediction guidance question are input into the initial fusion model for fusion processing to determine the feature embedding vector of the predicted image corresponding to each training image. The initial fusion model is trained based on the predicted image feature embedding vectors and the corresponding explanatory information embedding vectors to determine the fusion model.

5. The method according to claim 4, characterized in that, The step of inputting the reference interpretation information, the semantic information of the predicted image, and the embedding vector of the prediction guidance question into the initial fusion model for fusion processing to determine the prediction image feature embedding vector corresponding to each training image includes: Each of the aforementioned reference interpretation information and each of the aforementioned prediction guidance question embedding vectors are input into the initial fusion model, and self-attention fusion processing is performed on each of the aforementioned reference interpretation information and each of the aforementioned prediction guidance question embedding vectors to obtain each prediction fusion information; The semantic information of each predicted image is input into the initial fusion model, and interactive attention fusion processing is performed on the semantic information of each predicted image and the prediction fusion information to obtain the feature embedding vector of each predicted image.

6. The method according to claim 1, characterized in that, The step of generating explanatory information corresponding to the image under test based on the image semantic information, the guiding question information, and the preset language model includes: The image semantic information and the guidance question information are processed into the data format required by the language model; The processed image semantic information and the processed guidance question information are input into the language model for text generation processing to obtain explanatory information related to the image processing result in the image to be tested.

7. An explanatory information generation device, characterized in that, The device includes: The semantic information determination module is used to acquire the image semantic information corresponding to the image to be tested; the image semantic information is used to characterize the feature information related to the semantic description of the region of interest in the image to be tested, and the region of interest is used to characterize the lesion area in the test site corresponding to the image to be tested; The processing result determination module is used to determine the image processing result corresponding to the image to be tested using a preset image processing network. An explanation information generation module is used to determine guidance question information related to the image processing result based on the image processing result, and to generate explanation information corresponding to the image to be tested based on the image semantic information, the guidance question information, and a preset language model; the language model is trained based on the reference explanation information corresponding to each training image, and the explanation information is used to characterize the relevant information used by the image processing network to determine the image processing result; the guidance question information is used to guide the language model to generate relevant information for the image processing result.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images