Medical image report generation method based on analogy reasoning of disease evolution perception

Abstract

The application discloses a disease evolution perception-based medical image report generation method and device based on analogy reasoning, and relates to the technical field of medical image analysis. The method comprises the following steps: based on an adaptive alignment mechanism, aligning the features of two images in a key lesion area; based on a visual evolution consistency loss function, constraining the learning of the evolution relationship between images; according to a current medical image report and a historical medical image report, extracting features by using a text encoder; based on a text evolution consistency loss function, constraining the learning of the evolution relationship between reports; and generating a medical image report based on an optimized visual encoder, an optimized adaptive region alignment module, an optimized evolution relationship learner, a text encoder, an optimized relationship mapper and an optimized large language model. The application is an accurate and efficient medical image report generation method based on cross-time disease evolution modeling.

Description

Technical Field

[0001] This invention relates to the field of medical image analysis technology, and in particular to a method and apparatus for generating medical image reports based on analogical reasoning and disease evolution perception. Background Technology

[0002] Medical imaging reports are an indispensable part of the clinical diagnostic process, typically interpreted and written by experienced radiologists based on medical images. With the increasing volume of medical images, the workload of physicians is gradually increasing; therefore, AI-based automated medical imaging report generation technology has become a hot research topic.

[0003] Most existing medical image report generation methods are based on single-phase images, meaning they rely solely on the image features of the current examination to generate descriptive text. While these methods can capture static information from images, they struggle to capture the evolutionary trends of diseases across different time points, thus limiting their ability to describe disease progression. Furthermore, because lesion location, extent, and imaging conditions can change over time, fusion methods based solely on image-level or local features struggle to establish stable spatial correspondences across different time phases, thereby affecting the effective identification of lesion changes.

[0004] By linking current images with historical images or reports, the temporal consistency and clinical validity of reports can be improved. However, existing methods often employ simple fusion methods such as direct feature stitching or attention mechanisms, failing to explicitly model the evolution of the disease over time. When there are significant differences in information across time phases, obvious spatial misalignments, or complex lesion changes, these methods often struggle to accurately resolve disease progression patterns, and the temporal consistency and clinical reliability of the generated results still have room for improvement.

[0005] In the existing technology, there is a lack of an accurate and efficient method for generating medical image reports based on cross-temporal disease evolution modeling. Summary of the Invention

[0006] To address the technical problems of existing technologies, such as the inability to effectively model cross-temporal disease evolution relationships; spatial misalignment in cross-temporal images, making accurate regional correspondence difficult; lack of unified modeling of cross-modal evolution relationships; and the need to improve the clinical consistency of generated text, this invention provides a method and apparatus for generating medical image reports based on analogical reasoning for disease evolution perception. The technical solution is as follows:

[0007] On the one hand, a method for generating medical image reports based on analogical reasoning for disease evolution perception is provided. This method is implemented by a medical image report generation device and includes: Obtain the patient's current medical images, current medical image reports, historical medical images, and historical medical image reports; Based on current and historical medical images, a visual encoder is used to extract features, obtaining current local visual feature vectors, current global visual feature vectors, historical local visual feature vectors, and historical global visual feature vectors. Based on the current local visual feature vectors and historical local visual features, an adaptive region alignment module is used to align key lesion regions, obtaining aligned current image feature vectors and aligned historical image feature vectors. Based on the visual evolution consistency loss function, an evolution relationship learner is used to model the evolution relationship according to the current global visual feature vector, the historical global visual feature vector, the aligned current image feature vector, and the aligned historical image feature vector, to obtain the visual evolution relationship vector and the visual evolution consistency loss. Based on current and historical medical image reports, a text encoder is used to extract features to obtain current text feature vectors and historical text feature vectors. Based on the text evolution consistency loss function, an evolution relationship mapping is performed using a relationship mapper according to the current text feature vector, historical text feature vector and visual evolution relationship vector to obtain the text evolution relationship vector and text evolution consistency loss. Based on the autoregressive language modeling loss function, the language modeling loss is obtained by using a large language model to infer from the current medical images, historical medical image reports, text evolution relationship vectors and the current medical image reports. Based on the joint optimization loss function, the parameters of the visual encoder, adaptive region alignment module, evolutionary relation learner, relation mapper, and large language model are optimized according to the visual evolution consistency loss, text evolution consistency loss, and language modeling loss, resulting in optimized visual encoder, optimized adaptive region alignment module, optimized evolutionary relation learner, optimized relation mapper, and optimized large language model. The system acquires the target patient's current medical images, historical medical images, and historical medical image reports. Based on an optimized visual encoder, an optimized adaptive region alignment module, an optimized evolutionary relation learner, a text encoder, an optimized relation mapper, and an optimized large language model, the system generates a report using the target patient's current medical images, historical medical images, and historical medical image reports.

[0008] On the other hand, a medical image report generation device based on analogical reasoning for disease evolution perception is provided. This device is applied to a medical image report generation method based on analogical reasoning for disease evolution perception. The device includes: The image and report acquisition module is used to acquire the patient's current medical images, current medical image reports, historical medical images, and historical medical image reports. The image feature alignment module is used to extract features from the current medical image and historical medical images using a visual encoder to obtain the current local visual feature vector, the current global visual feature vector, the historical local visual feature vector, and the historical global visual feature vector; and to align key lesion regions using an adaptive region alignment module based on the current local visual feature vector and historical local visual features to obtain the aligned current image feature vector and the aligned historical image feature vector. The visual evolution calibration module is used to model evolution relationships based on the visual evolution consistency loss function, using an evolution relationship learner to obtain the visual evolution relationship vector and the visual evolution consistency loss, according to the current global visual feature vector, the historical global visual feature vector, the aligned current image feature vector, and the aligned historical image feature vector. The text feature extraction module is used to extract features from the current medical image report and historical medical image reports using a text encoder to obtain the current text feature vector and the historical text feature vector. The cross-modal alignment module is used to perform evolution relationship mapping based on the text evolution consistency loss function, according to the current text feature vector, historical text feature vector and visual evolution relationship vector, using a relationship mapper to obtain the text evolution relationship vector and text evolution consistency loss; The language modeling loss calculation module is used to obtain the language modeling loss based on the autoregressive language modeling loss function, using a large language model to infer from the current medical image, historical medical image reports, text evolution relationship vectors and the current medical image report. The parameter optimization module is used to optimize the parameters of the visual encoder, adaptive region alignment module, evolutionary relation learner, relation mapper, and large language model based on the joint optimization loss function, according to the visual evolution consistency loss, text evolution consistency loss, and language modeling loss, to obtain optimized visual encoder, optimized adaptive region alignment module, optimized evolutionary relation learner, optimized relation mapper, and optimized large language model. The image report generation module is used to acquire the target patient's current medical images, target patient's historical medical images, and target patient's historical medical image reports. Based on the optimized visual encoder, optimized adaptive region alignment module, optimized evolutionary relation learner, text encoder, optimized relation mapper, and optimized large language model, the module generates reports according to the target patient's current medical images, target patient's historical medical images, and target patient's historical medical image reports.

[0009] On the other hand, a medical image report generation device is provided, the medical image report generation device comprising: a processor; a memory storing computer-readable instructions, wherein when the computer-readable instructions are executed by the processor, any one of the above-described methods for generating medical image reports based on analogical reasoning and disease evolution perception is implemented.

[0010] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement any of the above-described methods for generating medical image reports based on analogical reasoning and disease evolution perception.

[0011] The beneficial effects of the technical solutions provided in the embodiments of the present invention include at least the following: This invention proposes a medical impact report generation method based on analogical reasoning for disease evolution perception. By modeling visual evolution relationships and text evolution relationships, the model can capture changes in images over time, improving the ability to identify disease progression or improvement. The adaptive region alignment module can automatically align lesion areas, effectively reducing cross-temporal feature shifts caused by changes in body position and shooting angle, and improving the robustness of cross-temporal analysis. By introducing a cross-modal evolution consistency mechanism, a correspondence is established between structured visual changes and textual expression changes, thereby improving the temporal logic of the final generated report at the semantic level; large-scale language models are more likely to generate report content that conforms to the actual disease progression under the guidance of evolutionary relationships, thus enhancing diagnostic auxiliary value; Each module of this invention can be replaced with different neural network structures or inference models, enabling it to adapt to various medical image types and clinical scenarios. This invention is an accurate and efficient method for generating medical image reports based on cross-temporal disease evolution modeling. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a medical image report generation method based on analogical reasoning for disease evolution perception, provided by an embodiment of the present invention. Figure 2 This is a schematic diagram of an analogical reasoning model architecture provided in an embodiment of the present invention; Figure 3This is a block diagram of a medical image report generation device based on analogical reasoning for disease evolution perception, provided in an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of a medical image report generation device provided in an embodiment of the present invention. Detailed Implementation

[0014] The technical solution of the present invention will now be described with reference to the accompanying drawings.

[0015] In embodiments of the present invention, words such as "exemplarily," "for example," etc., are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in the present invention should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present the concept in a concrete manner. Furthermore, in embodiments of the present invention, the meaning expressed by "and / or" can be both, or either one.

[0016] In the embodiments of this invention, the terms "image" and "picture" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning. Similarly, the terms "of," "corresponding (relevant)," and "corresponding" may sometimes be used interchangeably. It should be noted that, without emphasizing the distinction between them, they convey the same meaning.

[0017] In this embodiment of the invention, sometimes a subscript such as W1 may be written in a non-subscript form such as W1. When the difference is not emphasized, the meaning they express is the same.

[0018] To make the technical problems, technical solutions and advantages of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0019] This invention provides a method for generating medical image reports based on analogical reasoning and disease evolution perception. This method can be implemented by a medical image report generation device, which can be a terminal or a server. Figure 1 The flowchart shown is for a medical image report generation method based on analogical reasoning for disease evolution perception. The processing flow of this method may include the following steps:

[0020] S1. Obtain the patient's current medical images, current medical image reports, historical medical images, and historical medical image reports; In one feasible implementation, the present invention proposes an analogical reasoning model architecture, the schematic diagram of which is shown below. Figure 2As shown. The imaging modalities used in this invention include medical imaging data of the target patient in the current time phase and the most recent historical time phase. The imaging data may be in the form of chest X-ray, computed tomography (CT), or magnetic resonance imaging (MRI), etc.

[0021] The reporting modality includes historical medical image reports corresponding to the historical images, as well as current medical images and their corresponding current medical image reports. Historical medical image reports provide disease-language descriptions of the historical images, serving as the basis for inference.

[0022] Because the implicit disease evolution relationship between the imaging modality and the reporting modality has structural consistency, this invention formalizes the medical image report generation task as a multimodal analogical reasoning problem. Historical medical images Medical images in the current time period The data is considered analogous to source domain data, while the corresponding medical reports... and This is considered analogous to target domain data. The goal of this invention is to enhance the accuracy and effectiveness of the reasoning process by modeling consistent relationship patterns across time phases to infer missing reports corresponding to the current image.

[0023] S2. Based on the current medical images and historical medical images, use a visual encoder to extract features and obtain the current local visual feature vector, the current global visual feature vector, the historical local visual feature vector, and the historical global visual feature vector; based on the current local visual feature vector and the historical local visual features, use an adaptive region alignment module to align key lesion regions and obtain the aligned current image feature vector and the aligned historical image feature vector. In one feasible implementation, for historical images and current images Each image is processed using a visual encoder. Encoding is performed to obtain their respective global visual feature vectors and local visual feature sequences. The visual encoder may include a convolutional neural network, a visual Transformer, or other deep neural networks capable of extracting spatial structure information. The encoding process is as follows (1):

[0024] (1); in, and These represent the global and local features of the image, respectively. The number of visual feature units, is the dimension of the feature unit.

[0025] To effectively align disease-related regions in historical and current images, this invention introduces an Adaptive Region Alignment (ARA) mechanism. This mechanism adaptively resamples semantically aligned feature units from local features in historical and current images, ensuring that relevant regions at different time points can be effectively aligned.

[0026] Predefined learnable latent input queries , This represents the number of output feature units. Cross-attention is used to... and feedforward network The iterative update query is performed as follows (2):

[0027] (2); in, Indicates the index of the current iteration layer. This represents the number of iterations. After... After iterations, the final resampled feature representation is obtained as follows: .

[0028] This adaptive region alignment mechanism ensures effective alignment of relevant regions in historical and current images, providing a consistent feature representation for modeling disease evolution relationships.

[0029] S3. Based on the visual evolution consistency loss function, the evolution relationship is modeled using an evolution relationship learner according to the current global visual feature vector, the historical global visual feature vector, the aligned current image feature vector, and the aligned historical image feature vector, to obtain the visual evolution relationship vector and the visual evolution consistency loss. Among them, evolutionary relation learners include multilayer perceptrons, visual Transformers, or graph convolutional neural networks.

[0030] In one feasible implementation, the current temporal region features, after region alignment, are concatenated with historical temporal region features and input into the visual evolution relationship learning module to obtain a visual evolution relationship vector describing the trend of disease changes over time. This visual evolution relationship vector can characterize the temporal changes of lesions in density, extent, location, and structure.

[0031] The evolutionary relationship learner used in this invention includes multilayer perceptrons, visual Transformers, graph convolutional neural networks, or other deep neural networks capable of extracting spatial structure information. This is achieved by obtaining aligned region features. and Then, the two are concatenated and input into the evolutionary relation learner. This is used to model the disease evolution relationships between images. The process of extracting the disease evolution relationships between images is as follows (3):

[0032] (3); in, The learned visual evolution relationship vectors represent implicit evolutionary patterns between historical and current images. These relationships can capture implicit but clinically significant disease progression patterns, avoiding relationships that are difficult to unravel using traditional multimodal fusion methods.

[0033] Optionally, based on the visual evolution consistency loss function, an evolution relationship learner is used to model the evolution relationship according to the current global visual feature vector, the historical global visual feature vector, the aligned current image feature vector, and the aligned historical image feature vector, to obtain the visual evolution relationship vector and the visual evolution consistency loss, including: Based on the aligned current image feature vector and the aligned historical image feature vector, an evolutionary relationship learner is used to extract cross-temporal phase relationships to obtain a visual evolutionary relationship vector. Based on the visual evolution consistency loss function, the visual evolution consistency loss is calculated according to the current global visual feature vector, the historical global visual feature vector, and the visual evolution relationship vector.

[0034] In one feasible implementation, to improve the reliability of visual evolution relationships, we further propose a visual evolution consistency loss constraint during the construction of cross-temporal relationship representation learning, so that the generated visual evolution relationships can effectively connect the visual features of historical images and current images. The pooled image relationship representation is projected onto the image feature space, as shown in equation (4):

[0035] (4); Define the visual evolution consistency loss function This loss ensures the implicit relationships modeled. Able to correctly connect from historical images To the current image The global feature evolution. The mathematical expression of the loss function is as follows (5):

[0036] (5); This consistency loss ensures the consistency of disease evolution relationships in image modalities between historical and current images, providing a solid foundation for subsequent cross-modal inference.

[0037] S4. Based on the current medical image report and historical medical image reports, use a text encoder to extract features and obtain the current text feature vector and historical text feature vector. In one possible implementation, the present invention uses a text encoder. The report is encoded to extract semantic features from the text modality. The text encoder may include a Transformer, a Bidirectional Encoder Representations from Transformers (BERT), a Generative Pre-trained Transformer (GPT), a convolutional neural network, a recurrent neural network, or other deep learning models capable of mapping text to semantic vector representations. The mathematical expression for this process is as follows (6):

[0038] (6); S5. Based on the text evolution consistency loss function, the evolution relationship is mapped using a relation mapper according to the current text feature vector, historical text feature vector and visual evolution relationship vector, to obtain the text evolution relationship vector and text evolution consistency loss. The relation mapper is a fully connected layer that incorporates mapping functions.

[0039] In one feasible implementation, the visual evolution relation vector is input into a relation mapper to obtain the corresponding text evolution relation vector. This vector describes how disease changes in images are represented in the linguistic feature space, including lesion absorption, changes in shadow enhancement range, structural alterations, and changes in device or pipeline location. This invention projects relations extracted from the source domain (i.e., the visual domain) onto the target domain (i.e., the text domain) to achieve cross-modal disease evolution relation representation learning. Mapping function Representing disease evolution relationships learned from the visual domain by incorporating fully connected layers. Projected into the report relation space The process is as follows (7):

[0040] (7); in, This is the mapped disease evolution relationship vector, representing the implicit evolution pattern between historical reports and current reports.

[0041] Optionally, based on the text evolution consistency loss function, a text evolution relation vector is obtained using a relation mapper based on the current text feature vector, historical text feature vector, and visual evolution relation vector. The text evolution consistency loss is then calculated, including: Based on the visual evolution relation vector, a relation mapper is used to perform cross-modal text vector mapping to obtain the text evolution relation vector. Based on the text evolution consistency loss function, the text evolution consistency loss is calculated according to the current text feature vector, historical text feature vector and text evolution relationship vector.

[0042] In one feasible implementation, to improve the reliability of text evolution relationships, a text evolution consistency constraint is introduced during the mapping process. By regularizing the text evolution relationship vector, the semantic feature vectors of historical reports are mapped to a semantic direction close to the target report, ensuring the logical consistency between text evolution relationships and visual evolution relationships, and improving the coherence and clinical usability of cross-modal reasoning. The pooled inter-report relationship representation is projected onto the report feature space, and the mathematical expression of this process is as follows (8):

[0043] (8); Define text evolution consistency loss This loss ensures the implicit relationships modeled. Able to correctly connect from historical reports Up to the current report The global feature evolution. The loss function is as follows (9):

[0044] (9); S6. Based on the autoregressive language modeling loss function, the language modeling loss is obtained by using a large language model to infer from the current medical images, historical medical image reports, text evolution relationship vectors and the current medical image reports. In one feasible implementation, the present invention is based on a text evolution relation vector of analogy transfer. and historical reports Predicting current radiological reports Unlike traditional multimodal fusion methods, this invention explicitly guides the extraction of relational patterns from visual progression, ensuring that the generated reports remain temporally consistent and clinically uniform.

[0045] This invention employs a Large Language Model (LLM) as the analogy reasoning engine, leveraging its powerful knowledge integration and contextual reasoning capabilities. In this process, the input to the LLM includes predefined prompts. Text evolution relation vector Historical Report And other optional auxiliary information (e.g., current medical images) The process is as follows: (10)

[0046] (10); In this invention, a preset report generation prompt is provided. This prompt template guides an LLM-based analogy reasoning module to generate a comprehensive and detailed diagnostic report for the current image. Depending on the input information, the prompt template may generally include the following:

[0047] “ Generate a comprehensive and detailed diagnostic report of the current chest X-ray image. This is a historical diagnostic report. tell <report>< / report> This is the relationship from historical diagnosis to current diagnosis. <relation>< / relation> .

[0048] The autoregressive language modeling loss is optimized to ensure the accuracy of the generated report. The loss function is as follows (11):

[0049] (11); in, Indicates the first generation of the report Each word element, Indicates the preceding Each word element.

[0050] S7. Based on the joint optimization loss function, the parameters of the visual encoder, adaptive region alignment module, evolutionary relation learner, relation mapper and large language model are optimized according to the visual evolution consistency loss, text evolution consistency loss and language modeling loss, to obtain optimized visual encoder, optimized adaptive region alignment module, optimized evolutionary relation learner, optimized relation mapper and optimized large language model. Optionally, based on the joint optimization loss function, and according to the visual evolution consistency loss, text evolution consistency loss, and language modeling loss, the parameters of the visual encoder, adaptive region alignment module, evolutionary relation learner, relation mapper, and large language model are optimized to obtain optimized visual encoder, optimized adaptive region alignment module, optimized evolutionary relation learner, optimized relation mapper, and optimized large language model, including: Based on the joint optimization loss function, the joint optimization loss is obtained by weighting the visual evolution consistency loss, text evolution consistency loss and language modeling loss. Based on the joint optimization loss, the parameters of the visual encoder, adaptive region alignment module, evolutionary relation learner, relation mapper, and large language model are optimized to obtain optimized visual encoder, optimized adaptive region alignment module, optimized evolutionary relation learner, optimized relation mapper, and optimized large language model.

[0051] In one feasible implementation, in order to achieve synergistic optimization of medical image report generation and disease evolution modeling, a joint optimization loss function is constructed to train the model, and its expression is as follows (12): (12); in, This represents the generation loss term used to measure the difference between the generated medical image report and the target report. This represents the first regularization term used to constrain the consistency of disease evolution relationships in the image domain. This represents the second regularization term used to constrain the consistency of disease evolution relationships within the text domain; and These are preset weight parameters used to adjust the strength of each consistency constraint in the overall optimization process.

[0052] During training, a teacher-mandated strategy is adopted, and through the aforementioned joint optimization method, the relationship between disease evolution in the image domain and the text domain is constrained while generating report content. This ensures that the model balances the accuracy of report generation with the rationality of disease evolution modeling during training, thereby improving the quality of the generated reports.

[0053] S8. Obtain the target patient's current medical images, target patient's historical medical images, and target patient's historical medical image reports; generate reports based on the target patient's current medical images, target patient's historical medical images, and target patient's historical medical image reports, using an optimized visual encoder, optimized adaptive region alignment module, optimized evolutionary relation learner, text encoder, optimized relation mapper, and optimized large language model.

[0054] In one feasible implementation, this invention uses a large language model as an analogy inference engine. It takes the visual features of the current image (optional), historical report content, and a text evolution vector mapped from visual evolution relationships as joint inputs, and guides the model to focus on cross-temporal disease changes through prompts. Based on the aforementioned multimodal inputs, the large language model performs analogy inference, associating the semantic structure of historical reports with the evolutionary relationships between temporal phases to generate a medical image report reflecting the current lesion state and its temporal evolution trend.

[0055] Large language models can include, but are not limited to, pre-trained language models based on the Transformer architecture, and large-scale text generation models that can be fine-tuned or frozen. The generated report includes a structured description of the lesions shown in the current image, as well as temporal auxiliary diagnostic content related to disease evolution.

[0056] This invention proposes a medical impact report generation method based on analogical reasoning for disease evolution perception. By modeling disease evolution relationships, the model can capture changes in images over time, improving the ability to identify disease progression or improvement. The adaptive region alignment module can automatically align lesion areas, effectively reducing cross-temporal feature shifts caused by changes in body position and shooting angle, and improving the robustness of cross-temporal analysis. By introducing a cross-modal evolution consistency mechanism, a correspondence is established between structured visual changes and textual expression changes, thereby improving the temporal logic of the final generated report at the semantic level; large-scale language models are more likely to generate report content that conforms to the actual disease progression under the guidance of evolutionary relationships, thus enhancing diagnostic auxiliary value; Each module of this invention can be replaced with different neural network structures or inference models, enabling it to adapt to various medical image types and clinical scenarios. This invention is an accurate and efficient method for generating medical image reports based on cross-temporal disease evolution modeling.

[0057] Figure 3 This is a block diagram of a medical image report generation device based on analogical reasoning for disease evolution perception, provided in an embodiment of the present invention. This device is used in a method for generating medical image reports based on analogical reasoning for disease evolution perception. (Refer to...) Figure 3 The device includes an image and report acquisition module 310, an image feature alignment module 320, a visual evolution calibration module 330, a text feature extraction module 340, a cross-modal alignment module 350, a language modeling loss calculation module 360, a parameter optimization module 370, and an image report generation module 380. Among them:

[0058] The image and report acquisition module 310 is used to acquire the patient's current medical images, current medical image reports, historical medical images, and historical medical image reports. The image feature alignment module 320 is used to extract features using a visual encoder based on the current medical image and historical medical images to obtain the current local visual feature vector, the current global visual feature vector, the historical local visual feature vector, and the historical global visual feature vector; and to align key lesion regions using an adaptive region alignment module based on the current local visual feature vector and the historical local visual features to obtain the aligned current image feature vector and the aligned historical image feature vector. The visual evolution calibration module 330 is used to model the evolution relationship based on the visual evolution consistency loss function, according to the current global visual feature vector, the historical global visual feature vector, the aligned current image feature vector, and the aligned historical image feature vector, and to obtain the visual evolution relationship vector and the visual evolution consistency loss. The text feature extraction module 340 is used to extract features based on the current medical image report and historical medical image reports using a text encoder to obtain the current text feature vector and the historical text feature vector. The cross-modal alignment module 350 is used to perform evolution relationship mapping using a relationship mapper based on the text evolution consistency loss function, according to the current text feature vector, historical text feature vector and visual evolution relationship vector, to obtain the text evolution relationship vector and text evolution consistency loss. The language modeling loss calculation module 360 ​​is used to infer the language modeling loss based on the autoregressive language modeling loss function, using a large language model to perform inference based on the current medical image, historical medical image reports, text evolution relationship vectors and the current medical image report; The parameter optimization module 370 is used to optimize the parameters of the visual encoder, adaptive region alignment module, evolutionary relation learner, relation mapper and large language model based on the joint optimization loss function, according to the visual evolution consistency loss, text evolution consistency loss and language modeling loss, to obtain optimized visual encoder, optimized adaptive region alignment module, optimized evolutionary relation learner, optimized relation mapper and optimized large language model. The image report generation module 380 is used to acquire the target patient's current medical images, target patient's historical medical images, and target patient's historical medical image reports; based on the optimized visual encoder, optimized adaptive region alignment module, optimized evolutionary relation learner, text encoder, optimized relation mapper, and optimized large language model, the module generates a report according to the target patient's current medical images, target patient's historical medical images, and target patient's historical medical image reports.

[0059] Among them, evolutionary relation learners include multilayer perceptrons, visual Transformers, or graph convolutional neural networks.

[0060] Optionally, the visual evolution calibration module 330 is further used for: Based on the aligned current image feature vector and the aligned historical image feature vector, an evolutionary relationship learner is used to extract cross-temporal phase relationships to obtain a visual evolutionary relationship vector. Based on the visual evolution consistency loss function, the visual evolution consistency loss is calculated according to the current global visual feature vector, the historical global visual feature vector, and the visual evolution relationship vector.

[0061] The relation mapper is a fully connected layer that incorporates mapping functions.

[0062] Optionally, the cross-modal alignment module 350 is further used for: Based on the visual evolution relation vector, a relation mapper is used to perform cross-modal text vector mapping to obtain the text evolution relation vector. Based on the text evolution consistency loss function, the text evolution consistency loss is calculated according to the current text feature vector, historical text feature vector and text evolution relationship vector.

[0063] Optionally, the parameter optimization module 370 is further used for: Based on the joint optimization loss function, the joint optimization loss is obtained by weighting the visual evolution consistency loss, text evolution consistency loss and language modeling loss. Based on the joint optimization loss, the parameters of the visual encoder, adaptive region alignment module, evolutionary relation learner, relation mapper, and large language model are optimized to obtain optimized visual encoder, optimized adaptive region alignment module, optimized evolutionary relation learner, optimized relation mapper, and optimized large language model.

[0064] This invention proposes a medical impact report generation method based on analogical reasoning for disease evolution perception. By modeling visual evolution relationships and text evolution relationships, the model can capture changes in images over time, improving the ability to identify disease progression or improvement. The adaptive region alignment module can automatically align lesion areas, effectively reducing cross-temporal feature shifts caused by changes in body position and shooting angle, and improving the robustness of cross-temporal analysis. By introducing a cross-modal evolution consistency mechanism, a correspondence is established between structured visual changes and textual expression changes, thereby improving the temporal logic of the final generated report at the semantic level; large-scale language models are more likely to generate report content that conforms to the actual disease progression under the guidance of evolutionary relationships, thus enhancing diagnostic auxiliary value; Each module of this invention can be replaced with different neural network structures or inference models, enabling it to adapt to various medical image types and clinical scenarios. This invention is an accurate and efficient method for generating medical image reports based on cross-temporal disease evolution modeling.

[0065] Figure 4 This is a schematic diagram of the structure of a medical image report generation device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the medical image report generation device may include the above-mentioned Figure 3 The illustrated medical image report generation device is based on analogical reasoning and is designed to perceive disease evolution. Optionally, the medical image report generation device 410 may include a first processor 2001.

[0066] Optionally, the medical image report generation device 410 may also include a memory 2002 and a transceiver 2003.

[0067] The first processor 2001, memory 2002, and transceiver 2003 can be connected via a communication bus.

[0068] The following is combined with Figure 4 A detailed description of each component of the medical image report generation device 410 is provided below: The first processor 2001 is the control center of the medical image report generation device 410. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0069] Optionally, the first processor 2001 can perform various functions of the medical image report generation device 410 by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0070] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0071] In a specific implementation, as one example, the medical image report generation device 410 may also include multiple processors, such as... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0072] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0073] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently and be connected to the interface circuit of the medical image report generation device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0074] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0075] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0076] Optionally, the transceiver 2003 can be integrated with the first processor 2001, or it can exist independently and be connected to the interface circuit of the medical image report generation device 410. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0077] It should be noted that, Figure 4 The structure of the medical image report generation device 410 shown does not constitute a limitation on the router. Actual medical image report generation devices may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0078] Furthermore, the technical effects of the medical image report generation device 410 can be referenced from the technical effects of the medical image report generation method based on analogical reasoning for disease evolution perception described in the above method embodiments, and will not be repeated here.

[0079] It should be understood that the first processor 2001 in the embodiments of the present invention may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or it may be any conventional processor, etc.

[0080] It should also be understood that the memory in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0081] The above embodiments can be implemented, in whole or in part, by software, hardware (such as circuits), firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0082] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0083] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0084] It should be understood that, in various embodiments of the present invention, the order of the above-mentioned process numbers does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0085] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0086] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0087] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0088] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0089] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0090] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0091] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for generating medical image reports based on analogical reasoning for disease evolution perception, characterized in that, The method includes: Obtain the patient's current medical images, current medical image reports, historical medical images, and historical medical image reports; Based on current and historical medical images, a visual encoder is used to extract features, obtaining current local visual feature vectors, current global visual feature vectors, historical local visual feature vectors, and historical global visual feature vectors. Based on the current local visual feature vectors and historical local visual features, an adaptive region alignment module is used to align key lesion regions, obtaining aligned current image feature vectors and aligned historical image feature vectors. Based on the visual evolution consistency loss function, an evolution relationship learner is used to model the evolution relationship according to the current global visual feature vector, the historical global visual feature vector, the aligned current image feature vector, and the aligned historical image feature vector, to obtain the visual evolution relationship vector and the visual evolution consistency loss. Based on current and historical medical image reports, a text encoder is used to extract features to obtain current text feature vectors and historical text feature vectors. Based on the text evolution consistency loss function, an evolution relationship mapping is performed using a relationship mapper according to the current text feature vector, historical text feature vector and visual evolution relationship vector to obtain the text evolution relationship vector and text evolution consistency loss. Based on the autoregressive language modeling loss function, the language modeling loss is obtained by using a large language model to infer from the current medical images, historical medical image reports, text evolution relationship vectors and the current medical image reports. Based on the joint optimization loss function, the parameters of the visual encoder, adaptive region alignment module, evolutionary relation learner, relation mapper, and large language model are optimized according to the visual evolution consistency loss, text evolution consistency loss, and language modeling loss, resulting in optimized visual encoder, optimized adaptive region alignment module, optimized evolutionary relation learner, optimized relation mapper, and optimized large language model. The system acquires the target patient's current medical images, historical medical images, and historical medical image reports. Based on an optimized visual encoder, an optimized adaptive region alignment module, an optimized evolutionary relation learner, a text encoder, an optimized relation mapper, and an optimized large language model, the system generates a report using the target patient's current medical images, historical medical images, and historical medical image reports.

2. The medical image report generation method based on analogical reasoning for disease evolution perception according to claim 1, characterized in that, The evolutionary relation learner includes a multilayer perceptron, a visual Transformer, or a graph convolutional neural network.

3. The medical image report generation method based on analogical reasoning for disease evolution perception according to claim 1, characterized in that, The method based on the visual evolution consistency loss function, using an evolution relationship learner to model evolution relationships based on the current global visual feature vector, historical global visual feature vector, aligned current image feature vector, and aligned historical image feature vector, obtains a visual evolution relationship vector and a visual evolution consistency loss, including: Based on the aligned current image feature vector and the aligned historical image feature vector, an evolutionary relationship learner is used to extract cross-temporal phase relationships to obtain a visual evolutionary relationship vector. Based on the visual evolution consistency loss function, the visual evolution consistency loss is calculated according to the current global visual feature vector, the historical global visual feature vector, and the visual evolution relationship vector.

4. The medical image report generation method based on analogical reasoning for disease evolution perception according to claim 1, characterized in that, The relation mapper is a fully connected layer that incorporates mapping functions.

5. The medical image report generation method based on analogical reasoning for disease evolution perception according to claim 1, characterized in that, The text evolution consistency loss function, based on the current text feature vector, historical text feature vector, and visual evolution relationship vector, uses a relationship mapper to perform evolution relationship mapping to obtain the text evolution relationship vector and the text evolution consistency loss, including: Based on the visual evolution relation vector, a relation mapper is used to perform cross-modal text vector mapping to obtain the text evolution relation vector. Based on the text evolution consistency loss function, the text evolution consistency loss is calculated according to the current text feature vector, historical text feature vector and text evolution relationship vector.

6. The medical image report generation method based on analogical reasoning for disease evolution perception according to claim 1, characterized in that, The method, based on a joint optimization loss function, optimizes the parameters of the visual encoder, adaptive region alignment module, evolutionary relation learner, relation mapper, and large language model according to visual evolution consistency loss, text evolution consistency loss, and language modeling loss, to obtain optimized visual encoder, optimized adaptive region alignment module, optimized evolutionary relation learner, optimized relation mapper, and optimized large language model, including: Based on the joint optimization loss function, the joint optimization loss is obtained by weighting the visual evolution consistency loss, text evolution consistency loss and language modeling loss. Based on the joint optimization loss, the parameters of the visual encoder, adaptive region alignment module, evolutionary relation learner, relation mapper, and large language model are optimized to obtain optimized visual encoder, optimized adaptive region alignment module, optimized evolutionary relation learner, optimized relation mapper, and optimized large language model.

7. A medical image report generation device based on analogical reasoning for disease evolution perception, wherein the medical image report generation device based on analogical reasoning for disease evolution perception is used to implement the medical image report generation method based on analogical reasoning for disease evolution perception as described in any one of claims 1-7, characterized in that, The device includes: The image and report acquisition module is used to acquire the patient's current medical images, current medical image reports, historical medical images, and historical medical image reports. The image feature alignment module is used to extract features from the current medical image and historical medical images using a visual encoder to obtain the current local visual feature vector, the current global visual feature vector, the historical local visual feature vector, and the historical global visual feature vector; and to align key lesion regions using an adaptive region alignment module based on the current local visual feature vector and historical local visual features to obtain the aligned current image feature vector and the aligned historical image feature vector. The visual evolution calibration module is used to model evolution relationships based on the visual evolution consistency loss function, using an evolution relationship learner to obtain the visual evolution relationship vector and the visual evolution consistency loss, according to the current global visual feature vector, the historical global visual feature vector, the aligned current image feature vector, and the aligned historical image feature vector. The text feature extraction module is used to extract features from the current medical image report and historical medical image reports using a text encoder to obtain the current text feature vector and the historical text feature vector. The cross-modal alignment module is used to perform evolution relationship mapping based on the text evolution consistency loss function, according to the current text feature vector, historical text feature vector and visual evolution relationship vector, using a relationship mapper to obtain the text evolution relationship vector and text evolution consistency loss; The language modeling loss calculation module is used to obtain the language modeling loss based on the autoregressive language modeling loss function, using a large language model to infer from the current medical image, historical medical image reports, text evolution relationship vectors and the current medical image report. The parameter optimization module is used to optimize the parameters of the visual encoder, adaptive region alignment module, evolutionary relation learner, relation mapper, and large language model based on the joint optimization loss function, according to the visual evolution consistency loss, text evolution consistency loss, and language modeling loss, to obtain optimized visual encoder, optimized adaptive region alignment module, optimized evolutionary relation learner, optimized relation mapper, and optimized large language model. The image report generation module is used to acquire the target patient's current medical images, target patient's historical medical images, and target patient's historical medical image reports. Based on the optimized visual encoder, optimized adaptive region alignment module, optimized evolutionary relation learner, text encoder, optimized relation mapper, and optimized large language model, the module generates reports according to the target patient's current medical images, target patient's historical medical images, and target patient's historical medical image reports.

8. The medical image report generation device for disease evolution perception based on analogical reasoning according to claim 7, characterized in that, The parameter optimization module is further used for: Based on the joint optimization loss function, the joint optimization loss is obtained by weighting the visual evolution consistency loss, text evolution consistency loss and language modeling loss. Based on the joint optimization loss, the parameters of the visual encoder, adaptive region alignment module, evolutionary relation learner, relation mapper, and large language model are optimized to obtain optimized visual encoder, optimized adaptive region alignment module, optimized evolutionary relation learner, optimized relation mapper, and optimized large language model.

9. A medical image report generation device, characterized in that, The medical image report generation device includes: processor; A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1 to 7.

Images