Physical examination report interpretation method and device, electronic equipment and storage medium

Abstract

The invention provides a physical examination report interpretation method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: disassembling a to-be-interpreted physical examination report into at least text content, picture content and image content; inputting the text content, the picture content and the image content into a multi-modal analysis model to obtain a structured information abstract; and generating an interpretation instruction text based on the structured information abstract, inputting the interpretation instruction text into a report interpretation model, and obtaining report interpretation content. Wherein the multi-modal analysis model comprises an encoder cluster and a feature fusion unit. According to the method, cross-modal attention is applied to the multi-modal analysis model, abnormal modal features in a physical examination report can be automatically recognized, the weight of the abnormal modal features can be improved, multi-feature fusion can better meet the physical examination interpretation requirement, multi-modal chain type enhancement is formed through mutual guidance of image, image and text features, the limitation of single-modal coding is broken through, and the method has the advantages of being high in practicability and the like. And information omission caused by a single mode is avoided.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and storage medium for interpreting medical examination reports. Background Technology

[0002] As an important means of preventing diseases and monitoring health status, the interpretation of physical examination reports is crucial.

[0003] Traditional medical examination report interpretation relies primarily on manual operation by doctors. Doctors use their professional knowledge and clinical experience to analyze and interpret the textual descriptions, numerical indicators, and imaging results in the report. However, this model has significant drawbacks. Firstly, from an efficiency standpoint, manual interpretation is time-consuming and labor-intensive. Secondly, it is heavily influenced by subjective factors. Different doctors have varying levels of professional competence and experience, resulting in different sensitivities to abnormal indicators and differing interpretation focuses, potentially leading to biased assessments of patient health risks. This uncertainty confuses patients and undermines the consistency and scientific rigor of medical decisions.

[0004] Furthermore, traditional methods struggle to deeply mine unstructured information in medical examination reports. Most reports contain a large amount of complex data, including text descriptions, images, and pictures. It is difficult for humans to quickly sort out the relationships between these data and extract key information, thus failing to fully realize the potential value of all the data in the reports and hindering the advancement of precision medicine and personalized medicine. Summary of the Invention

[0005] This invention provides a method, device, electronic device, and storage medium for interpreting physical examination reports, which solves the shortcomings of traditional physical examination reports that rely on manual interpretation by doctors, which are time-consuming, labor-intensive, and subject to serious interference from subjective factors, potentially leading to biases in the assessment of patients' health risks. It can effectively improve the comprehensive and accurate interpretation of physical examination reports.

[0006] This invention provides a method for interpreting a medical examination report, comprising the following steps: The medical examination report to be interpreted is broken down into multimodal content; The multimodal content is input into a multimodal parsing model to obtain a structured information summary output by the multimodal parsing model; An interpretation instruction text is generated based on the structured information digest, and the interpretation instruction text is input into the report interpretation model to obtain the report interpretation content output by the report interpretation model; The multimodal parsing model includes an encoder cluster and a feature fusion unit. The encoder cluster extracts features for each modality in the multimodal content. The feature fusion unit generates the structured information summary by mapping weighted modal features to a unified multimodal embedding space. The weighted modal features are obtained by weighting the initial modal features output by the encoder cluster based on the attention weights between the modal features.

[0007] According to the present invention, a method for interpreting a medical examination report is provided, wherein the multimodal content includes multiple types of text content, image content, and video content.

[0008] According to a method for interpreting a medical examination report provided by the present invention, the step of generating interpretation instruction text based on the structured information digest includes: Obtain user profile information, wherein the user profile information is the profile information of the user of the physical examination report; Determine the target prompt template from a pre-built instruction configuration strategy; The target prompt template is used to integrate the user profile information and the structured information summary into the interpretation instruction text.

[0009] According to a method for interpreting a medical examination report provided by the present invention, the step of determining a target prompt template from a pre-built instruction configuration strategy includes: Determine the type of health checkup package for the user; The target prompt template is obtained by mapping the physical examination package type from the instruction configuration strategy.

[0010] According to a method for interpreting a physical examination report provided by the present invention, the encoder cluster includes a text encoder, an image encoder, and an image encoder; the text encoder is a network model based on the Transformer architecture, the image encoder is a visual Transformer model, and the image encoder is a three-dimensional convolutional neural network.

[0011] According to a method for interpreting a medical examination report provided by the present invention, the multimodal content includes at least text content; before inputting the multimodal content into a multimodal parsing model, the method further includes: performing word segmentation on the text content using byte pair encoding.

[0012] According to a method for interpreting a medical examination report provided by the present invention, the step of extracting features from the image content using the visual Transformer model includes: For each image in the image content, it is divided into multiple image blocks; A self-attention mechanism is used to calculate the association weights between the multiple image patches, and the feature representations of the multiple image patches are aggregated based on the association weights to generate an image feature vector representing each image.

[0013] According to a method for interpreting a medical examination report provided by the present invention, the step of inputting the multimodal content into a multimodal parsing model and obtaining a structured information summary output by the multimodal parsing model includes: The text content, the image content, and the video content are input into the encoder cluster of the multimodal parsing model to obtain the initial modal features of the text, the initial modal features of the image, and the initial modal features of the video output by the multimodal parsing model. Determine the first attention weight between image features and text features and image features, the second attention weight between image features and text features and image features, and the third attention weight between text features and image features and image features. Based on the first attention weight, the second attention weight, and the third attention weight, the initial text modal features, the initial image modal features, and the initial video modal features are weighted and fused to obtain weighted text modal features, weighted image modal features, and weighted video modal features; The text-weighted modal features, image-weighted modal features, and video-weighted modal features are mapped to the multimodal embedding space and then fused to obtain the structured information summary.

[0014] According to the present invention, a method for interpreting a medical examination report is provided, wherein the multimodal analytical model is trained based on the following steps: Construct a training dataset, wherein the training data in the training dataset includes at least image-text pairs consisting of image samples and corresponding text samples, and image-text pairs consisting of video samples and corresponding text samples. The training data in the training dataset is used as the input to the multimodal parsing model, and the contrastive loss is calculated based on the output of the multimodal parsing model. Based on the contrast loss, the trainable parameters of the text encoder, the image encoder, the video encoder, and the feature fusion unit are jointly updated.

[0015] According to a method for interpreting a medical examination report provided by the present invention, the step of calculating the contrast loss based on the output of a multimodal analytical model includes: Obtain the image features generated by the image encoder processing the image samples in the image-text pair, and obtain the image features generated by the image encoder processing the image samples in the image-text pair; Obtain the image text features and image text features generated by the text encoder after processing the text samples in the image text pair and the text samples in the image text pair respectively; Calculate the image-text contrast loss between the image features and the image-text features, and calculate the image-text contrast loss between the image features and the image-text features; The sum of the image-text contrast loss and the video-text contrast loss is taken as the contrast loss.

[0016] According to a method for interpreting a medical examination report provided by the present invention, before calculating the image-text contrast loss between the image features and the image-text features, it further includes... The image features, the video features, the image text features, and the video text features are projected into the multimodal embedding space using their respective projection matrices. The image embedding representation, video embedding representation, image-text embedding representation, and video-text embedding representation obtained after projection are subjected to L2 normalization to obtain normalized features. The image-text contrast loss and the video-text contrast loss are calculated based on the normalized features.

[0017] According to a method for interpreting a medical examination report provided by the present invention, the image-text contrast loss is determined based on the following steps: Based on the image features and the image text features, an image-text similarity matrix is ​​calculated. The first-direction loss and the second-direction loss are calculated based on the image-text similarity matrix, respectively. The first directional loss is determined based on the similarity between each image sample and its matching and non-matching text samples calculated from the image-text similarity matrix; the second directional loss is determined based on the similarity between each text sample and its matching and non-matching image samples calculated from the image-text similarity matrix. The first directional loss is combined with the second directional loss to obtain the image-text contrast loss.

[0018] According to a method for interpreting a medical examination report provided by the present invention, the image-text contrast loss is determined based on the following steps: Based on the image features and the image text features, an image-text similarity matrix is ​​calculated. The third-direction loss and the fourth-direction loss are calculated based on the image-text similarity matrix. The third-direction loss is determined based on the similarity between each image sample and its matching and non-matching text samples calculated from the image-text similarity matrix; the fourth-direction loss is determined based on the similarity between each text sample and its matching and non-matching image samples calculated from the image-text similarity matrix. The third-direction loss is combined with the fourth-direction loss to obtain the image-text contrast loss.

[0019] According to a method for interpreting a medical examination report provided by the present invention, the step of jointly updating the trainable parameters of the text encoder, the image encoder, the video encoder, and the feature fusion unit based on the contrast loss specifically includes: A backpropagation algorithm is used to calculate the gradient of the contrast loss with respect to the trainable parameters of the text encoder, the image encoder, the video encoder, and the feature fusion unit; An optimizer is used to adjust the trainable parameters based on the gradient; The step of calculating the contrastive loss is performed iteratively and repeatedly for different batches of training data in the training dataset until the step of adjusting the trainable parameters according to the gradient is met, until a preset convergence condition is satisfied.

[0020] According to a method for interpreting a medical examination report provided by the present invention, after obtaining the report interpretation content output by the report interpretation model, the method further includes: Based on the preset application scenario requirements, the report interpretation content is formatted and processed to generate the final physical examination report; The formatting post-processing includes converting the numerical data contained in the report interpretation content into a preset text description; And / or, generate visualization charts based on the data in the report interpretation content.

[0021] The present invention also provides a medical examination report interpretation device, comprising: The report retrieval unit is used to break down the medical examination report to be interpreted into multimodal content; An information extraction unit is used to input the multimodal content into a multimodal parsing model to obtain a structured information summary output by the multimodal parsing model. The report interpretation unit is used to generate interpretation instruction text based on the structured information digest, and input the interpretation instruction text into the report interpretation model to obtain the report interpretation content output by the report interpretation model; The multimodal parsing model includes an encoder cluster and a feature fusion unit. The encoder cluster extracts features for each modality in the multimodal content. The feature fusion unit generates the structured information summary by mapping weighted modal features to a unified multimodal embedding space. The weighted modal features are obtained by weighting the initial modal features output by the encoder cluster based on the attention weights between the modal features.

[0022] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the medical examination report interpretation method described above.

[0023] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the medical examination report interpretation method as described above.

[0024] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the medical examination report interpretation method as described above.

[0025] The present invention provides a method, apparatus, electronic device, and storage medium for interpreting physical examination reports. By applying cross-modal attention in a multimodal parsing model, it can automatically identify abnormal modal features in physical examination reports and increase their weights, making multi-feature fusion more in line with the needs of physical examination interpretation. It realizes the mutual guidance of image, video, and text features to form a multimodal chain enhancement, breaks through the limitations of single-modal coding, and avoids information omissions caused by single modality. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0027] Figure 1 This is one of the flowcharts illustrating the method for interpreting physical examination reports provided by this invention.

[0028] Figure 2 This is a schematic diagram of the data processing flow of the multimodal analytical model provided by the present invention.

[0029] Figure 3 This is the second flowchart of the method for interpreting physical examination reports provided by the present invention.

[0030] Figure 4This is the third flowchart of the method for interpreting physical examination reports provided by this invention.

[0031] Figure 5 This is a schematic diagram of the training process of the multimodal analytical model provided by the present invention.

[0032] Figure 6 This is a schematic diagram of the contrastive loss calculation process during the training of the multimodal analytical model provided by the present invention.

[0033] Figure 7 This is a schematic diagram of the calculation process for image-text contrast loss provided by the present invention.

[0034] Figure 8 This is a schematic diagram of the calculation process for image-text contrast loss provided by the present invention.

[0035] Figure 9 This is a schematic diagram of the structure of the medical examination report interpretation device provided by the present invention.

[0036] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0038] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0039] The terms "first," "second," etc., used in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0040] Physical examinations are an important means of disease prevention and health monitoring, and the interpretation of examination reports is crucial. Currently, the interpretation of physical examination reports mainly relies on manual operation by doctors, who analyze and judge the textual descriptions, numerical indicators, and imaging results in the report based on their professional knowledge and clinical experience. However, this interpretation method has significant drawbacks: on the one hand, from an efficiency perspective, manual interpretation is time-consuming and labor-intensive; on the other hand, it is heavily influenced by subjective factors. Different doctors have different professional competence and experience, and their sensitivity to abnormal indicators and their focus of interpretation may vary, potentially leading to biases in the assessment of patients' health risks. This uncertainty confuses patients and is detrimental to the consistency and scientific nature of medical decisions.

[0041] Furthermore, traditional methods struggle to deeply mine unstructured information from medical examination reports. Most reports contain a large amount of complex data, including text descriptions, images, and pictures, making it difficult for humans to quickly analyze the relationships and extract key information. This hinders the full realization of the potential value of the data and impedes the advancement of precision and personalized medicine.

[0042] With the rapid development of artificial intelligence technology, the healthcare field has also ushered in technological innovation. Large-scale multimodal technology, with its superior automatic feature extraction and complex model construction capabilities, can learn latent patterns from massive amounts of data and multiple modalities. Natural language processing can understand text semantics, transforming medical terminology, symptom descriptions, and indicator parameters into structured knowledge. Image recognition can accurately analyze imaging examination results. Integrating these technologies into the field of medical examination report analysis and interpretation has become a key path to overcome current difficulties and improve the quality of medical services. Furthermore, different groups (such as age, gender, occupation, etc.) may have different interpretation needs and focuses regarding the same examination results.

[0043] Currently, the main methods for interpreting medical examination reports on the market include: manual interpretation, rule-based interpretation, machine learning / deep learning model interpretation, and large-scale model interpretation. Manual interpretation relies primarily on the professional knowledge and experience of doctors. Doctors will comprehensively analyze various indicators in the report, such as blood tests, imaging examinations, and electrocardiograms, combined with the patient's medical history, family history, and personal lifestyle habits, to provide diagnostic opinions or health advice. Rule-based interpretation uses a dictionary of items to perform semantic analysis of the text, first finding the item name and then matching the corresponding examination results; or it uses a pre-set template to extract data. Machine learning / deep model interpretation uses machine learning models (such as CRF), deep learning models (such as LSTM), or a combination of these models (such as LSTM+CRF) to extract text and tables from the medical examination report, and then uses model inference to provide an interpretation of the extracted text.

[0044] However, the above-mentioned methods for interpreting medical examination reports have some shortcomings. For example, they can only parse text content and do not support the interpretation of image and picture content. They are limited to a single scenario, lack versatility, and cannot provide personalized interpretations for patients' conditions.

[0045] The present invention provides a method, device, electronic device, and storage medium for interpreting medical examination reports, relating to fields such as natural language processing and smart healthcare. It combines large-scale multimodal technology, natural language processing, and image recognition technology to design a multimodal parsing model for general medical examination report parsing needs. This model can comprehensively interpret text content, image content, and picture content, effectively solving the challenges of structured processing and accurate personalized interpretation of medical examination reports, thus meeting the urgent need for efficient, accurate analysis, and personalized expression of medical examination reports in modern medical scenarios. The following is a detailed explanation... Figures 1-10 This invention describes the method, apparatus, electronic device, and storage medium for interpreting medical examination reports.

[0046] Figure 1 This is one of the flowcharts illustrating the method for interpreting medical examination reports provided by this invention, such as... Figure 1 As shown, this embodiment of the invention provides a method for interpreting physical examination reports, aiming to solve the problems of low interpretation efficiency, inability to comprehensively process multi-source heterogeneous information, and difficulty in achieving standardization and personalization in the prior art.

[0047] In one specific embodiment, this method can be applied to a smart healthcare platform or a doctor-assisted diagnostic system to automate and intelligently perform in-depth analysis and interpretation of physical examination reports. The method may include the following steps: Step 11: Decompose the medical examination report to be interpreted into multimodal content. This step is a preprocessing process for the original medical examination report. Its purpose is to decompose a medical examination report that is usually mixed in format and has diverse content into a standardized multimodal data stream that can be processed by the subsequent multimodal parsing model.

[0048] The medical examination report to be interpreted can be a multi-page PDF electronic file provided by the medical examination center, usually containing various forms of information. This report not only includes the user's basic information and the values ​​of various test indicators, but may also embed static screenshots such as chest X-rays and abdominal ultrasounds, and may even contain some video content.

[0049] As an optional embodiment, the multimodal content includes multiple types of content such as text, images, and video.

[0050] Text content refers to the data presented in text form in a medical examination report, including, for example, the names of the examination items, the numerical values ​​of the results, and the doctor's diagnosis and recommendations. This type of data is usually in natural language and generally requires structured extraction using natural language processing techniques.

[0051] Image content refers to static image data included in a physical examination report, such as electrocardiogram waveforms, X-ray or CT scan slice images, and pathological slide microscope images. This type of data can exist in the form of two-dimensional images, and its features can generally be extracted through image recognition technology.

[0052] Image content refers to the dynamic images or multi-slice tomography images included in the physical examination report, such as dynamic videos of echocardiography, multi-slice scan image sequences of CT or MRI, angiography videos, etc. This type of data generally contains temporal or three-dimensional spatial structural information, and features can be extracted through specialized image analysis techniques.

[0053] The breakdown of a medical examination report can be performed automatically by the system. For example, upon receiving the report, the system first segments it by page, then applies layout analysis technology to each page to identify and separate different types of information areas. For instance, it can identify the table area at the top of a page, which contains textual information about blood tests (such as "Hemoglobin: 150 g / L"); a rectangular area in the middle is identified as image content, i.e., a thumbnail of a chest X-ray; and if the report contains a QR code or link to a dynamic echocardiogram, the video data obtained after parsing the link is categorized as image content.

[0054] It should be noted that the image content in this invention is fundamentally different from static image content. It contains continuous change information over time (such as the dynamic opening and closing of heart valves within a cardiac cycle), providing richer diagnostic evidence. Through this decomposition, a complex medical examination report is clearly divided into three major data streams: text content, image content, and video content, laying the foundation for subsequent accurate analysis.

[0055] Of course, when breaking down the content of the medical examination report into different modalities, it is not limited to the three modalities mentioned above. Depending on the actual application scenario, it may also include other forms of information, such as audio information or other types of sensor data.

[0056] Step 12: Input the multimodal content into the multimodal parsing model to obtain a structured information digest.

[0057] After deconstructing the medical examination report, this step utilizes a core multimodal parsing model to perform parallel and in-depth feature extraction and information fusion on the different modalities of data produced in the previous step, ultimately generating a unified, machine-readable structured information summary.

[0058] Figure 2 This is a schematic diagram of the data processing flow of the multimodal analytical model provided by the present invention, as shown below. Figure 2 As shown, the internal working mechanism of the multimodal analytical model can be understood more clearly. This multimodal analytical model is not a single model, but a complex system that integrates multiple dedicated encoders and fusion modules. Specifically, it mainly consists of two parts: an encoder cluster and a feature fusion unit.

[0059] The encoder cluster is responsible for performing preliminary, independent feature extraction on the different modal contents decomposed in the previous step, generating their respective initial modal features. In a specific embodiment, this mainly includes the following parts: A text encoder, for text content, can employ a network model based on the Transformer architecture (such as BERT, RoBERTa, etc.) to convert text sequences into high-dimensional initial text modal features. For example, the text "high-density nodular shadows visible in the lungs" would be processed by this text encoder and transformed into a high-dimensional feature vector that captures its medical semantics.

[0060] An image encoder, used to process image content, divides each input image into multiple fixed-size patches. It then calculates the association weights between these patches using its internal self-attention mechanism, ultimately aggregating them to generate initial modal features representing the entire image content. For example, the aforementioned chest X-ray image would be input into this image encoder to extract the texture, shape, and location features visually corresponding to the "high-density nodule shadow," and represent these features as a feature vector.

[0061] An image encoder is used to process image content, including capturing information about changes in the image in space and time (or sequence level) and generating initial modal features of the image. For example, video clips of dynamic echocardiography would be processed by this image encoder, which not only analyzes each frame of the image, but more importantly, captures the dynamic changes between frames, such as the motion coordination of the ventricular walls, and generates corresponding feature vectors.

[0062] The feature vectors extracted by the three encoders mentioned above have different data structures and mathematical spaces due to the differences in their modal origins, making direct comparison and fusion impossible. Therefore, a crucial feature fusion unit is also included in the multimodal analytical model.

[0063] The core function of the feature fusion unit is to break down the barriers between modalities, allowing features from different sources to perceive and enhance each other, and ultimately merge into a unified and more informative expression, namely structured information summarization.

[0064] In the process of performing structured information summarization, the feature fusion unit of the present invention introduces a multi-head cross-attention (MHCA) mechanism, which allows different modal features to perceive the importance of other modalities in order to calculate the degree of correlation between any two modalities.

[0065] By calculating the attention weights of image features on the other two modal features (image features and text features), calculating the attention weights of image features on the other two modal features (image features and text features), and calculating the attention weights of text features on the other two modal features (image features and video features), these attention weights quantify the degree of attention or importance that a feature point in any modality has to all feature points in another modality.

[0066] After obtaining the attention weights among the modal features, the system uses these attention weights to dynamically adjust and fuse the initial modal features to generate weighted modal features. This can be achieved by combining the initial features of one modality with other modal features that have been weighted by their attention weights.

[0067] For example, when processing physical examination data for lung nodules, the input CT image sequence contains detailed malignant features of the nodules (such as lobulation and spiculation). The attention weight of the initial image modality features to the initial text modality features calculated by this process will be very high because it focuses on the descriptions of possible malignancy in the initial text modality features of the CT report. The weighted modality features generated after fusion simultaneously integrate the visual details of the images and the diagnostic tendencies of the text, greatly enhancing the expressive power.

[0068] To enable the comparison and eventual fusion of weighted modal features from different sources within a unified metric space, they need to be mapped to a unified multimodal embedding space, including: By linearly projecting weighted modal features with potentially different dimensions and distributions onto the same target embedding dimension using different projection matrices, corresponding embedding representations are obtained. Then, the projected embedding representations are normalized to eliminate the influence of modulus length, ensuring that subsequent similarity calculations depend only on direction, i.e., the semantics themselves. After this process, normalized image embedding representations, video embedding representations, and text embedding representations are obtained. These are all in the same high-dimensional space, and semantically related embedding representations of different modalities will be spatially closer.

[0069] Ultimately, these feature vectors, aligned and fused in a unified embedding space, together constitute the highly condensed and comprehensive structured information summary output by the model.

[0070] The resulting structured information summary is the output of a multimodal parsing model after a deep understanding of the entire medical examination report. It is no longer fragmented text or pixels, but rather highly formatted data, such as a JSON object, whose content might be: {"finding_id": "001", "location": "Right lower lobe of lung", "description": "High-density nodule shadow", "type": "Abnormal", "source_image": "page2_img1.png", "suggestion": "CT scan recommended"}. This structured information summary provides a clean, accurate, and machine-processable data foundation for subsequent personalized interpretation.

[0071] Step 13: Generate interpretation instruction text based on the structured information digest, and input the interpretation instruction text into the report interpretation model to obtain the report interpretation content output by the report interpretation model.

[0072] First, the system generates interpretation instruction text based on the obtained structured information summary. This interpretation instruction text can be understood as a designed task specification to guide the downstream report interpretation model in its thinking and expression. For example, the system can automatically construct the following interpretation instruction text based on the structured information summary: "You are a senior respiratory specialist. Please generate a clear and professional health interpretation for the patient based on the following structured information summary. The structured information summary is as follows: A high-density nodule was found in the lower right lobe of the patient's lung, and a follow-up CT scan is recommended. Please explain what this means and what the next steps should be."

[0073] Subsequently, this interpretation instruction text, containing clear roles, tasks, and core information, is input into the report interpretation model. The final text output by the model is then obtained, representing the final report interpretation content. For example, the report interpretation model might generate the following interpretation: "Hello, according to your physical examination report, a small 'nodular shadow' was found in the lower right lung. This is an imaging description, and it could have many causes, but in most cases it is benign, such as a scar left by old inflammation. However, to be absolutely certain, the doctor recommends that you have a CT scan to further clarify its specific situation. Please don't be overly anxious; just follow the doctor's instructions for the follow-up examination."

[0074] The report interpretation model can be a large language model, such as a pre-trained large language model with deep medical knowledge.

[0075] The method for interpreting physical examination reports provided by this invention comprehensively analyzes and integrates data from three modalities: text, images, and video. This enables a comprehensive and accurate interpretation of physical examination reports, avoiding information omissions caused by a single modality, and laying a solid foundation for providing personalized health management recommendations.

[0076] Most importantly, by applying cross-modal attention in the multimodal parsing model, this invention can automatically identify abnormal modal features in the physical examination report and increase their weights, making multi-feature fusion more in line with the needs of physical examination interpretation. It enables the mutual guidance of image, video, and text features to form a multimodal chain enhancement, breaking through the limitations of single-modal encoding and avoiding information omissions caused by a single modality.

[0077] It should be noted that although the various embodiments of the present invention are mainly described using text content, image content, and video content as typical examples of multimodal content, this does not imply any limitation on the present invention. The medical examination report interpretation framework proposed in this invention, especially its multimodal parsing model containing encoder clusters and feature fusion units, has good technical scalability. The multimodal content can fully cover any other medical data modality that can be digitized and have its features extracted.

[0078] For example, the multimodal content may also include audio information, such as a doctor's auscultation recording of heart and lung sounds, or a voice recording of a patient describing symptoms such as coughing and wheezing; the multimodal content may also include various bioelectrical signal data, such as one-dimensional time-series signals such as electrocardiogram (ECG / EKG) waveform data and electroencephalogram (EEG) data; in addition, it may also include omics data such as gene sequencing data.

[0079] In practical implementation, those skilled in the art can add a corresponding dedicated encoder to the encoder cluster based on the characteristics of the new modality to be processed. For example, for audio information, an audio encoder based on a one-dimensional convolutional neural network (1D-CNN) or Wave2Vec architecture can be added; for electrocardiogram signals, a sequence encoder based on a recurrent neural network (RNN) or Long Short-Term Memory (LSTM) network can be added. The initial modal features extracted by this newly added encoder can be seamlessly integrated into the subsequent feature weighted fusion based on a multi-head cross-attention mechanism and the mapping process to the multimodal embedding space. Therefore, those skilled in the art should understand that any data modality that can have effective features extracted by a specific encoder can be included within the scope of the multimodal content described in this invention.

[0080] Based on the foregoing embodiments, in order to make the final generated report interpretation content more personalized and targeted, this invention further explains the generation of interpretation instruction text based on structured information digest in step 13 above.

[0081] Figure 3 This is the second flowchart illustrating the method for interpreting medical examination reports provided by this invention. Figure 4 This is the third flowchart of the method for interpreting physical examination reports provided by this invention, combined with... Figure 3 and Figure 4 As shown, First, user profile information is obtained. This user profile information is a multi-dimensional, structured collection of information about the user in the health checkup report, aiming to comprehensively depict the user's personal health background. Unlike traditional methods that rely solely on a single health checkup result, this invention introduces user profile information, enabling a more comprehensive and accurate interpretation of subsequent data.

[0082] In one specific embodiment, the user profile information may include, but is not limited to, the following types of information: Basic user information, such as gender, age, and occupation, is crucial for assessing risk profiles for certain diseases, such as occupational diseases or diseases prevalent in specific age groups.

[0083] The user's past medical history information, such as the name of the previously diagnosed disease (e.g., "type 2 diabetes"), the duration of the disease (e.g., "5 years"), and previous physical examination results, etc. This past medical history information is an important reference for assessing the user's current health status and disease progression trend.

[0084] Information about a user's lifestyle habits, such as whether they smoke, drink excessively, or have irregular sleep patterns, could potentially affect their health. For example, even if a user with a long history of alcohol consumption has normal liver function indicators this time, the interpretation should still highlight the potential risks.

[0085] In addition, user profile information can also include abnormal information from the user's current physical examination (abnormal indicators, abnormal conclusions, etc.), previous imaging information (ultrasound images, CT images, etc.), family genetic information, etc., which can be collected specifically according to the actual situation.

[0086] Then, the target prompt template is determined from the pre-built instruction configuration strategy. To adapt to the differentiated needs of different health checkup scenarios and user groups, this invention can pre-build an instruction configuration strategy. This instruction configuration strategy can be understood as a library storing multiple "prompt templates," each targeting a specific interpretation scenario or goal. The process of determining the target prompt template involves intelligently matching and selecting the most suitable template from this instruction configuration strategy based on the type of the current health checkup report or the user's user profile information.

[0087] For example, the target prompt template could be designed for a "pre-employment physical examination" scenario, focusing on assessing the user's basic physical condition and the presence of infectious diseases. Alternatively, the target prompt template could be designed for a "comprehensive screening for middle-aged and elderly people" scenario, with a greater emphasis on risk assessment for cardiovascular diseases, diabetes, and early cancer screening.

[0088] Suppose the physical examination report to be interpreted is a comprehensive health screening package designed for a 71-year-old male, the system will determine the target prompt template corresponding to the "comprehensive screening for middle-aged and elderly people" scenario from the instruction configuration strategy.

[0089] Finally, the user profile information and the structured information summary are integrated into the interpretation instruction text using the target prompt template. For example... Figure 4As shown, this integration process is a procedural filling and building process. The target prompt template typically contains a preset text frame and placeholders for filling dynamic information. The system will fill the corresponding placeholders in the target prompt template with the user profile information obtained in the previous step and the generated structured information summary, thereby generating the final, complete interpretation instruction text.

[0090] Taking the aforementioned "comprehensive screening for middle-aged and elderly people" scenario as an example, the target prompt template might be: "You are a senior internist. Please provide a personalized interpretation of the patient's health status based on the following user information and the structured information summary of the physical examination report. User information: [User profile information placeholder]; Structured information summary of the physical examination report: [Structured information summary placeholder]."

[0091] During actual integration, the system will populate the acquired user profile information (e.g., {Age: 71 years old, Gender: Male, Past Medical History: None}) and structured information summaries (e.g., {Hypertension: 169 mmHg, Hyperlipidemia: 8.0 mmol / L}) into the system. The final generated interpretation instruction text is: "You are a senior internist. Please provide a personalized interpretation of this patient's health status based on the following user information and the structured information summary of the physical examination report. User information: {Age: 71 years old, Gender: Male, Past Medical History: None}; Structured information summary of the physical examination report: {Hypertension: 169 mmHg, Hyperlipidemia: 8.0 mmol / L}. Due to the patient's advanced age, please pay special attention to the potential for coronary heart disease and provide suggestions."

[0092] Through the above steps, this invention transforms the process of generating interpretation instruction text from a simple listing of information into an organic combination of the user's personal background and the key findings of the physical examination. It also guides users through scenario-based target prompt templates, enabling the final report interpretation to achieve truly personalized interpretation. This greatly enhances the relevance, relevance, and practical value of the interpretation results, thereby providing users with more considerate and effective health management guidance.

[0093] Specifically, the personalized interpretation instruction text generated in this way can guide the report interpretation model to output report interpretation content with varying depths and breadths. That is, the report interpretation content must include at least one of the following: auxiliary information for disease diagnosis, health management suggestions, or disease risk inference results. Specifically: 1) Assisting in disease diagnosis: The guided report interpretation model identifies the changing trends of key indicators in the structured information summary, and combines them with the past medical history in the user profile information, matching them with known disease characteristics to generate report interpretation content containing disease diagnosis assistance information, thereby providing doctors with strong auxiliary decision support, which is crucial for the early detection of diseases and reducing the risk of misdiagnosis.

[0094] 2) Providing personalized health management solutions: The report interpretation model guides the generation of report interpretation content that includes health management suggestions. This means it not only proposes solutions to current health problems but also helps users establish long-term, effective self-management mechanisms based on user profile information such as age, occupation, and lifestyle habits. For example, for young office workers with a history of alcohol consumption, even if their liver function is normal in the current report, the interpretation content will focus on warnings of fatty liver risk and lifestyle intervention suggestions.

[0095] 3) Achieving Disease Risk Inference: The report interpretation model combines the user's historical health check data with the current findings to uncover potential risk factor association patterns, predicting future chronic disease risks at the individual level and generating report interpretation content that includes the disease risk inference results. For example, for a user whose blood pressure and blood sugar are both at borderline levels, the report interpretation model can infer the probability of developing metabolic syndrome within the next 5 years and provide detailed prevention guidelines.

[0096] To further improve the accuracy and automation of target prompt template selection, and to ensure that the generated interpretation instruction text better matches the original intent of the health check service and the specific needs of users, this invention further provides a specific implementation method for determining the target prompt template from a pre-built instruction configuration strategy, which may include the following steps: First, determine the user's health checkup package type. Here, "health checkup package type" refers to the category of the health checkup service the user is currently participating in. Different health checkup packages typically have different examination items, focuses, and target groups; therefore, accurately identifying the health checkup package type is a prerequisite for differentiated and precise interpretation. In practical applications, health checkup package type information can be determined through various methods. For example, it can be extracted directly from the title or first page information of the health checkup report to be interpreted (e.g., the report title is "AA Company 2025 Pre-employment Health Checkup Report"). Alternatively, when the user uploads or the system receives the health checkup report, the user can be required to manually select the corresponding package type, or the system can automatically associate it based on the source information, such as "routine health checkup," "high-end cancer screening," or "pre-pregnancy checkup."

[0097] Then, the target prompt template is mapped from the instruction configuration strategy based on the physical examination package type. After determining the user's physical examination package type, the system uses this as an index to search and match within the pre-built instruction configuration strategy to map the most suitable target prompt template for the current scenario. This "mapping" can be understood as a key-value lookup process, where the physical examination package type is the key, and the corresponding target prompt template is the value. This mechanism ensures that different types of physical examination reports trigger different interpretation logics and focuses.

[0098] For example, the instruction configuration strategy can pre-store the following mapping relationship: Medical examination package type: "Pre-employment medical examination" corresponding to target prompt template A; Health checkup package type: "Comprehensive health screening for middle-aged and elderly people" corresponding target prompt template B; Medical examination package type: "Occupational disease-specific medical examination" corresponds to target prompt template C.

[0099] Thus, when the system determines that the current user's medical examination package type is "pre-employment medical examination," it will automatically call target prompt template A. Target prompt template A is designed with a special focus on assessing the user's basic physical functions (such as cardiopulmonary function, vision, and hearing) and risks of common infectious diseases. The interpretation instruction text generated by it will guide the report interpretation model to focus on these aspects and may include general suggestions on occupational health to meet the core concerns of employers.

[0100] When the system determines that a user's health check package type is "Comprehensive Health Screening for Middle-aged and Elderly," it will map to target prompt template B. Since the core objective of this type of health check package is the early detection of age-related chronic diseases, target prompt template B will contain more guiding instructions regarding cardiovascular disease, diabetes, tumor markers, and so on. The generated interpretation instruction text will prompt the report interpretation model not only to interpret the current abnormal indicators but also to conduct in-depth risk assessments based on user profile information such as age and gender, and to provide more detailed and easily understandable self-care suggestions for elderly users.

[0101] This invention, by implementing a differentiated instruction configuration strategy for different types of health checkup packages, ensures that each user receives professional guidance and service experience highly matched to their specific package type and actual needs. This flexible instruction configuration method not only significantly improves the relevance and effectiveness of health checkup interpretation services but also greatly optimizes the user experience, thereby helping to raise public awareness of the importance of regular health checkups and laying a solid foundation for achieving the goal of nationwide health management.

[0102] As a specific embodiment of the method for interpreting physical examination reports, in order to improve the depth and accuracy of the multimodal parsing model in extracting features from different modal data, this invention optimizes the specific selection of the text encoder, image encoder and video encoder that constitute the model.

[0103] Specifically, the text encoder can be a network model based on the Transformer architecture. The Transformer architecture is chosen because its powerful self-attention mechanism can effectively capture long-distance dependencies in text, which is crucial for understanding medical texts with strong contextual relationships.

[0104] An image encoder can be a Vision Transformer (ViT) model, specifically designed for processing static, two-dimensional image content, such as vascular stiffness maps or X-ray images. The ViT model operates in two-dimensional space, recognizing spatial features such as shape, texture, and color on a plane, much like processing a snapshot. It segments the input image content into a series of fixed-size patches and feeds them as a sequence into a Transformer structure for processing. The core advantage of this approach lies in its self-attention mechanism, allowing the ViT model to automatically learn the importance of different regions in the image. For example, it can automatically assign higher attention weights to patches containing lesion information, thereby achieving accurate capture of key visual features.

[0105] An image encoder can be a three-dimensional convolutional neural network (3D-CNN). 3D-CNN models can process sequences or volumes of data consisting of multiple consecutive frames, i.e., image content, such as a 20-frame dynamic ultrasound image or a CT scan result composed of 100 slices. Unlike two-dimensional models that process static images, 3D-CNN uses three-dimensional convolutional kernels (e.g., 3*3*3), which, while sliding through the data, can analyze not only the spatial relationships within a single frame but also simultaneously capture the temporal or depth relationships between adjacent frames. Because it analyzes spatio-temporal features, much like analyzing a short film or a 3D model, it can understand motion trajectories or three-dimensional shapes. Therefore, using 3D-CNN as an image encoder can most effectively extract core spatio-temporal or three-dimensional structural features from image content.

[0106] It's important to note that the third dimension in 3D-CNN varies depending on the nature of the image content. For dynamic images (such as dynamic echocardiograms), the third dimension is time, allowing 3D-CNN to capture the dynamic changes in the opening and closing of heart valves over time. For computed tomography (CT or MRI), the third dimension is depth (slice sequence), enabling 3D-CNN to understand the three-dimensional structure of organs or lesions, rather than treating each slice as an isolated image. Therefore, using 3D-CNN as an image encoder can most effectively extract core spatiotemporal features or three-dimensional structural features from image content.

[0107] By selecting a Transformer-based network model, a visual Transformer model, and a 3D convolutional neural network as dedicated encoders for text content, image content, and video content respectively, this invention can process the data using the most advanced and suitable models in the current field, tailored to the characteristics of each modality. This specialized model selection ensures that the features extracted from various data sources are of the highest quality and richest in information, thus providing a solid technical guarantee for subsequent information fusion in the multimodal embedding space and the generation of high-quality structured information summaries.

[0108] As an optional embodiment, in order to further optimize the input quality of the text encoder and improve its ability to understand text content, this embodiment of the invention further includes a preferred step of preprocessing the text content before inputting it into the multimodal parsing model.

[0109] Specifically, before inputting the text content, image content, and video content into the multimodal parsing model, the method further includes: performing word segmentation on the text content using Byte Pair Encoding (BPE).

[0110] The introduction of BPE word segmentation is primarily aimed at addressing several key technical issues in medical texts such as medical examination reports: On the one hand, it's to effectively handle out-of-vocabulary (OOV) words. The medical field is full of specialized, complex, and constantly emerging new terms (such as specific drug names, disease names, gene loci, etc.). Traditional word-based segmentation methods cannot understand the meaning of words they haven't seen during training, severely impacting the accuracy of interpretation. BPE, however, can break down any unseen word into known, smaller sub-word units. For example, the rare medical term "pheochromocytoma," even if the text encoder hasn't processed the complete term, can be broken down by BPE into ['pheo', 'chromium', 'cell', 'tumor'] or similar sub-word combinations to understand its composition and meaning, thus greatly enhancing its generalization ability for specialized terms and new words.

[0111] On the other hand, it is to balance the vocabulary size and information granularity. The word segmentation method with words as the basic unit has no OOV problem, but it will lose the semantic information at the word level (for example, separating "blood" and "pressure" is less meaningful than the whole "blood pressure"); if words are used as the basic unit, the vocabulary will be extremely large, increasing the computational burden and still unable to solve the OOV problem. The present invention achieves a balance between the two through BPE, which can retain the integrity of high-frequency words (such as "hypertension"), while splitting low-frequency or rare words into sub-words, maximizing the retention of semantic information in the text content while controlling the vocabulary size.

[0112] On yet another hand, it is to capture morphological information. Many medical terms have common roots or suffixes, and these morphological components themselves carry specific meanings (such as "...itis" indicating inflammation). Through BPE processing, these meaningful morphological components can be naturally recognized as independent sub-word units, enabling the text encoder to learn the meanings of these morphemes and draw inferences. For example, after the text encoder has learned "pneumonia" and "hepatitis", when it encounters "nephritis", even if it has not seen it before, it can infer that this is an inflammation related to the kidneys.

[0113] In a specific embodiment, the process of using BPE to process the text content until it is fed into the text encoder may include, but is not limited to, the following steps: First, input a segment of original text content, for example, a sentence in a physical examination report: "The patient has prehypertensive symptoms". A vocabulary pre-trained by the BPE algorithm can be used to segment this text content, resulting in a sub-word sequence composed of sub-words (Tokens). For example, the above text content may be segmented into: ['patient', 'has', 'hypertension', 'pre-', 'symptoms'].

[0114] Then, input the sub-word sequence generated in the previous step, and look up the unique digital index corresponding to each sub-word in the BPE vocabulary, and convert this sub-word sequence into an integer sequence.

[0115] For example, ['patient', 'has', 'hypertension', 'pre-','symptoms'] can be converted to [101, 872, 5321, 2345, 4598], achieving the conversion of the text into a digital format that the text encoder can directly process.

[0116] Because text encoders (typically Transformer architectures) require each batch of input data to have the exact same length (denoted as hyperparameter L, e.g., L=512) in order to enable efficient parallel computation, the integer sequence generated in the previous step needs to be length-aligned.

[0117] If the length of the integer sequence is less than L, it is padded with the ID corresponding to a special "padding character" (e.g., 0) until the sequence length reaches L. If the sequence length is greater than L, it is truncated from the end, and only the IDs of the first L sub-words are retained.

[0118] Finally, the integer sequence of fixed length L, processed through all the above steps, is fed into the text encoder as the final input. The text encoder uses its internal self-attention mechanism and other complex calculations to transform this integer sequence into a feature vector containing rich semantic information. This feature vector is the mathematical representation of the text content in the multimodal parsing model, which can be used for subsequent fusion, analysis, and interpretation.

[0119] This invention significantly improves the ability of the text encoder to process complex medical texts by adding a BPE segmentation processing step before inputting the multimodal parsing model. This ensures that the model can perform effective semantic understanding even when faced with obscure and professional terms, thereby improving the accuracy and reliability of the subsequent generation of structured information summaries.

[0120] To further clarify how the visual ViT model efficiently and accurately extracts key features from image content, this invention provides a more detailed explanation of its internal working mechanism, mainly including but not limited to the following steps: First, for each image in the image content, it is divided into multiple image blocks.

[0121] When the ViT model receives an input image (such as a chest X-ray), it doesn't perform a sliding convolution operation directly on the entire image like a traditional convolutional neural network. Instead, it preprocesses the two-dimensional image, dividing it into a sequence of multiple fixed-size, non-overlapping square patches. For example, a 224x224 pixel image can be divided into 196 16x16 pixel patches. This approach cleverly transforms the computer vision problem into a sequence processing problem similar to that in natural language processing, allowing direct application of the powerful capabilities of the Transformer architecture. Each patch is then linearly embedded into a vector, with positional encoding to preserve its spatial location information within the original image.

[0122] Then, a self-attention mechanism is used to calculate the association weights between multiple image patches, and the feature representations of multiple image patches are aggregated based on the association weights to generate an image feature vector representing each image.

[0123] The sequence of image patches containing location information is fed into the standard Transformer encoder of the ViT model. Inside this encoder, for each image patch in the sequence, a self-attention mechanism calculates its association weight with all other image patches in the sequence (including itself). This association weight represents the degree of interdependence or importance between different image patches. For example, for an X-ray containing a lung nodule, the self-attention mechanism learns that the image patch containing the nodule has a stronger association with other image patches that also depict lung tissue but are normal, and a weaker association with image patches depicting ribs or image edges. In other words, the ViT model automatically assigns higher attention weights to image patches containing key diagnostic information.

[0124] After calculating these association weights, the ViT model aggregates the feature representations of the multiple image patches based on these weights. Specifically, the feature representation of each image patch is a weighted sum of the original feature representations of all image patches, and this weight is the calculated association weight. After stacking and calculating through multiple layers of self-attention modules, the ViT model finally outputs a feature vector that aggregates the global information of the entire image. This feature vector is the high-dimensional, structured image feature vector representing the image, denoted as [image feature vector]. This image feature vector encapsulates the core content of the image and can be used for subsequent fusion operations in the multimodal embedding space.

[0125] By segmenting images into patches and processing them using a self-attention mechanism, the ViT model can overcome the limitations of traditional CNN models in terms of receptive field. This allows it to more effectively capture long-range dependencies and global contextual information in image content, enabling the model to intelligently focus on key regions in the image and ignore irrelevant backgrounds. This results in the extraction of more accurate and robust image feature vectors, laying a solid foundation for improving the performance of the entire multimodal parsing model.

[0126] To better illustrate the overall workflow of the multimodal analytical model in this invention, a more detailed comprehensive embodiment is provided below.

[0127] In this embodiment, the multimodal parsing model aims to collaboratively process data from three modalities: images, videos, and text. First, the input data is mathematically defined: let a batch of images be denoted as... I ={ , ,... }, each of which ∈ The image collection is V ={ , ,..., }, where each image contains Z frames, i.e. ∈ , H Represents the height of the image (in pixels). W 3 represents the width of the image (in pixels), and 3 represents the number of color channels in the image (e.g., RGB three channels). Z This refers to the number of frames or slices in the image. The set of text corresponding to the image is denoted as... ={ , ,..., The set of texts corresponding to the images is denoted as}. ={ , ,..., Each text describes one image.

[0128] The multimodal analytical model processes this data through three parallel encoder branches: All text content (including) and First, the text is segmented by BPE, converted into an integer sequence, and then length-aligned (fixed to L). The result is then input into a text encoder to generate a text feature vector.

[0129] Image content collection I The images are fed into the ViT model, which acts as an image encoder. Through image patch segmentation and a self-attention mechanism, features are extracted from each image, ultimately generating a set of image feature vectors. : ; in, I This refers to the data input to the image encoder, specifically a collection or batch of raw images. N Zhang picture; For image encoder input I Perform feature extraction calculations; This represents the dimension of the image feature vector.

[0130] Image content collection V The data is fed into a 3D convolutional neural network that acts as an image encoder, where 3D convolutional kernels are used to process spatiotemporal or 3D structural information, ultimately generating a set of image feature vectors. : ; in, M This represents the number of images contained in a processing batch. For image encoders, This represents the complete computational process of feature extraction from input V; The dimension of the image feature vector; Image features are the structured and numerical representations of core information obtained after the original image data has undergone deep processing by a 3D-CNN encoder.

[0131] Based on this, the present invention provides another specific implementation method for inputting the multimodal content into a multimodal parsing model to obtain a structured information summary output by the multimodal parsing model, which mainly includes, but is not limited to, the following steps: The text content, the image content, and the video content are input into the encoder cluster of the multimodal parsing model to obtain the initial modal features of the text, the initial modal features of the image, and the initial modal features of the video output by the multimodal parsing model. Determine the first attention weight between image features and text features and image features, the second attention weight between image features and text features and image features, and the third attention weight between text features and image features and image features. Based on the first attention weight, the second attention weight, and the third attention weight, the initial text modal features, the initial image modal features, and the initial video modal features are weighted and fused to obtain weighted text modal features, weighted image modal features, and weighted video modal features; The text-weighted modal features, image-weighted modal features, and video-weighted modal features are mapped to the multimodal embedding space and then fused to obtain the structured information summary.

[0132] Based on the aforementioned embodiments, and especially after determining the specific selection of the encoder cluster in the multimodal analytical model, in order to solve the problem of heterogeneity of different modal data in the spatiotemporal dimension and achieve real-time alignment and complementarity of cross-modal information, thereby significantly improving the accuracy of multimodal data fusion, this embodiment of the invention further provides a specific implementation method for obtaining structured information summaries.

[0133] First, the text content, the image content, and the video content are input into the encoder cluster of the multimodal parsing model to obtain the initial modal features of the text, the initial modal features of the image, and the initial modal features of the video output by the multimodal parsing model.

[0134] This step is the initial stage of feature extraction. The system feeds the preprocessed text content, image content, and video content into the corresponding encoders in the encoder cluster. Specifically, the text content is input to the text encoder (e.g., a network model based on the Transformer architecture) to extract its semantic features; the image content is input to the image encoder (e.g., a visual Transformer model) to extract its spatial visual features; and the video content is input to the image encoder (e.g., a 3D convolutional neural network) to extract its spatiotemporal or 3D structural features. After this step, three independent sets of initial feature vectors representing the original information of their respective modalities are obtained: initial text modal features, initial image modal features, and initial video modal features.

[0135] Regarding the initial modal features of the image and image initial modal features The calculation process has been provided in the foregoing embodiments and will not be repeated here. Regarding the initial modal features of the text, they mainly consist of two parts: one part is the text features corresponding to the image content. The other part consists of text features corresponding to the image content, and the corresponding calculation formula is: ; in, For text encoders, Image features; Image features; It is the dimension of the text feature vector. Image text features These are image text features.

[0136] Secondly, determine the first attention weight between image features and text features and image features, the second attention weight between image features and text features and image features, and the third attention weight between text features and image features and image features.

[0137] This step is the core of achieving cross-modal information interaction. It aims to enable each modality's features to perceive and quantify the importance of other modality features to itself through an attention mechanism. The system introduces a multi-head cross-attention mechanism to calculate the correlation weights between features of different modalities. Specifically, the system performs multiple cross-attention calculations: To calculate the first attention weight, the system uses the initial modal features of the image. As a query, the initial modal features of the image and initial modal features of text The combination of these elements is used as the key and value to calculate the attention weights between image features and video / text features: [ ]=MultiHeadCrossAttn( , [ ]).

[0138] To calculate the second attention weights, the system uses the initial modal features of the image. As a query, the initial modal features of the image and initial modal features of text The combination of these elements is used as the key and value to calculate the attention weights between image features and image / text features: [ ]=MultiHeadCrossAttn( [ , ]).

[0139] To calculate the third attention weight, the system uses the initial modal features of the text respectively. and As a query, the attention weights of text features and image / video features are calculated. For example: [ ]=MultiHeadCrossAttn( ,[ , ]); [ ]=MultiHeadCrossAttn( ,[ ]).

[0140] Among them, attention weight (For example ) is a weight matrix used to measure the degree of attention or correlation between each feature element in mode x and all feature elements in mode y.

[0141] Next, the initial text modal features, the initial image modal features, and the initial video modal features are weighted and fused based on the first attention weight, the second attention weight, and the third attention weight, respectively, to obtain weighted text modal features, weighted image modal features, and weighted video modal features.

[0142] This step involves dynamically and information-enhancingly weighting and fusing the initial modal features based on the attention weights calculated in the previous step, generating new features that incorporate multimodal contextual information. This fusion process allows each modal feature to selectively absorb key information from other modalities, thereby improving its own feature representation capabilities.

[0143] The specific fusion process can be achieved by weighted summation combined with residual connection.

[0144] Image weighted modal features (fused image features) The calculation method is as follows: the initial modal features of the image and initial modal features of text Multiply each by its corresponding attention weight (after being normalized by the Softmax function), and then combine with the original image's initial modal features. The specific formula for addition is as follows: .

[0145] Image weighted modal features (features of the fused image) Text-weighted modal features (fused text features) , The calculation also uses a similar weighted fusion method, and the specific calculation formula is expressed as follows: ; ; .

[0146] The advantages of this fusion process are illustrated below using an example of interpreting a medical examination report about lung nodules: Assume the input image content is a sequence of chest CT images (its initial features are...). The image revealed that the nodule in the right middle lobe of the lung exhibited typical imaging features of a malignant tumor, including lobulation and spiculation. The input image was a chest DR image (its initial features were...). This shows the approximate location of the nodule. Additionally, the text content corresponding to the CT image sequence (whose initial features are...) The document contains the description "consider the possibility of malignancy".

[0147] During feature fusion, since malignant features in CT image sequences are highly correlated with nodule locations in DR images and the description of "malignant" in text content, attention weights are applied to image-to-image and image-to-text fusion. and It will be very high. This makes it difficult to calculate image weighted modal features. At the same time, it can accurately and strongly correlate the detailed features of CT image sequences (such as spiculations) with the localization information of DR images and the diagnostic tendency descriptions in the text, generating... It is no longer isolated image information, but a feature expression that integrates multi-dimensional evidence and is richer in information.

[0148] Finally, the text-weighted modal features, the image-weighted modal features, and the video-weighted modal features are mapped to the multimodal embedding space and then fused to obtain the structured information summary.

[0149] This step involves the final alignment and integration of the weighted modal features enhanced with cross-modal information. The system will utilize their respective learnable projection matrices (e.g., Wᵢ , Wᵥ and W t ), weighted modal features of images Image weighted modal features Text-weighted modal features , Each modality is projected into a unified, shared multimodal embedding space. In this multimodal embedding space, the features of all modalities have the same dimensionality and comparability.

[0150] After projection, L2 normalization is usually performed to eliminate scale differences between modalities, resulting in the final embedding representation, such as an image embedding representation. Image embedding representation These tightly aligned embedding representations in the multimodal embedding space form the basis of feature fusion. The feature fusion unit can concatenate these aligned embedding representations, perform weighted summation again, or process them through a more complex fusion network to ultimately generate a highly condensed, machine-readable structured information summary for use by downstream report interpretation models.

[0151] This invention effectively solves the fusion challenge caused by heterogeneous multimodal data by introducing a cross-modal attention mechanism and a dynamic feature fusion process. This method enables the model to achieve deep interaction and complementarity of different modal information at the feature level, significantly enhancing the representational ability of each modal feature. This results in a more accurate and comprehensive structured information summary, laying a solid technical foundation for high-quality interpretation of medical examination reports.

[0152] In order to enable the multimodal parsing model to perform its function effectively and accurately, namely, to generate high-quality structured information summaries from multimodal inputs, this invention further provides a method for training it.

[0153] Figure 5 This is a schematic diagram of the training process of the multimodal analytical model provided by the present invention, as shown below. Figure 5 As shown, this training method aims to enable text encoders, image encoders, and video encoders to learn to map semantically related information from different modalities to similar locations in a multimodal embedding space through joint training. Specifically, it may include the following steps: Step 21: Construct a training dataset. The training data in the training dataset includes at least image-text pairs consisting of image samples and their corresponding text samples, and image-text pairs consisting of video samples and their corresponding text samples.

[0154] A high-quality training dataset is fundamental to model performance. In this embodiment, the constructed training dataset includes two core data pairs: image-text pairs and image-text pairs. Image-text pairs consist of an image sample and a text sample describing its content. For example, a chest X-ray image can serve as the image sample, while the corresponding radiologist's report, "A round, high-density shadow is visible in the right lower lung field, with clear edges; a follow-up CT scan is recommended," serves as the text sample. Image-text pairs consist of an image sample and a text sample describing its content. For example, a video recording of a cardiac ultrasound can serve as the image sample, while the corresponding ultrasound physician's diagnosis, "Left ventricular ejection fraction reduced to 45%, ventricular wall motion incoordination," serves as the text sample. By collecting and constructing these data pairs on a large scale, a foundation is laid for the subsequent comparative learning training of the multimodal analytical model.

[0155] Step 22: Use the training data in the training dataset as the input to the multimodal parsing model, and calculate the contrastive loss based on the output of the multimodal parsing model.

[0156] Specifically, the system inputs data from the training dataset in batches into the initial multimodal parsing model. The text encoder, image encoder, and video encoder in the multimodal parsing model process the inputs respectively to obtain the initial feature vectors.

[0157] Before calculating the contrastive loss, the features output by different encoders can be aligned. Specifically, the visual features extracted by the image encoder and video encoder, and the text features extracted by the text encoder, are aligned using a learnable projection matrix (denoted as ) in the feature fusion unit. Wᵢ , Wᵥ and W t The embeddings are mapped to a unified multimodal embedding space and L2 normalized to obtain normalized embedding representations, such as image embedding representations. Image embedding representation and the corresponding image text embedding representation and image text embedding representation The modal embeddings are represented as follows: =L2-Normalize · )∈ ; =L2-Normalize · )∈ ; =L2-Normalize · )∈ ; =L2-Normalize · )∈ .

[0158] in, , ,and These are the learnable projection matrices used for image features, video features, and text features, respectively. is the dimension of the multimodal embedding space; the L2-Normalize(·) function represents performing L2 normalization on the input embedding vector so that the output embedding vector has a unit length.

[0159] Subsequently, the contrastive loss is calculated based on these embedding representations located in the multimodal embedding space. The contrastive loss is calculated as follows: (1) Calculate the similarity matrix between image-text pairs and video-text pairs respectively. For example, the similarity matrix between image and text pairs. It is done by calculating the embeddings of all images within a batch. and all text embeddings The similarity matrix is ​​obtained by the dot product between the images and text. Similarly, the similarity matrix between the images and text can be calculated. The specific expression is: ; .

[0160] in, It is a learnable temperature parameter used to scale the similarity score and adjust the sharpness of the Softmax function, thereby controlling the degree to which the model pays attention to hard examples during training. This indicates transpose.

[0161] (2) Calculate the loss based on the similarity matrix. Taking image-text pairs as an example, the corresponding image-text comparison loss is... It consists of two-way loss: one is the image-to-text loss. One approach is to maximize the similarity of each image to the correct text description while minimizing its similarity to other irrelevant texts within the batch; the other is the text-to-image loss. Conversely, the loss in both directions can be calculated using the cross-entropy loss function. Similarly, the image-text contrast loss is calculated using the image-text pairing loss function. It is also a loss from image to text. and text-to-image loss The specific expression, determined comprehensively, is as follows: ; .

[0162] (3) Finally, the total contrast loss is calculated based on the determined image-to-text loss and image-to-text loss. The specific expression is: .

[0163] By minimizing the total loss This drives the optimization of dynamic weights and feature representations in multimodal parsing models, enabling better alignment of multimodal features in the embedding space and improving the performance of the parsing model. This is crucial for subsequent tasks such as disease diagnosis, health management suggestion generation, and disease prediction.

[0164] For example, during model training, for a chest CT image sequence (containing malignant nodule features) and its corresponding text content ("right middle lobe nodule, possibly malignant"), the loss function will narrow the distance between them in the embedding space. For chest CT image sequences and mismatched text content (such as "no obvious abnormalities seen in both lungs"), the distance will be increased. Simultaneously, chest DR images and their corresponding text content ("small nodule in the right middle lobe, irregular shape") will also be narrowed. Alignment is achieved. Through training, the multimodal parsing model can accurately associate the matching relationships between "CT malignant features → CT report text" and "DR nodules → DR report text".

[0165] Step 23: Based on the contrast loss, jointly update the trainable parameters of the text encoder, the image encoder, the video encoder, and the feature fusion unit.

[0166] After calculating the total contrast loss Next, the gradient of the contrastive loss with respect to all trainable parameters in the multimodal analytical model is calculated using the backpropagation algorithm. These trainable parameters include all weights and biases that make up the text encoder, image encoder, and video encoder, as well as the matrix parameters used for projection in the feature fusion unit.

[0167] Finally, optimization algorithms such as gradient descent (e.g., Adam) can be used to jointly update all these trainable parameters based on the contrastive loss and its gradient.

[0168] Joint update means that the various parts of the entire multimodal parsing model are optimized as a whole, rather than trained independently. By iterating this process (i.e., iteratively executing steps 22 to 23), the three encoders in the multimodal parsing model are learned to cooperate in feature extraction.

[0169] This invention employs the aforementioned contrastive learning-based training method, enabling the multimodal parsing model to learn a high-quality and semantically consistent multimodal embedding space. In this space, vector representations of content-related data from different modalities (such as a CT image of lung cancer and text describing a malignant lung tumor) naturally cluster together. This cross-modal semantic alignment capability allows this invention to accurately extract information from complex medical examination reports and generate structured information summaries.

[0170] Figure 6 This is a schematic diagram of the contrastive loss calculation process during the training of the multimodal analytical model provided by this invention, as shown below. Figure 6 As shown, the calculation process can specifically include the following steps: Step 201: Obtain the image features generated by the image encoder processing the image samples in the image-text data pair, and obtain the image features generated by the image encoder processing the image samples in the image-text pair.

[0171] Within a training batch, the encoders of the multimodal parsing model process the input data in parallel. Specifically, this step refers to the system processing all image samples (e.g., within a batch) of the image-text pair. N An X-ray image is input into an image encoder (e.g., a visual Transformer model). After forward propagation, the encoder outputs initial image features representing the content of these images. Simultaneously, the system will include all image samples in the image-text pair (e.g., within a batch). M The dynamic ultrasound images are input into an image encoder (e.g., a 3D convolutional neural network) to obtain initial image features representing the content of these images from its output. .

[0172] Step 202: Obtain the image text features and image text features generated by the text encoder after processing the text samples in the image text data pair and the text samples in the image text pair, respectively.

[0173] Step 202 runs in parallel with step 201, where the text encoder processes text samples from both sources. Specifically, the system processes all text samples from the image-text pairs (i.e., those from the aforementioned sources) N Each image corresponds to one of the images.N The text description is input into the text encoder to obtain the image text features generated by it that represent the semantics of these texts. At the same time, all text samples in the image-text pair (i.e., those mentioned above) M One-to-one correspondence of segment images M The segmented text descriptions are also input into the same text encoder to obtain the image text features generated by it that represent the semantics of these texts. .

[0174] Step 203: Calculate the image-text contrast loss between the image features and the image-text features, and calculate the image-text contrast loss between the image features and the image-text features.

[0175] After obtaining the initial features for all modalities, independent contrastive loss calculations will be performed: First, the system focuses on image-text pairing. The initial image features obtained in step 201 are then used... and the image text features obtained in step 202 The image-text contrast loss is then calculated by mapping the images to a unified embedding space and normalizing them, as described in the previous embodiments. Image-text contrast loss The goal is to bring the correctly paired image and text embeddings closer together within a batch, while pushing away the embeddings of unrelated images and text.

[0176] Secondly, in exactly the same way, the system focuses on image-text pairing. The initial image features acquired in step 201 are then used... and the image text features obtained in step 202 The images are processed, and then the image-text contrast loss between them is calculated. .

[0177] Step 204: The sum of the image-text contrast loss and the video-text contrast loss is taken as the contrast loss.

[0178] The image-text contrast loss was calculated separately for each image-text modality pair. Image-text contrast loss for image-text modal pairs Then, these two losses are accumulated, for example, by summing them directly, to obtain the final contrastive loss for the entire training batch. This overall comparative loss This will serve as the basis for subsequent backpropagation and model parameter updates.

[0179] This invention achieves total contrast loss By decomposing the loss into two independent parts—image-text contrast loss and image-text contrast loss—and then adding them together, we achieve decoupling and parallel optimization of the two cross-modal learning tasks: image-text and image-text. This structured loss calculation method enables the model to clearly and explicitly learn the correspondence between the two different visual modalities (static images and dynamic images) and the language modal during training, which helps to improve the overall convergence speed and final performance of the multimodal parsing model.

[0180] Figure 7 This is a schematic diagram of the calculation process for image-text contrast loss provided by the present invention. The following is in conjunction with... Figure 7 The following details how this invention calculates the image-text contrast loss from the similarity matrix. The calculation steps include, but are not limited to: Step 301: Calculate the image-text similarity matrix based on the image features and the image-text features.

[0181] This step aims to quantify the semantic relevance between each image and each piece of text within a training batch. As described in the previous embodiment, the system first processes the initial image features... Image and text features Image embedding representations are obtained by mapping each image to a unified multimodal embedding space using its respective projection matrix and then normalizing the representation. and text embedding representation Then, by calculating the dot product of these two embedding matrices and using the temperature parameter... Scaling is performed to obtain N * N Image-text similarity matrix In the matrix In the middle, located in ( i , j The element value at position () represents the element at position () in the batch. i The picture and the first j Similarity scores between text segments. Ideally, when i = j When the image and text are correctly paired, the score for that position should be significantly higher than [previous score]. i ≠ j The score at that time.

[0182] Step 302: Calculate the first direction loss and the second direction loss based on the image-text similarity matrix.

[0183] After obtaining the image-text similarity matrix, the contrastive loss is calculated from two symmetrical directions: For each image in the batch (i.e., each row of the matrix), the goal is to maximize its similarity score with the correctly paired text. Specifically, each row of the image-text similarity matrix can be viewed as the output of a classification problem, where... N Categories (i.e.) N (Segment text), and there is only one correct category (i.e., the one on the diagonal). The Softmax function can be applied to each line to transform it into a probability distribution, and then the cross-entropy loss between the distribution and the true label (a one-hot vector with a 1 at the correct text index) can be calculated. All N The loss calculated from each image is averaged to obtain the loss in the first direction. .

[0184] Completely symmetric to the first-direction loss, for each text segment in the batch (i.e., each column of the matrix), the goal is to maximize its similarity score with the correctly paired image. Similarly, the Softmax function can be applied to each column of the matrix, and the cross-entropy loss can be computed. All N The loss calculated from each segment of text is averaged to obtain the second-direction loss. .

[0185] Step 303: Combine the first direction loss with the second direction loss to obtain the image-text contrast loss.

[0186] After calculating the loss in the first direction respectively Second direction loss Then, the two losses are combined to obtain the final image-text contrast loss. The combination method is to directly sum or average.

[0187] Figure 8 This is a schematic diagram of the calculation process for image-text contrast loss provided by the present invention. The following is in conjunction with... Figure 8 As shown, this invention calculates the image-text contrast loss from the similarity matrix in detail. The calculation steps include, but are not limited to: Step 401: Calculate the image-text similarity matrix based on the image features and the image-text features.

[0188] This step aims to quantify the semantic relevance between each image and each piece of text within a training batch. Similar to the image-text processing method, the system first processes the initial image features... and image text features The image embedding representation is obtained by mapping each element to a unified multimodal embedding space using its respective projection matrix and then normalizing it. and text embedding representation Then, by calculating the dot product of these two embedding matrices and using the temperature parameter... Scaling is performed to obtain M * M Image-text similarity matrix In the matrix In the middle, located in ( i , j The element value at position () represents the element at position () in the batch. i Segment of video and the first j Similarity scores between paragraphs.

[0189] Step 402: Calculate the third-direction loss and the fourth-direction loss based on the image-text similarity matrix.

[0190] After obtaining the image-text similarity matrix, the contrast loss is also calculated from two symmetrical directions: Third-direction loss The computation proceeds from image to text. For each image in the batch (i.e., each row of the matrix), the goal is to maximize its similarity to the correctly paired text description. This is achieved by applying the Softmax function to each row of the matrix and calculating the cross-entropy loss with the true label, then combining all the images... M The loss calculated from the segment images is averaged to obtain the result. .

[0191] For each text segment in the batch (i.e., each column of the matrix), the goal is to maximize its similarity to the correctly paired image. This is achieved by applying the Softmax function to each column of the matrix and calculating the cross-entropy loss, then combining all the... M The fourth-direction loss can be obtained by averaging the losses calculated from the text segments. .

[0192] Step 403: Combine the third-direction loss with the fourth-direction loss to obtain the image-text contrast loss.

[0193] After calculating the third-party losses respectively and fourth-direction loss Then, the two losses are combined to obtain the final image-text contrast loss. .

[0194] This invention employs a parallel and symmetrical process to calculate the image-text contrast loss, ensuring that the multimodal parsing model can learn the correspondence between the two different visual modalities, static images and dynamic images, and the linguistic modalities in a consistent manner.

[0195] This invention provides a method based on contrast loss The implementation method for jointly updating the trainable parameters of the text encoder, image encoder, video encoder, and feature fusion unit in a multimodal parsing model mainly includes the following steps: First, a backpropagation algorithm is used to calculate the gradient of the contrast loss with respect to the trainable parameters of the text encoder, image encoder, video encoder, and feature fusion unit.

[0196] The total contrastive loss for a training batch is calculated using the method described in the foregoing embodiments. Next, the training process enters the parameter update phase. The backpropagation algorithm is a method used to calculate the gradient of the loss function with respect to each parameter in a neural network. The system will use the total contrastive loss... Starting from the first step, propagate backward along the computational graph of the multimodal analytical model. The loss is efficiently calculated using the chain rule. The partial derivative (i.e., gradient) of each trainable parameter in the multimodal analytical model.

[0197] These trainable parameters mainly include the weights and biases of all network layers (such as convolutional layers, fully connected layers, self-attention modules, etc.) that make up the text encoder, image encoder, and video encoder, as well as the projection matrix (such as the projection matrix of the feature fusion unit) that makes up the feature fusion unit. , , ) and temperature parameters The calculated gradients indicate the direction in which each parameter should be adjusted to improve the contrastive loss. It decreases the fastest.

[0198] Then, an optimizer is used to adjust the trainable parameters based on the gradients. After obtaining the gradients of all trainable parameters, the system uses an optimizer to perform the actual parameter updates. The optimizer is a program module that implements a specific optimization algorithm; common examples include stochastic gradient descent (SGD), Adam, and RMSprop. The optimizer adjusts the value of each trainable parameter based on the gradients and preset hyperparameters such as the learning rate.

[0199] Finally, the steps of calculating the contrastive loss and adjusting the trainable parameters according to the gradient are performed iteratively and repeatedly for different batches of training data in the training dataset until a preset convergence condition is met.

[0200] The system continuously extracts new training data from the training dataset in different batches and iteratively repeats the complete process described above, from "calculating the contrastive loss" to "adjusting parameters based on the gradient." With each iteration, the performance of the multimodal parsing model improves slightly. This iterative process continues until a preset convergence condition is met. This preset convergence condition may be: the total number of training epochs reaches a preset upper limit; or the performance metrics (such as loss value or accuracy) of the trained multimodal parsing model on the validation set no longer show significant improvement over multiple consecutive epochs; or the total training time reaches a preset duration.

[0201] By employing the iterative optimization method based on gradient descent and backpropagation described above, this invention can stably and effectively train a complex multimodal parsing model containing multiple encoder branches. This standardized training process ensures that the model can fully learn cross-modal semantic alignment capabilities from large-scale multimodal data.

[0202] To ensure that the original report interpretation content directly generated by the report interpretation model can better meet the reading habits of end users and the paradigm requirements in practical applications, this invention adds a post-processing and format conversion step.

[0203] Specifically, after obtaining the report interpretation content output by the report interpretation model, the method further includes: Based on the preset application scenario requirements, the report interpretation content is formatted and processed to generate the final physical examination report.

[0204] Post-formatting here is a process of polishing, rearranging, and enhancing text content. Its purpose is to transform machine-generated text, which may be relatively straightforward or have a simple format, into a beautifully formatted, concise, and easy-to-understand final health report.

[0205] The preset application scenario requirements can vary depending on the recipient of the report (e.g., whether it is directly aimed at ordinary users or used as a reference for professional doctors) or the display medium (e.g., whether it is displayed on the small screen of a mobile app or a printable A4 paper report).

[0206] This post-formatting process may specifically include one or more of the following operations: The numerical data contained in the report interpretation content is converted into preset text descriptions. This is an important means of enhancing the readability of the content. For example, the original report interpretation content output by the report interpretation model might contain a sentence like: "Your total cholesterol is 6.5 mmol / L, which is above the normal range (<5.2 mmol / L)." Directly showing these values ​​to non-professional users may cause confusion or unnecessary anxiety. The formatting post-processing module can identify such numerical comparisons and, according to a preset rule base, convert them into softer, more instructive preset text descriptions, such as: "Your total cholesterol level is slightly higher than normal, which suggests that you need to pay attention to your blood lipid health." This conversion makes the language of the report more humanized and easier for users to understand.

[0207] And / or, generate visualizations based on the data in the report interpretation. To more intuitively display the changing trends or abnormalities of health data, the post-processing module can also automatically generate visualizations based on the data in the report interpretation. For example, if the report interpretation mentions a user's blood pressure value, the post-processing module can automatically generate a sphygmomanometer-style dashboard, clearly marking the user's blood pressure value in areas such as "normal," "high," and "high risk." As another example, if the system has access to the user's historical physical examination data, when the report interpretation mentions changes in a certain indicator, it can automatically generate a line chart, clearly showing the trend of that indicator over the past few years. These visualizations help users grasp key information more quickly, greatly improving the intuitiveness and readability of the report.

[0208] This invention, by adding this formatting post-processing step, not only provides accurate report interpretation but also presents this content in a format best suited to the application scenario. Whether it's enhancing the accessibility of the content through numerical-to-text conversion or strengthening the intuitiveness of information by generating visual charts, it helps transform a professional medical examination report into a health guide that users can truly understand and act upon.

[0209] Figure 9 This is a schematic diagram of the structure of the medical examination report interpretation device provided by the present invention, as shown below. Figure 9 As shown, the present invention also provides a medical examination report interpretation device, which mainly includes, but is not limited to: The report calling unit 91 is used to break down the physical examination report to be interpreted into at least text content, image content and video content.

[0210] The information extraction unit 92 is used to input the text content, image content and video content into the multimodal parsing model to obtain the structured information summary output by the multimodal parsing model.

[0211] The report interpretation unit 93 is used to generate interpretation instruction text based on the structured information digest, and input the interpretation instruction text into the report interpretation model to obtain the report interpretation content output by the report interpretation model.

[0212] The multimodal parsing model includes an encoder cluster and a feature fusion unit. The encoder cluster extracts features for each modality in the multimodal content. The feature fusion unit generates the structured information summary by mapping weighted modal features to a unified multimodal embedding space. The weighted modal features are obtained by dynamically weighting the initial modal features output by the encoder cluster based on the attention weights between the modal features.

[0213] It should be noted that the medical examination report interpretation device provided by the present invention can execute the medical examination report interpretation method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0214] The medical examination report interpretation device provided by this invention can achieve a comprehensive and accurate interpretation of medical examination reports by comprehensively analyzing and fusing data in three modalities: text, images, and video. This avoids information omissions caused by a single modality and lays a solid foundation for providing personalized health management suggestions.

[0215] Figure 10 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 10 As shown, the electronic device may include a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040, wherein the processor 1010, communications interface 1020, and memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute a medical examination report interpretation method. This method includes: decomposing the medical examination report to be interpreted into at least text content, image content, and video content; inputting the text content, image content, and video content into a multimodal parsing model to obtain a structured information summary output by the multimodal parsing model; generating interpretation instruction text based on the structured information summary, and inputting the interpretation instruction text into a report interpretation model to obtain the report interpretation content output by the report interpretation model.

[0216] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0217] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the medical examination report interpretation method provided in the above embodiments, the method including: decomposing the medical examination report to be interpreted into at least text content, image content and video content; inputting the text content, image content and video content into a multimodal parsing model to obtain a structured information digest output by the multimodal parsing model; generating interpretation instruction text based on the structured information digest, and inputting the interpretation instruction text into a report interpretation model to obtain report interpretation content output by the report interpretation model.

[0218] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the method for interpreting medical examination reports provided in the above embodiments. The method includes: decomposing the medical examination report to be interpreted into at least text content, image content, and video content; inputting the text content, image content, and video content into a multimodal parsing model to obtain a structured information digest output by the multimodal parsing model; generating an interpretation instruction text based on the structured information digest, and inputting the interpretation instruction text into a report interpretation model to obtain the report interpretation content output by the report interpretation model.

[0219] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0220] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0221] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for interpreting a medical examination report, characterized in that, include: The medical examination report to be interpreted is broken down into multimodal content; The multimodal content is input into a multimodal parsing model to obtain a structured information summary output by the multimodal parsing model; An interpretation instruction text is generated based on the structured information digest, and the interpretation instruction text is input into the report interpretation model to obtain the report interpretation content output by the report interpretation model; The multimodal parsing model includes an encoder cluster and a feature fusion unit. The encoder cluster extracts features for each modality in the multimodal content. The feature fusion unit generates the structured information summary by mapping weighted modal features to a unified multimodal embedding space. The weighted modal features are obtained by weighting the initial modal features output by the encoder cluster based on the attention weights between the modal features.

2. The method for interpreting a medical examination report according to claim 1, characterized in that, The multimodal content includes various types of text content, image content, and video content.

3. The method for interpreting a medical examination report according to claim 1, characterized in that, The generation of interpretation instruction text based on the structured information digest includes: Obtain user profile information, wherein the user profile information is the profile information of the user of the physical examination report; Determine the target prompt template from a pre-built instruction configuration strategy; The target prompt template is used to integrate the user profile information and the structured information summary into the interpretation instruction text.

4. The method for interpreting a medical examination report according to claim 3, characterized in that, The step of determining the target prompt template from the pre-built instruction configuration strategy includes: Determine the type of health checkup package for the user; The target prompt template is obtained by mapping the physical examination package type from the instruction configuration strategy.

5. The method for interpreting a medical examination report according to claim 2, characterized in that, The encoder cluster includes a text encoder, an image encoder, and a video encoder; The text encoder is a network model based on the Transformer architecture, the image encoder is a visual Transformer model, and the video encoder is a three-dimensional convolutional neural network.

6. The method for interpreting a medical examination report according to claim 5, characterized in that, The multimodal content includes at least text content; Before inputting the multimodal content into the multimodal parsing model, the method further includes: performing word segmentation on the text content using byte pair encoding.

7. The method for interpreting a medical examination report according to claim 5, characterized in that, The steps of the visual Transformer model to extract features from the image content include: For each image in the image content, it is divided into multiple image blocks; A self-attention mechanism is used to calculate the association weights between the multiple image patches, and the feature representations of the multiple image patches are aggregated based on the association weights to generate an image feature vector representing each image.

8. The method for interpreting a medical examination report according to any one of claims 5-7, characterized in that, The step of inputting the multimodal content into a multimodal parsing model to obtain a structured information summary output by the multimodal parsing model includes: The text content, the image content, and the video content are input into the encoder cluster of the multimodal parsing model to obtain the initial modal features of the text, the initial modal features of the image, and the initial modal features of the video output by the multimodal parsing model. Determine the first attention weight between image features and text features and image features, the second attention weight between image features and text features and image features, and the third attention weight between text features and image features and image features. Based on the first attention weight, the second attention weight, and the third attention weight, the initial text modal features, the initial image modal features, and the initial video modal features are weighted and fused to obtain weighted text modal features, weighted image modal features, and weighted video modal features; The text-weighted modal features, image-weighted modal features, and video-weighted modal features are mapped to the multimodal embedding space and then fused to obtain the structured information summary.

9. The method for interpreting a medical examination report according to claims 5-7, characterized in that, The multimodal analytical model is trained based on the following steps: Construct a training dataset, wherein the training data in the training dataset includes at least image-text pairs consisting of image samples and corresponding text samples, and image-text pairs consisting of video samples and corresponding text samples. The training data in the training dataset is used as the input to the multimodal parsing model, and the contrastive loss is calculated based on the output of the multimodal parsing model. Based on the contrast loss, the trainable parameters of the text encoder, the image encoder, the video encoder, and the feature fusion unit are jointly updated.

10. The method for interpreting a medical examination report according to claim 9, characterized in that, The calculation of the contrast loss based on the output of the multimodal analytical model includes: Obtain the image features generated by the image encoder processing the image samples in the image-text pair, and obtain the image features generated by the image encoder processing the image samples in the image-text pair; Obtain the image text features and image text features generated by the text encoder after processing the text samples in the image text pair and the text samples in the image text pair respectively; Calculate the image-text contrast loss between the image features and the image-text features, and calculate the image-text contrast loss between the image features and the image-text features; The sum of the image-text contrast loss and the video-text contrast loss is taken as the contrast loss.

11. The method for interpreting a medical examination report according to claim 10, characterized in that, Before calculating the image-text contrast loss between the image features and the image-text features, the method also includes... The image features, the video features, the image text features, and the video text features are projected into the multimodal embedding space using their respective projection matrices. The image embedding representation, video embedding representation, image-text embedding representation, and video-text embedding representation obtained after projection are subjected to L2 normalization to obtain normalized features. The image-text contrast loss and the video-text contrast loss are calculated based on the normalized features.

12. The method for interpreting a medical examination report according to claim 10, characterized in that, The image-text contrast loss is determined based on the following steps: Based on the image features and the image text features, an image-text similarity matrix is ​​calculated. The first-direction loss and the second-direction loss are calculated based on the image-text similarity matrix, respectively. The first directional loss is determined based on the similarity between each image sample and its matching and non-matching text samples calculated from the image-text similarity matrix; the second directional loss is determined based on the similarity between each text sample and its matching and non-matching image samples calculated from the image-text similarity matrix. The first directional loss is combined with the second directional loss to obtain the image-text contrast loss.

13. The method for interpreting a medical examination report according to claim 10, characterized in that, The image-text contrast loss is determined based on the following steps: Based on the image features and the image text features, an image-text similarity matrix is ​​calculated. The third-direction loss and the fourth-direction loss are calculated based on the image-text similarity matrix. The third-direction loss is determined based on the similarity between each image sample and its matching and non-matching text samples calculated from the image-text similarity matrix; the fourth-direction loss is determined based on the similarity between each text sample and its matching and non-matching image samples calculated from the image-text similarity matrix. The third-direction loss is combined with the fourth-direction loss to obtain the image-text contrast loss.

14. The method for interpreting a medical examination report according to claim 9, characterized in that, The step of jointly updating the trainable parameters of the text encoder, the image encoder, the video encoder, and the feature fusion unit based on the contrast loss specifically includes: A backpropagation algorithm is used to calculate the gradient of the contrast loss with respect to the trainable parameters of the text encoder, the image encoder, the video encoder, and the feature fusion unit; An optimizer is used to adjust the trainable parameters based on the gradient; The step of calculating the contrastive loss is performed iteratively and repeatedly for different batches of training data in the training dataset until the step of adjusting the trainable parameters according to the gradient is met, until a preset convergence condition is satisfied.

15. The method for interpreting a medical examination report according to claim 1, characterized in that, After obtaining the report interpretation content output by the report interpretation model, the process further includes: Based on the preset application scenario requirements, the report interpretation content is formatted and processed to generate the final physical examination report; The formatting post-processing includes converting the numerical data contained in the report interpretation content into a preset text description; And / or, generate visualization charts based on the data in the report interpretation content.

16. A device for interpreting medical examination reports, characterized in that, include: The report retrieval unit is used to break down the medical examination report to be interpreted into multimodal content; An information extraction unit is used to input the multimodal content into a multimodal parsing model to obtain a structured information summary output by the multimodal parsing model. The report interpretation unit is used to generate interpretation instruction text based on the structured information digest, and input the interpretation instruction text into the report interpretation model to obtain the report interpretation content output by the report interpretation model; The multimodal parsing model includes an encoder cluster and a feature fusion unit. The encoder cluster extracts features for each modality in the multimodal content. The feature fusion unit generates the structured information summary by mapping weighted modal features to a unified multimodal embedding space. The weighted modal features are obtained by weighting the initial modal features output by the encoder cluster based on the attention weights between the modal features.

17. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for interpreting a medical examination report as described in any one of claims 1 to 15.

18. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for interpreting a medical examination report as described in any one of claims 1 to 15.

19. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for interpreting a medical examination report as described in any one of claims 1 to 15.

Images