A multimodal medical report structured analysis method, system, device, medium and program product

Abstract

The application provides a multi-modal medical report structured analysis method, system, device, medium and program product, and relates to the technical field of deep learning and medical image report analysis. The method comprises the following steps: identifying an input multi-modal file to obtain content information contained in the file; performing feature extraction on the content information to obtain a disease feature vector, and performing weighting processing on the disease feature vector to obtain corresponding disease information; performing weighted evaluation and marking on the disease data, and outputting the marked disease data. The application has the advantages of workflow context optimization and scalability based on modular skill units, noise robustness and illusion suppression in multi-modal data fusion, and disease extraction and missing detection avoidance with fine-grained context perception.

Description

Technical Field

[0001] This application relates to the fields of deep learning and medical image report analysis technology, and in particular to a multimodal medical report structured analysis method, system, device, medium and program product. Background Technology

[0002] In today's rapidly developing field of medical artificial intelligence, the automated processing of medical image reports has become one of the core challenges facing medical AI companies. Image reports generated by medical institutions exhibit significant multimodal characteristics, including but not limited to radiology images, structured reports in PDF format, and batch data in Excel or CSV formats. These heterogeneous data sources differ greatly in format, content specifications, and quality standards, posing serious technical challenges to the large-scale data collection and processing of medical AI companies. Compared to traditional natural image annotation, medical data annotation requires practitioners to have certain medical experience, which further increases the difficulty of data processing and significantly raises the company's data processing costs. Traditional medical report processing methods mainly rely on rule matching or template matching techniques.

[0003] Early systems, such as IBM Watson Health's image analysis module, employed regular expression-based keyword extraction methods. While these performed well in specific report formats, they proved severely inadequate for unstructured reports. Current research indicates that rule-based methods suffer from low accuracy in cross-institutional report processing and require extensive manual maintenance of the rule base. With the development of deep learning technology, some research has shifted towards sequence labeling models. For example, the BiLSTM-CRF architecture has made some progress in medical entity recognition tasks. However, these methods remain limited to a single text modality, failing to effectively handle multimodal inputs such as images and PDFs, and lacking a deep understanding of medical expertise.

[0004] In the field of multimodal data processing, existing technologies mainly fall into two categories: one is to use traditional optical character recognition (OCR) technology to convert non-text data into text before processing, such as Baidu PaddleOCR commercial services; the other is end-to-end multimodal models, such as Qwen-VL and its variants. Traditional OCR technology has inherent limitations when processing medical image reports: the specificity of medical terminology leads to a high error rate, and the complexity of report formats results in disordered text order. OCR based on large language models is still in the exploratory stage in the medical field, and how to apply large language models to industrial scenarios to reduce the error rate remains a challenge. Regarding structured information extraction, fine-grained disease classification is a key prerequisite for medical AI analysis. Traditional methods often employ a "one-size-fits-all" extraction strategy, failing to distinguish disease characteristics in different locations. For example, the descriptive standards for lung nodules (such as Lung-RADS classification) differ fundamentally from those for liver lesions (such as the LI-RADS standard), requiring domain-specific processing logic.

[0005] Recent research has begun exploring the application of Prompt Engineering in the medical field. Microsoft Research proposed the MedPrompt framework, which focuses on prompt design to improve the performance of LLM models in medical question answering, with significant results, fully demonstrating the powerful potential of prompt engineering. At the engineering implementation level, existing open-source frameworks such as LangChain and AutoGen provide basic agent capabilities, but they have significant shortcomings in the medical vertical: they lack specialized processing modules for multimodal medical data, and their workflow-based design is not conducive to project expansion, making it very inconvenient to extend to different types of diseases, requiring extensive refactoring. Therefore, we urgently need a completely new system to solve these challenges. Summary of the Invention

[0006] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a method, system, device, medium and program product for structured analysis of multimodal medical reports, in order to solve the technical problem that the prior art lacks professional processing for multimodal medical data in the medical vertical field.

[0007] To achieve the above and other related objectives, a first aspect of this application provides a method for structured analysis of multimodal medical reports, comprising: identifying an input multimodal file to obtain content information contained in the file; extracting features from the content information to obtain disease feature vectors; weighting the disease feature vectors to obtain corresponding disease information; weighting and evaluating the disease data and labeling it; and outputting the labeled disease data.

[0008] In some embodiments of the first aspect of this application, the weighted processing of the disease feature vector to obtain corresponding disease information includes: performing a first weighted processing on the disease feature vector to obtain global disease information; and performing a second weighted processing on the disease feature vector to obtain local disease information.

[0009] In some embodiments of the first aspect of this application, in the step of weighting the disease feature vector to obtain corresponding disease information, the disease feature vector serves as a guide word, and different disease analysis modules are invoked according to the different guide words to obtain corresponding disease information.

[0010] In some embodiments of the first aspect of this application, the input multimodal file is identified to obtain the content information contained in the file. Different content recognition modules are invoked according to the different types of the multimodal file to obtain the content information contained in the file.

[0011] In some embodiments of the first aspect of this application, the disease data is weighted and labeled, and the labeled disease data is output, including: performing a rule-weighted evaluation on the disease data, wherein the rule-weighted evaluation is a rule verification based on clinical knowledge; and performing a quality-weighted evaluation on the disease data, wherein the quality-weighted evaluation scores key fields for clinical rationality, descriptive completeness, and terminological standardization.

[0012] To achieve the above and other related objectives, a second aspect of this application provides a multimodal medical report structured analysis system, comprising: an identification module for identifying an input multimodal file to obtain content information contained in the file; a feature extraction module for extracting features from the content information to obtain disease feature vectors, and weighting the disease feature vectors to obtain corresponding disease information; and an output module for weighting and labeling the disease data, and outputting the labeled disease data.

[0013] To achieve the above and other related objectives, a third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal medical report structured analysis method.

[0014] To achieve the above and other related objectives, a fourth aspect of this application provides a computer program product comprising computer program code that, when executed on a computer, causes the computer to implement the multimodal medical report structured analysis method.

[0015] To achieve the above and other related objectives, a fifth aspect of this application provides a computer device / equipment / system / electronic terminal, including a memory, a processor, and a computer program stored in the memory; the processor executes the computer program to implement the multimodal medical report structured analysis method.

[0016] As described above, the multimodal medical report structured analysis method, system, device, medium, and program product of this application have the following beneficial effects:

[0017] 1) Workflow Context Optimization and Scalability Based on Modular Skill Units: Traditional workflows lack dynamic context optimization mechanisms, often requiring complete process refactoring when handling new reports or unknown diseases. This approach introduces a modular skill unit design, supporting minimal adjustments and plug-and-play functionality, significantly improving workflow scalability and adaptability while reducing maintenance and iteration costs.

[0018] 2) Noise robustness and illusion suppression in multimodal data fusion: Traditional methods are prone to "illusion" phenomena (i.e., generating false labels that do not conform to reality) when processing multimodal inputs due to data noise interference. This method, by invoking different skills at different levels, identifies and extracts multimodal inputs with finer granularity, significantly improving the accuracy and reliability of label extraction and fundamentally avoiding erroneous outputs.

[0019] 3) Fine-grained context-aware disease extraction and missed detection avoidance: Existing disease extraction methods are often limited by coarse-grained modeling and insufficient context adaptation, leading to the omission of key information. This method adopts a multi-gradient skill scheduling strategy, combined with the semantic association between anatomical sites and disease descriptions, to achieve accurate context-aware identification, thereby comprehensively covering disease features and effectively solving the problem of missed detection. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall process of the structured analysis method for multimodal medical image reports in the embodiments of this application.

[0021] Figure 2 This is a flowchart illustrating the sub-step of step 2) in the embodiments of this application.

[0022] Figure 3 This is a flowchart illustrating step 3) in the embodiments of this application.

[0023] Figure 4 This is an architecture diagram of the multimodal medical image report structured analysis method in the embodiments of this application.

[0024] Figure 5 This is a schematic diagram of a specific medical image report in an embodiment of this application.

[0025] Figure 6 This is a schematic diagram of the structure of the multimodal medical image report structured analysis system in the embodiments of this application.

[0026] Figure 7 This is a schematic diagram of the structure of the computer device in the embodiments of this application. Detailed Implementation

[0027] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0028] To address the technical problems described above, this application provides a multimodal medical report structured analysis method. It should be understood that the method provided in this application, besides being applied to medical reports, can also be applied to reports in other fields, such as industrial inspection and equipment maintenance reports, financial analysis and investment reports, legal documents and case summaries, etc.

[0029] Figure 1 and Figure 4 This document illustrates a flowchart of a multimodal medical report structured analysis method according to an embodiment of this application. The method in this embodiment mainly includes the following steps:

[0030] In one embodiment, such as Figure 1 As shown, a multimodal medical report structured analysis method is provided. This embodiment illustrates the method's application to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0031] Step 1) Identify the input multimodal medical report to obtain the content information contained in the document.

[0032] In practice, the multimodal medical report is input into the terminal, and the terminal identifies the input modal file to obtain the content information contained in the file.

[0033] In practical applications, multimodal medical reports are transmitted to the terminal via wired or wireless means. The terminal has a certain storage capacity to store the multimodal medical reports. Subsequently, the terminal calls the processor to perform recognition and calculation on the multimodal medical reports using pre-stored program logic, thereby obtaining the content information contained in the file.

[0034] Multimodal refers to the types of medical reports, such as medical images, medical text, or structured sources like CSV files.

[0035] In one example embodiment, different recognition methods are used for medical reports from different modalities of medicine. For instance, for image input, a multimodal large language model (such as LLaVA or Qwen-VL) is invoked, leveraging its native visual-language joint understanding capabilities to directly extract semantic text from pixels, significantly improving the robustness of parsing complex layouts such as tables, arrow annotations, and overlaid text in medical screenshots. Another example is for PDF input, where the original text layer is extracted using PyMuPDF. If the document is identified as a scanned document, it is automatically converted into a high-fidelity image and processed by a multimodal LLM (large language model). Yet another example is for structured sources such as CSV files, which are directly parsed.

[0036] In addition, the paths, formats, processing strategies, and timestamps of all original files can be written to an extensible metadata list in real time, laying the foundation for end-to-end traceability.

[0037] Furthermore, after obtaining the content information contained in the file, the content information is trimmed and compressed. Trimming involves filtering out redundant information from the content information; for example, the end of a medical report may contain unnecessary information such as the doctor's signature, which can be trimmed using keywords. Compression involves filtering out noise from the content information and retaining valid information; for example, when inputting long text, excessive noise can be compressed using specific guiding words. Both trimming and compression functions can be directly implemented using large language models, such as the Qianwen language model.

[0038] Step 2) Extract features from the content information to obtain disease feature vectors, and perform weighted processing on the disease feature vectors to obtain corresponding disease information.

[0039] The disease information refers to information related to the disease, such as disease name, disease region, disease severity, and disease onset time. For clarity, this content information typically includes noise information unrelated to the disease, such as identifiers or text in headers and page numbers. This noise information needs to be filtered out during feature extraction. Specifically, for images, a large language model is first used to recognize all text information in the image, converting the image into a text sequence. Then, a prompt word engineering process is used to extract disease-related content from the text sequence. The disease feature vector serves as a guide word. Based on the different guide words, the terminal automatically selects a first weighted model suitable for processing the disease feature vector and obtains global disease information through this model.

[0040] In one exemplary embodiment, the disease feature vectors are all text sequences. For example, for image-based medical reports, a multimodal large language model (such as LLaVA or Qwen-VL) can be used as a weighted model. Its native visual-language joint understanding capability directly extracts semantic text from pixels, significantly improving the robustness of parsing complex layouts such as tables, arrow annotations, and overlaid text in medical screenshots. Another example: for PDF-type medical reports, the native text layer is extracted first using PyMuPDF. If it is determined to be a scanned document (without a valid text stream), it is automatically converted into a high-fidelity image and processed by a multimodal LLM. Yet another example: for medical reports from structured sources such as CSV files, they are directly parsed. Furthermore, the paths, formats, processing strategies, and timestamps of all original files are written to an extensible metadata list in real time, laying the foundation for end-to-end traceability.

[0041] As an explanation, LLM is based on the Transformer architecture, and can be guided by Prompt (prompt word engineering) to make the features extracted by LLM more targeted.

[0042] Specifically, the implementation process of step 2) can be further divided into, for example, Figure 2 The following steps are shown:

[0043] Step 2-1) Perform a first weighted processing on the disease feature vector to obtain global disease information.

[0044] It should be understood that the global disease information refers to all diseases contained in the disease feature vector. For example, for an abdominal MRI report, it contains diseases in multiple regions, such as the lung region, the kidney region, etc. The global disease information includes information on diseases in all regions and comprehensive diseases.

[0045] In its specific implementation, the terminal performs a first weighted processing on the disease feature vector. This first weighted processing involves a unified initial screening of lesions across all anatomical sites, invoking the general Propmt algorithm. This process occupies minimal context; Propmt does not pre-define any anatomical region or disease type, but only requires the LLM to identify all mentioned abnormalities or diseases from the report, along with their corresponding anatomical regions, and outputs a structured result as the basis for scheduling subsequent processing.

[0046] For example, such as Figure 5 The image shows an MRI report. When performing the first weighted processing on an abdominal MRI report, LLM extracts global disease information such as lung and kidney regions from the report and outputs structured results. Simply put, it performs structured analysis on a relatively messy MRI report, divides the disease-related parts according to disease regions, and outputs them in a structured form.

[0047] Step 2-2) Perform a second weighted processing on the disease feature vector to obtain local disease information. Specifically, the second weighted processing is guided by different regions or different diseases as guiding terms to obtain local disease information for the corresponding regions.

[0048] It should be understood that the local disease information refers to the local diseases contained in the disease feature vector. For example, a single abdominal MRI report may contain diseases of multiple organs, such as lung diseases, stomach diseases, kidney diseases, etc., which are independent of each other. Therefore, in general, local disease information is included in the global disease information, and the second weighted processing is based on the global disease information extracted by the first weighted processing, further extracting the local diseases separately.

[0049] In the specific implementation, the terminal performs a second weighting process on the disease feature vectors. Further, multiple second weighting models are pre-trained for different disease feature vectors. Each weighting model is used to extract local diseases from the disease feature vectors. For example, for diseases in the lung region, a second weighting model specifically for identifying lung diseases can be trained; for kidney diseases, a second weighting model specifically for identifying kidney diseases can be trained; and for stomach diseases, a second weighting model specifically for identifying stomach diseases can be trained. Furthermore, for the anatomical regions identified in the initial screening, the corresponding dedicated structured extraction skills (second weighting models) are dynamically activated. Each skill embeds a guiding word for disease extraction from a specific medical imaging site and a corresponding adapted large language model, ultimately outputting a specified structured format.

[0050] In addition, to optimize the context structure, the original report is intelligently pruned using context engineering skills before calling LLM. The context engineering skills combine pre-set guide words containing relevant knowledge with the large language model to retain only text fragments that are highly relevant to the target part, avoiding redundant information from crowding out the limited attention budget, while optimizing the output results.

[0051] For example, such as Figure 5 The image shows an MRI report. For an abdominal MRI report that has already undergone the first weighting process, different techniques (equivalent to the second weighting model) are used to perform the second weighting process for different disease areas.

[0052] For example, after the first weighted processing, the resulting regional diseases include diseases in the stomach and lung regions. Then, for the stomach region diseases, a skill specific to the stomach (the second weighted model) is invoked to further process the stomach disease features to obtain stomach disease information. Similarly, for the lung region diseases, a skill specific to the lungs (the second weighted model) is invoked to further process the lung disease features to obtain stomach disease information. In short, the first weighted processing acts as a coarse-grained process, modularly processing the regions, such as identifying diseases in the stomach and lungs. The second weighted processing acts as a fine-grained process, extracting details such as the names of diseases in the stomach and related data, and the names of diseases in the lungs and related data.

[0053] Steps 2-3) Output the global and local disease information in a structured format as disease data.

[0054] It should be understood that "structured" refers to a predefined format, such as a predefined JSON format. The advantage of structured formats is that, for example, JSON formats can be validated using format validation trees to ensure field integrity and type compliance.

[0055] In one example implementation, the structured results formed from all global and local disease information are aggregated and mapped to a unified CSV Schema, with fields including: report_id, image_path (e.g., .. / images / xxx.png), anatomy_region, disease_type, lesion_size_mm, density, etc. A JSON manifest is generated simultaneously, recording the extraction skill, prompt version, confidence level, and original context snapshot for each lesion to ensure evaluability.

[0056] Step 3) Perform weighted evaluation and labeling on the disease data, and output the labeled disease data.

[0057] It should be understood that the weighted assessment refers to first weighting the disease data, and then evaluating the results based on the weighted data. The labeling refers to adding labels to the evaluated results.

[0058] The assessment may include rule-based assessment under defined rules and quality assessment of the disease data. Rule-based assessment may include, for example, evaluating the completeness of the disease data and the presence of potential errors. Quality assessment may include, for example, assessing the quality of key lesions represented in the disease data.

[0059] Specifically, the implementation process of step 3) can be further divided into, for example, Figure 3 The following steps are shown:

[0060] Step 3-1) Perform rule-weighted evaluation on the disease data. The rule-weighted evaluation is a rule verification based on clinical knowledge.

[0061] In a specific implementation, the terminal performs rule-weighted evaluation on the disease data. This rule-weighted evaluation can be based on clinical knowledge-based rule verification, such as: "Lung nodule diameter must not be negative," or "LGE sequence must not show fat signals," etc.

[0062] In one example embodiment, the rule-weighted evaluation uses a pre-trained model, which can be an Alibaba Qianwen series model.

[0063] Step 3-2) Perform a quality-weighted assessment on the disease data, which scores the key fields for clinical rationality, descriptive completeness, and terminological standardization.

[0064] In its implementation, the terminal performs a quality-weighted assessment of the disease data. This assessment incorporates an LLM-as-a-Judge mechanism, scoring key fields on clinical rationality, descriptive completeness, and terminological standardization. Cases with a comprehensive score below a threshold are automatically marked as requiring manual review and pushed to the annotation platform, forming a continuous optimization loop through human-machine collaboration.

[0065] In one example embodiment, the quality-weighted evaluation uses a pre-trained model, which can be an Alibaba Qianwen series model.

[0066] For ease of understanding, combined with Figure 4 The flowchart in the document provides a complete illustration of the model inference process in the embodiments of this application:

[0067] Step 1: Enter the medical report;

[0068] Step 2: Use LLM to identify the content information in the medical report;

[0069] Step 3: Use LLM to extract global disease information from the content information;

[0070] Step 4: Using the anatomical information in the global disease information as the Pormpt, call the associated LLM to extract local disease information;

[0071] Step 5: Structure and output the global and local disease information as disease information;

[0072] Step 6: Perform rule scoring and quality scoring on the structured output disease information.

[0073] Step 7: Obtain the final disease information from the medical report.

[0074] Example:

[0075] Multimodal Input and Metadata Registration: The input is an image file (.jpg / .png). After the system recognizes the file input, it activates the scheduling module. After scanning the skill list, it no longer calls the traditional OCR engine, but directly configures and calls a multimodal large language model (such as LLaVA, Qwen-VL). The model input is raw pixel data and specific prompts for the corresponding skills, and the output is semantic text. It focuses on parsing complex layouts such as tables, arrow annotations, and overlaid text. After processing, the system uses the original file path, format identifier, processing strategy, and current timestamp as metadata parameters, writing them in real time to an extensible metadata list, completing the standardized registration of the input layer.

[0076] Text Refinement and Persistent Storage: The input is the semantic text generated in the previous stage. The system schedules and activates the context trimming and compression skill. This skill performs text trimming operations, with parameter configuration including rules for removing visual noise such as headers, footers, and advertising watermarks. Subsequently, based on the rule-based expression correction parameters, medical expression calibration, sentence fragment recombination, and redundant description elimination are performed. The processed data generates a context representation with high signal-to-noise ratio and high semantic density, which is persistently stored in JSONL format at a specified path, with an attached file tag parameter to ensure that subsequent processes can use the tag parameter to determine whether to skip already generated items.

[0077] Two-level progressive disease information extraction: The input is refined JSONL text. The system scheduling module initiates a context-aware two-level progressive parsing architecture. The first level schedules a lightweight whole-body lesion coarse regression skill, configures general guide word parameters, and outputs a coarse mapping of lesions to anatomical regions. The second level, based on this mapping result, schedules the system to dynamically load multi-level information extraction modules based on site information, combining different lesion region guide words with the large language model. At runtime, only the minimum context fragment parameters strongly related to the current task are loaded, and constraint guide words are injected. The output is in a predefined JSON format, and parameters are validated using a JSON format verification mechanism to ensure field integrity and type compliance.

[0078] Results standardization and traceability list generation: The input is a validated structured JSON result, which the system aggregates and maps to a unified CSVSchema parameter set. Specific fields include report ID, file path, labeled area, disease type, etc. Simultaneously, the system generates a JSON list recording the extraction skill identifier, Prompt version parameter, confidence score, and original context snapshot for each lesion, ensuring that all output results have evaluable traceability parameters.

[0079] Two-layer quality assessment and human-machine collaborative closed loop: The input consists of structured results and a traceability list. The system-integrated verification module initiates a two-layer quality assessment closed loop. The first layer configures rule-based validation parameters based on clinical knowledge (such as "lung nodule diameter must not be negative" and "LGE sequence must not show fat signal"). The second layer introduces an LLM-as-a-Judge mechanism, where a medically fine-tuned evaluation model scores key fields for clinical rationality, descriptive completeness, and terminological standardization. If the overall score is lower than a set threshold parameter, the case is automatically marked as "requiring manual review" and pushed to the annotation platform, forming a continuous optimization loop of human-machine collaboration. The final output is a result file containing the evaluation score.

[0080] It should also be noted that in the embodiments of this application, the terms "first" and "second" are used to distinguish identical or similar items with essentially the same function and effect.

[0081] It should be noted that, in the embodiments of this application, the words "exemplary" or "for example" indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0082] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0083] Figure 6 This is a schematic block diagram of the multimodal medical report structured analysis system provided in the embodiments of this application. Figure 6 As shown, the system includes: an identification module 51, used to identify the input multimodal file to obtain the content information contained in the file; a feature extraction module 52, used to extract features from the content information to obtain disease feature vectors, and to perform weighted processing on the disease feature vectors to obtain corresponding disease information; and an output module 53, used to perform weighted evaluation and labeling on the disease data, and to output the labeled disease data.

[0084] It should be understood that the specific processes by which each module performs the corresponding steps described above have been detailed in the above method embodiments, and will not be repeated here for the sake of brevity. It should also be understood that the module division in the embodiments of this application is illustrative and merely a logical functional division; other division methods may exist in actual implementation. Furthermore, the functional modules in the various embodiments of this application can be integrated into a single processor, exist as separate physical entities, or have two or more modules integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0085] Figure 5 This is a schematic block diagram of a computer device provided in an embodiment of this application. Figure 7 As shown, the computer device includes at least one processor 601, a memory 602, at least one network interface 603, and a user interface 605. The various components in the device are coupled together via a bus system 604. It is understood that the bus system 604 is used to implement communication between these components. In addition to a data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 6 The general will label all buses as bus systems.

[0086] The user interface 605 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0087] It is understood that memory 602 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this application are intended to include, but are not limited to, these and any other suitable categories of memory.

[0088] In this embodiment, the memory 602 is used to store various types of data to support the operation of the electronic terminal 600. Examples of this data include any executable program that operates on the electronic terminal 600, such as the operating system 6021 and application programs 6022. The operating system 6021 contains various system programs, such as the framework layer, core library layer, and driver layer, for implementing various basic services and handling hardware-based tasks. The application program 6022 may contain various applications, such as a media player and a browser, for implementing various application services. The implementation of the multimodal medical report structured analysis method provided in this embodiment can be included in the application program 6022.

[0089] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 601. Processor 601 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 601 or by instructions in software form. The processor 601 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 601 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. General-purpose processor 601 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of this application can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a storage medium, which is located in memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0090] In an exemplary embodiment, the electronic terminal 600 may be used by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs) to perform the aforementioned method.

[0091] According to the method provided in the embodiments of this application, this application also provides a computer program product, which includes: computer program code, which, when run on a computer, causes the computer to execute... Figures 1 to 4The multimodal medical report structured analysis method of any of the embodiments shown.

[0092] According to the method provided in the embodiments of this application, this application also provides a computer-readable storage medium storing program code, which, when run on a computer, causes the computer to perform the above-described method.

[0093] As used in this specification, the terms "component," "module," "system," etc., are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on computing devices and computing devices can both be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0094] Those skilled in the art will recognize that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this application.

[0095] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0096] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0097] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0098] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0099] In the above embodiments, the functions of each functional unit can be implemented entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. A computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. Computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., high-density digital video discs, DVDs), or semiconductor media (e.g., solid-state disks, SSDs, etc.).

[0100] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0101] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0102] In summary, this application provides a method, system, device, medium, and program product for structured analysis of multimodal medical reports. Therefore, this application effectively overcomes the various shortcomings of the prior art and has high industrial applicability.

[0103] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

[0104] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and substitutions can be made without departing from the principles of the present invention, and these improvements and substitutions should also be considered within the scope of protection of the present invention.

Claims

1. A method for structured analysis of multimodal medical reports, characterized in that, include: The input multimodal file is identified to obtain the content information contained in the file; The content information is subjected to feature extraction to obtain a disease feature vector, and the disease feature vector is subjected to weighted processing to obtain the corresponding disease information; The disease data is weighted, evaluated, and labeled, and the labeled disease data is output.

2. The method according to claim 1, characterized in that, The weighted processing of the disease feature vector to obtain corresponding disease information includes: The disease feature vector is subjected to a first weighting process to obtain global disease information; The disease feature vector is subjected to a second weighting process to obtain local disease information.

3. The method according to claim 2, characterized in that, The disease feature vector is subjected to a second weighting process to obtain local disease information. Different regions or different diseases are used as guiding words to guide the second weighting process to obtain local disease information of the corresponding regions.

4. The method according to claim 1, characterized in that, The input multimodal file is identified to obtain the content information contained in the file. Different content recognition modules are called according to the different types of multimodal files to obtain the content information contained in the file.

5. The method according to claim 1, characterized in that, The disease data is weighted, evaluated, and labeled, and the labeled disease data is output, including: The disease data is subjected to rule-weighted evaluation, which is a rule verification based on clinical knowledge; The disease data is subjected to a quality-weighted assessment, which scores key fields for clinical rationality, descriptive completeness, and terminological standardization.

6. The method according to claim 1, characterized in that, The process involves identifying the input multimodal file to obtain its content information, and then cropping and compressing that content information. The cropping refers to filtering out redundant information from the content information; The compression process filters out noise from the content information while retaining the valid information.

7. A multimodal medical report structured analysis system, characterized in that, include: The recognition module is used to recognize the input multimodal files in order to obtain the content information contained in the files; The feature extraction module is used to extract features from the content information to obtain disease feature vectors, and to perform weighted processing on the disease feature vectors to obtain corresponding disease information. The output module is used to perform weighted evaluation and labeling on the disease data, and output the labeled disease data.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal medical report structured analysis method according to any one of claims 1 to 6.

9. A computer program product, characterized in that, The computer program product includes computer program code, which, when run on a computer, causes the computer to implement the multimodal medical report structured analysis method as described in any one of claims 1 to 6.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the multimodal medical report structured analysis method according to any one of claims 1 to 6.

Images