Structured mapping methods, devices, equipment, and media for multimodal medical data

Abstract

This disclosure provides a method, apparatus, device, and medium for structured mapping of multimodal medical data, comprising: acquiring multimodal medical data, which includes first multi-source data and / or second multi-source data; identifying corresponding key fields from the first multi-source data using region localization and character recognition strategies, and identifying corresponding key parameters from the second multi-source data using regular expression matching and model recognition strategies, to extract features from the multimodal medical data to obtain multimodal feature data; splitting the multimodal feature data into minimum semantic units to obtain multimodal unit data; performing multi-level mapping on the multimodal unit data based on a multi-level mapping engine to obtain multimodal mapped data; performing data mapping detection on the multimodal mapped data, and contextually associating target unit data that has not been mapped in the multimodal mapped data to obtain medical structured data corresponding to the multimodal medical data. This effectively improves data mapping efficiency.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of data structuring technology, and more specifically, to a method, apparatus, device, and medium for structuring mapping of multimodal medical data. Background Technology

[0002] Structured medical data can quickly and accurately transmit medical information, reduce repetitive work and communication costs in the medical process, and provide doctors with comprehensive and accurate clinical information to assist them in making diagnostic and treatment decisions, thereby improving the quality and safety of medical care.

[0003] In related technologies, international / national standard codes (such as ICD-9 / ICD-10, FHIR) are typically used to classify and map medical data such as diagnoses, medications, and surgeries, thereby standardizing variables. For example, local hospital terminology can be mapped to national or international standards through unified coding rules, and disease-specific data models can be used to integrate data on visits, medications, and surgeries.

[0004] However, using existing methods results in inefficient data processing. Summary of the Invention

[0005] The embodiments described herein provide a method, apparatus, device, and medium for structured mapping of multimodal medical data that overcomes the aforementioned problems.

[0006] Firstly, based on the content of this disclosure, a structured mapping method for multimodal medical data is provided, including:

[0007] Acquire multimodal medical data, which includes: first multi-source data and / or second multi-source data, wherein the first multi-source data is used to describe unstructured PDF data and unstructured image data, and the second multi-source data is used to describe unstructured text data;

[0008] For the first multi-source data in multimodal medical data, a region localization and character recognition strategy is used to identify the corresponding key fields from the first multi-source data. For the second multi-source data in multimodal medical data, a regular expression matching and model recognition strategy is used to identify the corresponding key parameters from the second multi-source data, so as to extract features from the multimodal medical data and obtain multimodal feature data.

[0009] Multimodal feature data is split into minimum semantic units to obtain multimodal unit data;

[0010] Multi-modal unit data is mapped in multiple levels based on a multi-level mapping engine to obtain multi-modal mapped data. The multi-level mapping engine includes a rule mapping engine, a semantic mapping engine, and a statistical mapping engine. The rule mapping engine is used to map data using regular rules, the semantic mapping engine is used to map data using knowledge graphs, and the statistical mapping engine is used to map data using historical data.

[0011] Data mapping detection is performed on the multimodal mapping data, and context association is performed on the target unit data that has not been mapped in the multimodal mapping data to obtain the medical structured data corresponding to the multimodal medical data.

[0012] Secondly, according to the present disclosure, a structured mapping apparatus for multimodal medical data is provided, comprising:

[0013] The acquisition module is used to acquire multimodal medical data, which includes: first multi-source data and / or second multi-source data. The first multi-source data is used to describe unstructured PDF data and unstructured image data, and the second multi-source data is used to describe unstructured text data.

[0014] The identification module is used to identify the corresponding key fields from the first multi-source data in the multimodal medical data using regional positioning and character recognition strategies, and to identify the corresponding key parameters from the second multi-source data using regular expression matching and model recognition strategies, so as to extract features from the multimodal medical data and obtain multimodal feature data.

[0015] The splitting module is used to split multimodal feature data into minimum semantic units to obtain multimodal unit data;

[0016] The mapping module is used to perform multi-level mapping on multimodal unit data based on a multi-level mapping engine to obtain multimodal mapped data. The multi-level mapping engine includes a rule mapping engine, a semantic mapping engine, and a statistical mapping engine. The rule mapping engine is used to map data using regular rules, the semantic mapping engine is used to map data using knowledge graphs, and the statistical mapping engine is used to map data using historical data.

[0017] The determination module is used to perform data mapping detection on multimodal mapping data and to perform context association on target unit data that has not been mapped in multimodal mapping data to obtain medical structured data corresponding to multimodal medical data.

[0018] Thirdly, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the structured mapping method for multimodal medical data as described in any of the above embodiments.

[0019] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, and when executed by a processor, the computer program implements the steps of the structured mapping method for multimodal medical data as described in any of the above embodiments.

[0020] The structured mapping method for multimodal medical data provided in this application embodiment acquires multimodal medical data, which includes: first multi-source data and / or second multi-source data. The first multi-source data is used to describe unstructured PDF data and unstructured image data, and the second multi-source data is used to describe unstructured text data. For the first multi-source data in the multimodal medical data, a region localization and character recognition strategy is used to identify corresponding key fields from the first multi-source data. For the second multi-source data in the multimodal medical data, a regular expression matching and model recognition strategy is used to identify corresponding key parameters from the second multi-source data, so as to perform feature extraction on the multimodal medical data and obtain multimodal medical data. The system extracts multimodal feature data; it then performs minimum semantic unit decomposition on the multimodal feature data to obtain multimodal unit data; based on a multi-level mapping engine, it performs multi-level mapping on the multimodal unit data to obtain multimodal mapped data. The multi-level mapping engine includes a rule mapping engine, a semantic mapping engine, and a statistical mapping engine. The rule mapping engine is used for data mapping using regular rules, the semantic mapping engine is used for data mapping using knowledge graphs, and the statistical mapping engine is used for data mapping using historical data. Data mapping detection is performed on the multimodal mapped data, and context association is performed on the target unit data in the multimodal mapped data to obtain the medical structured data corresponding to the multimodal medical data. Thus, by extracting features from different multi-source data, performing minimum semantic unit decomposition and data mapping, and combining context for data association of unmapped unit data, the efficiency of data mapping is effectively improved.

[0021] The above description is merely an overview of the technical solutions of the embodiments of this application. In order to better understand the technical means of the embodiments of this application and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of this application more obvious and understandable, specific implementation methods of this application are described below. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. It should be understood that the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure, wherein:

[0023] Figure 1 This is a flowchart illustrating a structured mapping method for multimodal medical data provided in this publication.

[0024] Figure 2 This is a schematic diagram of a structured mapping device for multimodal medical data provided in this disclosure.

[0025] Figure 3 This is a schematic diagram of the structure of a computer device provided in this disclosure.

[0026] It should be noted that the elements in the attached diagram are schematic and not drawn to scale. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are also within the scope of protection of this disclosure.

[0028] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the specification and in the relevant art, and shall not be interpreted in an idealized or overly formal form unless otherwise explicitly defined herein. As used herein, the statement of “connecting” or “coupling” two or more parts together shall mean that these parts are directly joined together or joined through one or more intermediate components.

[0029] The term "embodiment" as used herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of the phrase "embodiment" in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0030] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists, A and B exist simultaneously, or B exists. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Terms such as "first" and "second" are only used to distinguish one component (or part of a component) from another component (or another part of a component).

[0031] In the description of this application, unless otherwise stated, "multiple" means two or more (including two), and similarly, "multiple groups" means two or more (including two groups).

[0032] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0033] Figure 1 This is a flowchart illustrating a structured mapping method for multimodal medical data provided in this disclosure, as shown in the embodiment. Figure 1 As shown, the specific process of the structured mapping method for multimodal medical data includes:

[0034] S110. Acquire multimodal medical data, which includes: first multi-source data and / or second multi-source data.

[0035] Multimodal medical data can originate from raw CT (Computed Tomography) report data, such as unstructured text / PDF / image reports from hospital PACS (Picture Archiving and Communication System) systems and physical examination institutions, covering 24 key parameters; medical terminology standard libraries, such as ICD-11 (tumor morphology coding), RadLex (radiological terminology), LOINC (examination item coding), etc.; expert-annotated datasets, such as standardized report samples annotated by radiologists, containing terminology mapping relationships (e.g., "mixed ground-glass nodules → mixed nodules"); publicly available medical literature, such as the terminology and parameter relationships defined in lung cancer diagnosis guidelines and imaging diagnosis consensus (e.g., lung nodule classification standards); and historical structured data, such as partially structured hospital electronic medical record (EMR) data, which can be used to verify the effectiveness of mapping rules.

[0036] The first multi-source data can be used to describe unstructured PDF data and unstructured image data, such as CT PDF reports and CT image reports; the second multi-source data can be used to describe unstructured text data, such as CT text reports.

[0037] S120. For the first multi-source data in the multimodal medical data, a region localization and character recognition strategy is used to identify the corresponding key fields from the first multi-source data. For the second multi-source data in the multimodal medical data, a regular expression matching and model recognition strategy is used to identify the corresponding key parameters from the second multi-source data, so as to extract features from the multimodal medical data and obtain multimodal feature data.

[0038] Among them, the region positioning and character recognition strategies can be used to locate fixed regions and identify data within those regions. The regular expression matching and model recognition strategies can be used to identify data using regular expressions and to identify information related to the data using models.

[0039] In this embodiment, for the first multi-source data in multimodal medical data, a region localization and character recognition strategy is used to identify the corresponding key fields from the first multi-source data, including:

[0040] A preset layout recognition model is used to locate key fields in the first multi-source data of multimodal medical data to obtain the field location area; optical character recognition is then performed on the field location area in the multimodal medical data to identify the corresponding key fields from the first multi-source data.

[0041] Among them, regular expression matching and model recognition strategies are a type of version analysis algorithm, such as a CNN-based document layout recognition model. This model uses a preset layout recognition model to locate key fields in the first multi-source data of multimodal medical data, resulting in field location regions such as the "patient name" region in a CT image report. Optical Character Recognition (OCR) technology is then used to perform optical character recognition on the field location regions in the multimodal medical data to identify the corresponding key fields from the first multi-source data, such as identifying the "patient name" from the "patient name" region in a CT image report.

[0042] The key fields identified from the first multi-source data are a type of semi-structured ISON data with location tags, as shown in the example below:

[0043]

[0044] Therefore, for unstructured PDF data and unstructured image data, the corresponding key fields can be effectively identified through region positioning and data recognition methods.

[0045] In this embodiment, for the second multi-source data in multimodal medical data, a regular matching and model recognition strategy is used to identify corresponding key parameters from the second multi-source data, including:

[0046] The target data is extracted from the second multi-source data in the multimodal medical data using a target regular expression; the key parameters corresponding to the target data are identified from the second multi-source data in the multimodal medical data using a preset parameter identification model.

[0047] Among them, the preset parameter recognition model is such as the BiLSTM-CRF model. The target data extracted from the second multi-source data in the multimodal medical data using the target regular expression can be such as "mixed ground-glass nodules seen in the upper lobe of the right lung". The key parameters corresponding to the target data identified from the second multi-source data in the multimodal medical data through the preset parameter recognition model can be such as "approximately 8mm × 6mm in size" and "coordinates".

[0048] The key fields identified from the second multi-source data are a type of semi-structured ISON data with location tags, as shown in the example below:

[0049]

[0050] Therefore, for unstructured text data, the corresponding key fields can be effectively identified through regular expressions and model recognition methods.

[0051] This embodiment uses OCR and layout analysis to locate field regions, and then uses the NER model to extract semantic entities, solving the problem of "cross-template recognition of the same parameter in different locations". In addition, it can also perform cross-paragraph semantic splicing for scattered descriptions (such as "edge burrs, visible lobes" scattered in different paragraphs) to generate complete parameters.

[0052] S130. Perform minimum semantic unit decomposition on the multimodal feature data to obtain multimodal unit data.

[0053] Among them, the smallest semantic unit is decomposed into a term atomization, which can break down a complete sentence into a combination of multiple words to achieve semantic reduction and improve the accuracy of text expression.

[0054] In this embodiment, the multimodal feature data is split into minimum semantic units to obtain multimodal unit data, including:

[0055] The multimodal feature data is semantically categorized to obtain multiple data semantic categories corresponding to the multimodal feature data. Taking each data semantic category as the smallest semantic unit, the data content corresponding to each data semantic category is decomposed from the multimodal feature data, and the data content corresponding to each data semantic category is coded to its category to obtain multimodal unit data.

[0056] For example, the multimodal feature data is "mixed ground-glass nodule in the upper lobe of the right lung, with a size of 8mm × 6mm". Semantic classification is performed to obtain multiple semantic categories corresponding to the multimodal feature data, namely location category, type category, and size category. Using each semantic category as the smallest semantic unit, the data content corresponding to each semantic category is decomposed from the multimodal feature data. For example, the data content corresponding to the location category is "upper lobe of the right lung", the data content corresponding to the type category is "mixed ground-glass nodule", and the data content corresponding to the size category is "8mm × 6mm". The data content corresponding to each semantic category is then coded to its corresponding category to obtain multimodal unit data. For example, location: "upper lobe of the right lung" → anatomical location code (SEGMENT_CODE:RB1), type: "mixed ground-glass nodule" → standard type code (NODULE_TYPE:MIXED), size: "8mm × 6mm" → major axis / minor axis value (LONG_AXIS:8, SHORT_AXIS:6).

[0057] S140. Multi-level mapping is performed on the multi-modal unit data based on the multi-level mapping engine to obtain multi-modal mapping data.

[0058] The multi-level mapping engine includes a rule mapping engine, a semantic mapping engine, and a statistical mapping engine. The rule mapping engine is used to map data using regular expressions; the semantic mapping engine is used to map data using knowledge graphs; and the statistical mapping engine is used to map data using historical data.

[0059] In this embodiment, multi-level mapping is performed on the multi-modal unit data based on a multi-level mapping engine to obtain multi-modal mapping data, including:

[0060] A rule mapping engine is used to perform regular expression mapping on the corresponding unit data in the multimodal unit data; a semantic mapping engine is used to perform term expansion on the corresponding unit data in the multimodal unit data to perform data mapping; and a statistical mapping engine is used to perform high-frequency word substitution description on the corresponding unit data in the multimodal unit data to perform data mapping.

[0061] Specifically, a rule mapping engine is used to perform regular expression mapping on the corresponding unit data in the multimodal unit data, such as predefined regular expression rules (e.g., "ground-glass" → "ground-glass nodule"); a semantic mapping engine is used to perform term expansion on the corresponding unit data in the multimodal unit data, such as term expansion based on medical knowledge graphs (e.g., the inclusion relationship between "partial solid nodule" and "mixed nodule"); and a statistical mapping engine is used to perform high-frequency word substitution description on the corresponding unit data in the multimodal unit data, such as statistically analyzing high-frequency substitution words through historical data (e.g., 80% of "mixed nodule" is described as "partial ground-glass nodule").

[0062] Thus, by combining rule mapping (regular expression matching), semantic mapping (knowledge graph expansion), and statistical mapping (high-frequency word analysis of historical data), a precise conversion from "text fragments → semantic units → standard encoding" can be achieved. Furthermore, this embodiment can adapt to the continuous evolution of medical terminology through dynamic weight adjustments (e.g., statistical mapping weight = historical frequency × semantic similarity).

[0063] In this embodiment, it also includes:

[0064] If there are at least two mapping data corresponding to the target medical data in the multimodal unit data, then the target medical data and the corresponding at least two mapping data are sent to the review node so that the review node can review the target medical data and the corresponding at least two mapping data; the target mapping data corresponding to the target medical data sent by the review node is received, and the data mapping result of the target medical data is updated based on the target mapping data corresponding to the target medical data.

[0065] When multiple mapping results exist for the same parameter (such as "mixed nodules" matching two codes simultaneously), an expert review process is triggered to ensure the accuracy of the target medical data mapping results.

[0066] In addition, when there are multiple candidate mapping results for the same parameter, the optimal solution can be dynamically selected through contextual features (such as the combined probability of nodule location and signs) to obtain the mapping result of the target medical data.

[0067] S150. Perform data mapping detection on the multimodal mapping data, and perform context association on the target unit data in the multimodal mapping data that has not been mapped, to obtain the medical structured data corresponding to the multimodal medical data.

[0068] Specifically, by constructing a term co-occurrence network, the correlation between parameters is analyzed, so as to associate the mapping results of target unit data that have not been mapped in the context.

[0069] In this embodiment, context association is performed on the target unit data that has not been mapped in the multimodal mapping data to obtain the medical structured data corresponding to the multimodal medical data, including:

[0070] A term co-occurrence network is constructed, which includes multiple co-occurrence node groups. Each co-occurrence node group contains two medical data points, and each co-occurrence node group has a corresponding co-occurrence weight. From the term co-occurrence network, co-occurrence node groups that are associated with target unit data that has not been mapped in the multimodal mapping data are identified. The opposing medical data in the co-occurrence node groups that are associated with the target unit data are determined as the target mapping data corresponding to the target unit data, thereby determining the medical structured data corresponding to the multimodal medical data.

[0071] Specifically, if at least two co-occurrence node groups include target unit data, the co-occurrence node group with the largest co-occurrence weight is determined to be the co-occurrence node group associated with the target unit data; if one co-occurrence node group includes target unit data, that co-occurrence node group is determined to be the co-occurrence node group associated with the target unit data. For example, in multimodal mapping data, the target unit data that has not been mapped is "semi-real nodule". There are at least two co-occurrence node groups that include target unit data, namely co-occurrence node group 1: "semi-real nodule → mixed nodule" and co-occurrence node group 2: "semi-real nodule → xx nodule". The co-occurrence weight of co-occurrence node group 1 is greater than the co-occurrence weight of co-occurrence node group 2. Therefore, "mixed nodule" is determined to be the target mapping data of the target unit data "semi-real nodule".

[0072] In addition, this embodiment can also be used for context-related recommendations of new terms by constructing a co-occurrence probability matrix between parameters (such as the co-occurrence weights of "spiculation sign" and "lobulation sign").

[0073] In this embodiment, multimodal medical data is acquired, including: first multi-source data and / or second multi-source data. The first multi-source data describes unstructured PDF data and unstructured image data, while the second multi-source data describes unstructured text data. For the first multi-source data, a region localization and character recognition strategy is used to identify corresponding key fields. For the second multi-source data, a regular expression matching and model recognition strategy is used to identify corresponding key parameters, thereby extracting features from the multimodal medical data to obtain multimodal feature data. Multimodal unit data is obtained by splitting the feature data into its smallest semantic units. Multimodal unit data is then mapped at multiple levels using a multi-level mapping engine, resulting in multimodal mapped data. This multi-level mapping engine includes a rule-based mapping engine, a semantic mapping engine, and a statistical mapping engine. The rule-based mapping engine uses regular rules for data mapping, the semantic mapping engine uses knowledge graphs, and the statistical mapping engine uses historical data. Data mapping detection is then performed on the multimodal mapped data, and contextual association is established for target unit data that has not yet been mapped, resulting in structured medical data corresponding to the multimodal medical data. Thus, by extracting features from different multi-source data, splitting them into their smallest semantic units, and mapping them, and by associating unmapped unit data with context, data mapping efficiency is effectively improved.

[0074] This embodiment also provides a specific example analysis, where the multimodal medical data is a CT report (PNG image format) from a hospital, as follows:

[0075]

[0076] By locating fields in multimodal medical data and identifying the "Patient Information" and "Image Description" regions through layout analysis, the parameter extraction results are as follows: "Name: XXX" is extracted using regular expression matching and mapped to `Patient Name`; "Mixed Ground-Glass Nodule" is semantically parsed and mapped to `Nodule Nature: Mixed Nodule` (standard encoding `NODULE_TYPE:MIXED`); "12mm × 9mm" is parsed as `Lung Nodule Long Axis: 12` and `Lung Nodule Short Axis: 9` (units standardized to mm); "Spiculation and Lobeculation Sign" is split into `Spiculation Sign: Positive` and `Lobeculation Sign: Positive`; Contextual completion: `Pleural Sign: Positive` is added based on "Adjacent Pleural Retraction". The resulting structured medical data is shown in the example below.

[0077]

[0078]

[0079] Furthermore, this embodiment uses a hierarchical storage structure to store different data in layers, supporting efficient querying and version management. For example, the basic mapping rule table is stored in the first data layer, with the following data structure: field name: original expression (regular expression), patient_name: `^(name), nodule_type: `(ground glass); the terminology extension graph is stored in the second data layer, with the following data structure: node: standard terminology; edge: semantic relationship (synonym, hyponym, mutually exclusive), such as "mixed nodules" -- (synonym) → "partial solid nodules"; the context feature library is stored in the third data layer, with the following data structure: parameter combination mode (such as "spiculated sign + lobulated sign → high risk"), stored as <Feature vector, risk weight>, such as "[HAS_SPICULATION:1,HAS_LOBULATION:1]→RISK_WEIGHT+0.3"; Version management logs are stored in the fourth data layer, and the data structure is: record the effective time, coverage of institutions, and update reason of the term mapping rule (such as "Add XX hospital template"), such as "{date:2024-08-01,hospital_id:H0382,rule_id:R_CT_05,action:ADD}".

[0080] Figure 2 This is a schematic diagram of a structured mapping device for multimodal medical data provided in this embodiment. The structured mapping device for multimodal medical data may include: an acquisition module 210, an identification module 220, a splitting module 230, a mapping module 240, and a determination module 250.

[0081] The acquisition module 210 is used to acquire multimodal medical data, which includes: first multi-source data and / or second multi-source data. The first multi-source data is used to describe unstructured PDF data and unstructured image data, and the second multi-source data is used to describe unstructured text data.

[0082] The recognition module 220 is used to identify corresponding key fields from the first multi-source data in the multimodal medical data using a region positioning and character recognition strategy, and to identify corresponding key parameters from the second multi-source data in the multimodal medical data using a regular expression matching and model recognition strategy, so as to extract features from the multimodal medical data and obtain multimodal feature data.

[0083] The splitting module 230 is used to split the multimodal feature data into minimum semantic units to obtain multimodal unit data.

[0084] The mapping module 240 is used to perform multi-level mapping on multimodal unit data based on a multi-level mapping engine to obtain multimodal mapped data. The multi-level mapping engine includes a rule mapping engine, a semantic mapping engine, and a statistical mapping engine. The rule mapping engine is used to map data using regular rules, the semantic mapping engine is used to map data using knowledge graphs, and the statistical mapping engine is used to map data using historical data.

[0085] The determination module 250 is used to perform data mapping detection on the multimodal mapping data and to perform context association on the target unit data that has not been mapped in the multimodal mapping data to obtain the medical structured data corresponding to the multimodal medical data.

[0086] In this embodiment, optionally, the identification module 220 is specifically used for:

[0087] A preset layout recognition model is used to locate key fields in the first multi-source data of multimodal medical data to obtain the field location area; optical character recognition is then performed on the field location area in the multimodal medical data to identify the corresponding key fields from the first multi-source data.

[0088] In this embodiment, optionally, the identification module 220 is specifically used for:

[0089] The target data is extracted from the second multi-source data in the multimodal medical data using a target regular expression; the key parameters corresponding to the target data are identified from the second multi-source data in the multimodal medical data using a preset parameter identification model.

[0090] In this embodiment, optionally, the split module 230 is specifically used for:

[0091] The multimodal feature data is semantically categorized to obtain multiple data semantic categories corresponding to the multimodal feature data. Taking each data semantic category as the smallest semantic unit, the data content corresponding to each data semantic category is decomposed from the multimodal feature data, and the data content corresponding to each data semantic category is coded to its category to obtain multimodal unit data.

[0092] In this embodiment, optionally, the mapping module 240 is specifically used for:

[0093] A rule mapping engine is used to perform regular expression mapping on the corresponding unit data in the multimodal unit data; a semantic mapping engine is used to perform term expansion on the corresponding unit data in the multimodal unit data to perform data mapping; and a statistical mapping engine is used to perform high-frequency word substitution description on the corresponding unit data in the multimodal unit data to perform data mapping.

[0094] In this embodiment, optionally, the determining module 250 is specifically used for:

[0095] A term co-occurrence network is constructed, comprising multiple co-occurrence node groups. Each co-occurrence node group includes two medical data sets, and each co-occurrence node group has a corresponding co-occurrence weight. From the term co-occurrence network, co-occurrence node groups that are associated with target unit data in the multimodal mapping data that has not yet been mapped are identified. The opposing medical data in the co-occurrence node groups associated with the target unit data are determined as the target mapping data corresponding to the target unit data, thereby determining the medical structured data corresponding to the multimodal medical data. Specifically, if at least two co-occurrence node groups include target unit data, the co-occurrence node group with the largest co-occurrence weight is determined as the co-occurrence node group associated with the target unit data. If one co-occurrence node group includes target unit data, that co-occurrence node group is determined as the co-occurrence node group associated with the target unit data.

[0096] In this embodiment, optionally, it also includes a sending module, a receiving module, and an updating module.

[0097] The sending module is used to send the target medical data and the corresponding at least two mapping data to the review node if there are target medical data in the multimodal unit data, so that the review node can review the target medical data and the corresponding at least two mapping data.

[0098] The receiving module is used to receive the target mapping data corresponding to the target medical data sent by the review node.

[0099] The update module is used to update the data mapping result of the target medical data based on the target mapping data corresponding to the target medical data.

[0100] The structured mapping apparatus for multimodal medical data provided in this disclosure can execute the above-described method embodiments. For its specific implementation principle and technical effects, please refer to the above-described method embodiments, which will not be repeated here.

[0101] This application also provides a computer device. Please refer to the following for details. Figure 3 , Figure 3 This is a basic structural block diagram of the computer device in this embodiment.

[0102] The computer device includes a memory 310 and a processor 320 that are communicatively connected to each other via a system bus. It should be noted that only a computer device with memory 310 and processor 320 is shown in the figure; however, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented alternatively. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0103] Computer devices can include desktop computers, laptops, handheld computers, and cloud servers. These devices allow for human-computer interaction with users through keyboards, mice, remote controls, touchpads, or voice-activated devices.

[0104] The memory 310 includes at least one type of readable storage medium, including non-volatile memory or volatile memory, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. RAM may include static RAM or dynamic RAM. In some embodiments, the memory 310 may be an internal storage unit of a computer device, such as the hard disk or RAM of the computer device. In other embodiments, the memory 310 may also be an external storage device of the computer device, such as a plug-in hard drive, smart media card (SMC), secure digital (SD) card, or flash card equipped on the computer device. Of course, the memory 310 may include both internal storage units and external storage devices of the computer device. In this embodiment, the memory 310 is typically used to store the operating system and various application software installed on the computer device, such as the program code of the methods described above. Furthermore, the memory 310 may also be used to temporarily store various types of data that have been output or will be output.

[0105] Processor 320 is typically used to perform overall operations of a computer device. In this embodiment, memory 310 is used to store program code or instructions, including computer operation instructions, and processor 320 is used to execute the program code or instructions stored in memory 310 or process data, such as program code that runs the methods described above.

[0106] In this article, the bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus system can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0107] Another embodiment of this application also provides a computer-readable medium, which may be a computer-readable signal medium or a computer-readable medium. A processor in a computer reads computer-readable program code stored in the computer-readable medium, enabling the processor to execute the functional actions specified in each step or combination of steps in the above method; and to generate means for implementing the functional actions specified in each block or combination of blocks in the block diagram.

[0108] Computer-readable media include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared memory or semiconductor systems, devices or apparatuses, or any suitable combination thereof, wherein the memory is used to store program code or instructions, the program code including computer operation instructions, and the processor is used to execute the program code or instructions of the above-described methods stored in the memory.

[0109] The definitions of memory and processor can be found in the description of the foregoing computer device embodiments, and will not be repeated here.

[0110] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0111] In the various embodiments of this application, the functional units or modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0112] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" as described in this application does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims listing several means, several units of these means may be embodied by the same item of hardware. The use of "first," "second," and "third," etc., does not indicate any order and these words should be interpreted as names. Unless otherwise specified, the steps in the above embodiments should not be construed as limiting the order of execution.

[0114] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method of structured mapping of multi-modal medical data, characterized in that, The method comprises the following steps: acquiring multi-modal medical data, wherein the multi-modal medical data comprises: first multi-source data and / or second multi-source data, the first multi-source data is used for describing unstructured PDF data and unstructured image data, and the second multi-source data is used for describing unstructured text data; for the first multi-source data in the multi-modal medical data, a region positioning and character recognition strategy is adopted to identify corresponding key fields from the first multi-source data, and for the second multi-source data in the multi-modal medical data, a regular matching and model recognition strategy is adopted to identify corresponding key parameters from the second multi-source data, so as to extract features of the multi-modal medical data and obtain multi-modal feature data; performing minimum semantic unit splitting on the multi-modal feature data to obtain multi-modal unit data; performing multi-level mapping on the multi-modal unit data based on a multi-level mapping engine to obtain multi-modal mapping data, wherein the multi-level mapping engine comprises: a rule mapping engine, a semantic mapping engine and a statistical mapping engine, the rule mapping engine is used for data mapping by adopting a regular rule, the semantic mapping engine is used for data mapping by adopting a knowledge graph, and the statistical mapping engine is used for data mapping by adopting historical data; the multi-level mapping on the multi-modal unit data based on the multi-level mapping engine to obtain the multi-modal mapping data comprises: regular mapping of corresponding unit data in the multi-modal unit data by adopting the rule mapping engine; data mapping of corresponding unit data in the multi-modal unit data by adopting the semantic mapping engine to perform term expansion on the corresponding unit data; and data mapping of corresponding unit data in the multi-modal unit data by adopting the statistical mapping engine to perform high-frequency word replacement description on the corresponding unit data; performing data mapping detection on the multi-modal mapping data, and performing context association on target unit data in the multi-modal mapping data which has not been mapped, to obtain medical structured data corresponding to the multi-modal medical data.

2. The method of claim 1, wherein, The method comprises the following steps: adopting a preset layout recognition model to position key fields in the first multi-source data in the multi-modal medical data to obtain a field positioning region; performing optical character recognition on the field positioning region in the multi-modal medical data to identify corresponding key fields from the first multi-source data.

3. The method of claim 1, wherein, The method comprises the following steps: adopting a target regular expression to extract target data from the second multi-source data in the multi-modal medical data; adopting a preset parameter recognition model to identify corresponding key parameters of the target data from the second multi-source data in the multi-modal medical data to identify corresponding key parameters from the second multi-source data.

4. The method of claim 1, wherein, The minimum semantic unit splitting of the multi-modal feature data comprises: performing semantic category division on the multi-modal feature data to obtain a plurality of data semantic categories corresponding to the multi-modal feature data; each data semantic category is taken as a minimum semantic unit, data content corresponding to each data semantic category is decomposed from the multi-modal feature data, and category coding is performed on the data content corresponding to each data semantic category to obtain multi-modal unit data.

5. The method of claim 1, wherein, The context association of the target unit data with incomplete mapping in the multi-modal mapping data comprises: a term co-occurrence network is constructed, the term co-occurrence network comprises a plurality of co-occurrence node groups, each co-occurrence node group comprises two medical data, and each co-occurrence node group corresponds to a co-occurrence weight; from the term co-occurrence network, a co-occurrence node group having an association relationship with the target unit data with incomplete mapping in the multi-modal mapping data is matched, and the opposite medical data in the co-occurrence node group having the association relationship with the target unit data is determined as target mapping data corresponding to the target unit data, so as to determine medical structured data corresponding to the multi-modal medical data; wherein, if there are at least two co-occurrence node groups comprising the target unit data, the co-occurrence node group corresponding to the maximum co-occurrence weight is determined as the co-occurrence node group having the association relationship with the target unit data; if there is one co-occurrence node group comprising the target unit data, the one co-occurrence node group is determined as the co-occurrence node group having the association relationship with the target unit data.

6. The method of claim 1, wherein, Further comprising: if there are at least two mapping data corresponding to a target medical data in the multi-modal unit data, the target medical data and the at least two mapping data corresponding thereto are sent to an audit node, so that the audit node audits the target medical data and the at least two mapping data corresponding thereto; the target mapping data corresponding to the target medical data sent by the audit node is received, and the data mapping result of the target medical data is updated based on the target mapping data corresponding to the target medical data.

7. An apparatus for structured mapping of multi-modal medical data, characterized by, Comprise: an acquisition module configured to acquire multi-modal medical data, the multi-modal medical data comprising: first multi-source data and / or second multi-source data, the first multi-source data being used to describe unstructured PDF data and unstructured image data, and the second multi-source data being used to describe unstructured text data; an identification module configured to identify corresponding key fields from the first multi-source data by using a region positioning and character recognition strategy, and identify corresponding key parameters from the second multi-source data by using a regular matching and model recognition strategy, so as to perform feature extraction on the multi-modal medical data and obtain multi-modal feature data; a splitting module configured to perform minimum semantic unit splitting on the multi-modal feature data to obtain multi-modal unit data; The mapping module is configured to perform multi-level mapping on the multi-modal unit data based on a multi-level mapping engine to obtain multi-modal mapping data. The multi-level mapping engine includes a rule mapping engine, a semantic mapping engine, and a statistical mapping engine. The rule mapping engine is configured to perform data mapping by using regular rules. The semantic mapping engine is configured to perform data mapping by using a knowledge graph. The statistical mapping engine is configured to perform data mapping by using historical data. The mapping module is specifically configured to perform regular mapping on corresponding unit data in the multi-modal unit data by using the rule mapping engine. The mapping module is specifically configured to perform term expansion on corresponding unit data in the multi-modal unit data by using the semantic mapping engine to perform data mapping on the corresponding unit data in the multi-modal unit data. The mapping module is specifically configured to perform high-frequency word replacement description on corresponding unit data in the multi-modal unit data by using the statistical mapping engine to perform data mapping on the corresponding unit data in the multi-modal unit data. The determining module is configured to perform data mapping detection on the multi-modal mapping data and context association on target unit data in the multi-modal mapping data that has not completed mapping to obtain medical structured data corresponding to the multi-modal medical data.

8. A computer device, comprising: The computer program is executed by the processor to implement the structured mapping method of the multi-modal medical data according to any one of claims 1-6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the structured mapping method of the multi-modal medical data according to any one of claims 1-6.

Images