Multi-modal medical health data index construction method and system

Abstract

The invention discloses an index construction method and system for multi-modal medical and health data, and relates to the technical field of medical and health, and the method comprises the steps: obtaining the multi-modal medical and health data containing at least one diagnosis and treatment activity of a single patient; performing semantic alignment in the multi-modal medical and health data of the single diagnosis and treatment activity, and after semantic alignment, based on medical domain knowledge, performing semantic association among different modalities of medical and health data of the multi-modal medical and health data of the single diagnosis and treatment activity to obtain a single diagnosis and treatment activity association map; based on medical field knowledge, setting a label for the single-time diagnosis and treatment activity association map, constructing a plurality of first-level indexes of the single-time diagnosis and treatment activity association map based on the label, and vectorizing the plurality of first-level indexes to obtain a plurality of first-level index vectors; a plurality of primary index vectors are stored in a vector label library, and a single diagnosis and treatment activity association map is stored in a map library, so that the constructed index can realize cross-modal data query.

Description

Technical Field

[0001] The present application relates to the field of medical and health technology, and in particular to a method and system for constructing an index of multimodal medical and health data. Background Art

[0002] The data modalities in the healthcare field are diverse, and traditional data indexes, such as the Enterprise Master Patient Index (EMPI), usually use patient or medical record IDs as index tags, which to some extent solves the problem of data organization. However, these methods mainly focus on the identifier level of data, and do not go deep into the intrinsic meaning or semantic level of the data, making it difficult to build data assets that can be efficiently used by artificial intelligence and big data analysis.

[0003] At present, the data standardization and specification construction in the medical and health industry has achieved remarkable results. Health regulatory departments at all levels have established or plan to establish medical and health data centers. On the one hand, they hope to support the needs of remote diagnosis and treatment and tiered diagnosis and treatment for cross-hospital access to medical and health data; on the other hand, they also hope to rely on artificial intelligence and big data analysis technologies to carry out data-centric scientific research activities. For the former scenario, EMPI technology can better solve the need to index the patient's medical records according to their identity information. However, for the latter scenario, technologies such as EMPI have major problems, which are mainly reflected in:

[0004] When performing data queries, you cannot simply search for data that meets your needs through keywords. For example, when studying "acute mountain sickness", you need to query the symptoms, radiological images, physical and chemical test indicators associated with "acute mountain sickness" separately to ensure the coverage of the data set. Although the retrieval of single-modality data can be associated with related items through technologies such as knowledge graphs, when faced with multi-modal data, existing technical means are stretched to the limit and it is difficult to achieve cross-modal queries. Summary of the invention

[0005] In view of this, the present application provides a method and system for constructing an index of multimodal medical and health data, aiming to realize cross-modal query of medical and health data.

[0006] The first aspect of the present application provides a method for constructing an index of multimodal medical and health data, the method comprising:

[0007] Obtain multimodal medical and health data containing at least one diagnosis and treatment activity of a single patient;

[0008] In the multimodal medical and health data of a single diagnosis and treatment activity, semantic alignment is performed between medical and health data of different modalities, and after the semantic alignment, based on the pre-configured medical field knowledge, semantic association is performed between the multimodal medical and health data of a single diagnosis and treatment activity, so as to obtain a single diagnosis and treatment activity association map;

[0009] Based on medical field knowledge, labels are set for the association map of a single diagnosis and treatment activity, multiple first-level indexes of the association map of a single diagnosis and treatment activity are constructed based on the labels, and multiple first-level indexes are vectorized to obtain multiple first-level index vectors;

[0010] Multiple first-level index vectors are stored in the vector label library, and the associated maps of a single diagnosis and treatment activity are stored in the map library.

[0011] In some implementations of the first aspect of the present application, the method further includes:

[0012] For multimodal medical and health data of a single diagnosis and treatment activity, the medical and health data of different modalities are correspondingly stored in multiple data storage pools, wherein each data storage pool is used to store medical and health data of one modality;

[0013] A single diagnosis and treatment activity data view associated with the patient is established. The single diagnosis and treatment activity data view is used to obtain multimodal medical and health data of a single diagnosis and treatment activity from multiple data stores.

[0014] In some implementations of the first aspect of the present application, the method further includes:

[0015] According to the patient's diagnosis and treatment activity time, the data views of each single diagnosis and treatment activity corresponding to the patient are associated to obtain the diagnosis and treatment activity data view time series.

[0016] In some implementations of the first aspect of the present application, the method further includes:

[0017] Generate a first-level query index based on the data retrieval request sent by the user, and vectorize the first-level query index to obtain a first-level query index vector;

[0018] According to the first-level query index vector, a target index vector is determined from multiple first-level index vectors in the vector tag library;

[0019] Get the target data based on the target index vector.

[0020] In some implementations of the first aspect of the present application, the method further includes:

[0021] When at least two primary query indexes are generated based on the data retrieval request, combining the at least two primary query indexes to obtain a second query index, and vectorizing the secondary query index to obtain a secondary query index vector;

[0022] According to the secondary query index vector, a target index vector is determined in the vector tag library.

[0023] In some implementations of the first aspect of the present application, acquiring target data based on the target index vector includes:

[0024] determining a target primary index associated with the target index vector;

[0025] Determine the target diagnosis and treatment activity association map associated with the target primary index;

[0026] Determine a target diagnosis and treatment activity data view associated with the target diagnosis and treatment activity association map;

[0027] Obtain target data based on target diagnosis and treatment activity data view.

[0028] In some implementations of the first aspect of the present application, the method further includes:

[0029] De-anonymize the target data.

[0030] In some implementations of the first aspect of the present application, the method further includes:

[0031] Create a primary index vector library and a secondary index vector library. The primary index vector library is used to cache the primary index vectors extracted from the vector label library, and the secondary index vector library is used to cache the secondary query index vectors generated based on the data retrieval request.

[0032] In some implementations of the first aspect of the present application, the method further includes:

[0033] The secondary index vector library periodically cleans up low-frequency secondary query index vectors.

[0034] The second aspect of the present application provides an index construction system for multimodal medical and health data, the system comprising:

[0035] A data acquisition module, used to acquire multimodal medical and health data including at least one diagnosis and treatment activity of a single patient;

[0036] A graph generation module is used to perform semantic alignment between different modalities of medical and health data in the multimodal medical and health data of a single diagnosis and treatment activity, and after the semantic alignment, perform semantic association between different modalities of the multimodal medical and health data of a single diagnosis and treatment activity based on pre-configured medical field knowledge to obtain a single diagnosis and treatment activity association graph;

[0037] An index building module is used to set labels for a single diagnosis and treatment activity association map based on medical field knowledge, build multiple first-level indexes of the single diagnosis and treatment activity association map based on the labels, and vectorize the multiple first-level indexes to obtain multiple first-level index vectors;

[0038] The data storage module is used to store multiple first-level index vectors in the vector label library and store the associated maps of a single diagnosis and treatment activity in the map library.

[0039] The technical solution provided by this application has the following beneficial effects:

[0040] In the technical solution provided in the present application, first, multimodal medical and health data containing at least one diagnosis and treatment activity of a single patient is obtained. Then, in the multimodal medical and health data of a single diagnosis and treatment activity, semantic inconsistency between different modalities is solved by performing semantic alignment, so that the relevant data of different modalities can be associated together. After completing the semantic alignment, the medical field knowledge is used to perform semantic associations between medical and health data of different modalities to obtain a single diagnosis and treatment activity association map, which associates the corresponding medical and health data of different modalities and belonging to the same diagnosis and treatment activity. Afterwards, a label is set for the single diagnosis and treatment activity association map based on the medical field knowledge, and multiple first-level indexes of the single diagnosis and treatment activity association map are constructed based on the label, and multiple first-level indexes are vectorized to obtain multiple first-level index vectors, and these first-level index vectors are stored in the vector label library, and the single diagnosis and treatment activity association map is stored in the map library to form a complete retrieval system. These first-level index vectors vectorize the semantic information of the association map of a single diagnosis and treatment activity, so that the system can perform data retrieval through vector matching; since the data of different modalities have formed a unified index and label through semantic alignment and association maps, the system can realize cross-modal query. For example, if the user enters "acute altitude sickness" as a keyword, the system will not only return relevant text records, but also associated imaging data and detection indicators. It can be seen that this application realizes cross-modal query of medical and health data, which is not only limited to a single modality, but also can associate data of multiple modalities, ensuring the comprehensiveness of the query. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of a method for constructing an index of multimodal medical and health data provided in an embodiment of the present application;

[0042] Figure 2 A flowchart of another method for constructing an index of multimodal medical and health data provided in an embodiment of the present application;

[0043] Figure 3 A flowchart of another method for constructing an index of multimodal medical and health data provided in an embodiment of the present application;

[0044] Figure 4 A flowchart of another method for constructing an index of multimodal medical and health data provided in an embodiment of the present application;

[0045] Figure 5 A network topology diagram of a cluster computing environment provided in an embodiment of the present application;

[0046] Figure 6 A diagram of the software operation architecture provided for the embodiment of the present application;

[0047] Figure 7 A schematic diagram of a system for constructing an index for multimodal medical and health data provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0049] See also Figure 1 As shown, in order to realize cross-modal query of multimodal medical and health data, an embodiment of the present application provides an index construction method for multimodal medical and health data, which may specifically include the following steps:

[0050] S101: Acquire multimodal medical and health data including at least one diagnosis and treatment activity of a single patient.

[0051] In the embodiments of the present application, multimodal medical and health data refers to a collection of medical data from different data sources and types. Diagnosis and treatment activities refer to a series of medical behaviors within a medical institution, including but not limited to consultation, examination, treatment and other links. It is understandable that for patients, a single diagnosis and treatment activity will generate medical and health data of multiple modalities, such as electronic medical records, medical images, patient complaint data, physical and chemical test data, etc. Therefore, these diverse data sets generated by a series of medical links in a single diagnosis and treatment activity can be referred to as multimodal medical and health data of a single or single diagnosis and treatment activity.

[0052] S102: Perform semantic alignment between medical and health data of different modalities in the multimodal medical and health data of a single diagnosis and treatment activity, and after the semantic alignment, perform semantic association between medical and health data of different modalities on the multimodal medical and health data of a single diagnosis and treatment activity based on pre-configured medical field knowledge to obtain a single diagnosis and treatment activity association map.

[0053] In the embodiments of the present application, semantic alignment refers to matching data of different modalities to make them consistent at the semantic level. For example, the lesion area found in the medical image is semantically matched with the diagnosis description recorded in the electronic medical record to ensure that these different data actually refer to the same condition. Semantic association is based on semantic alignment and is based on the professional knowledge base in the medical field to build the relationship between different data, such as associating the doctor's diagnosis with the patient's symptoms, and associating the patient's symptoms with the patient's chief complaint.

[0054] Specifically, for the multimodal medical and health data of a single diagnosis and treatment activity, semantic alignment is first performed to unify the concepts in the medical data of different modalities into the same semantic representation, establish commonalities at the semantic level between the multimodal data, and ensure that the concepts of all data can be understood at the same semantic level, so as to eliminate the differences in semantic understanding caused by different data modalities. Subsequently, based on the pre-configured medical field knowledge, the medical and health data of different modalities after semantic alignment are mutually associated to form a knowledge graph that reflects the association relationship between the medical and health data of each modality in the diagnosis and treatment activity, that is, the association graph of a single diagnosis and treatment activity.

[0055] The embodiments of the present application can achieve effective unification of multimodal data and eliminate data silos by implementing the processes of semantic alignment and semantic association. This can solve the problem of difficulty in associating medical and health data of different modalities, thereby improving the utilization value of the data and providing a basis for subsequent data queries.

[0056] S103: Based on medical field knowledge, set labels for the single diagnosis and treatment activity association map, construct multiple first-level indexes of the single diagnosis and treatment activity association map based on the labels, and vectorize the multiple first-level indexes to obtain multiple first-level index vectors.

[0057] In an embodiment of the present application, in order to realize the retrieval of multimodal medical and health data, corresponding labels are set for the association map of a single diagnosis and treatment activity based on medical field knowledge. These labels are used to represent the keywords or conceptual entities of this diagnosis and treatment activity. Afterwards, multiple first-level indexes are generated based on these labels, and multiple first-level index vectors are obtained by vectorization. Among them, the first-level index refers to the basic index generated based on each label, covering a variety of possible classification methods. For example, an index can be established according to the type of disease in the diagnosis conclusion, or it can be indexed according to the typical symptoms that appear or the prescribed drugs. Each first-level index vector represents the data characteristics of a specific aspect of a diagnosis and treatment activity, so that in the subsequent data retrieval process, data can be quickly found by vector matching.

[0058] S104: storing multiple first-level index vectors into a vector label library, and storing a single diagnosis and treatment activity associated map into a map library.

[0059] The vector tag library is used to store the first-level index vectors so that subsequent retrieval operations can be efficiently performed through similarity matching in the vector space. The graph library is used to store the association graph of single diagnosis and treatment activities formed after the semantic association of multimodal medical and health data, which describes the association relationship between different modal data.

[0060] Specifically, after completing the vectorization of the primary index, the system of the embodiment of the present application stores the generated primary index vector in the vector tag library, and the single diagnosis and treatment activity association map is stored in the map library. The vector tag library can support vector-based retrieval. When a patient queries, the relevant medical diagnosis and treatment data can be quickly located by vector matching. The map library saves the association relationship between specific medical and health data of each modality to support cross-modal data acquisition based on data association. At the same time, by storing the retrieval vector and the map in the database, the data persistence and retrieval efficiency can be improved.

[0061] exist Figure 1 In the process shown, first, multimodal medical and health data containing at least one diagnosis and treatment activity of a single patient is obtained. Then, in the multimodal medical and health data of a single diagnosis and treatment activity, semantic inconsistency between different modalities is solved by semantic alignment, so that the relevant data of different modalities can be associated together. After completing the semantic alignment, the medical field knowledge is used to perform semantic associations between medical and health data of different modalities to obtain a single diagnosis and treatment activity association map, which associates the corresponding medical and health data of different modalities and belonging to the same diagnosis and treatment activity. Afterwards, a label is set for the single diagnosis and treatment activity association map based on the medical field knowledge, and multiple first-level indexes of the single diagnosis and treatment activity association map are constructed based on the label, and multiple first-level indexes are vectorized to obtain multiple first-level index vectors, and these first-level index vectors are stored in the vector label library, and the single diagnosis and treatment activity association map is stored in the map library to form a complete retrieval system. These first-level index vectors vectorize the semantic information of the association map of a single diagnosis and treatment activity, so that the system can perform data retrieval through vector matching; since the data of different modalities have formed a unified index and label through semantic alignment and association maps, the system can realize cross-modal query. For example, if the user enters "acute altitude sickness" as a keyword, the system will not only return relevant text records, but also associated imaging data and detection indicators. It can be seen that this application realizes cross-modal query of medical and health data, which is not only limited to a single modality, but also can associate data of multiple modalities, ensuring the comprehensiveness of the query.

[0062] See also Figure 2 As shown, in Figure 1Based on the process shown, another method for indexing multimodal medical and health data provided in an embodiment of the present application stores multimodal medical and health data in corresponding data storage pools respectively, and associates the data in each data storage pool through a single diagnosis and treatment activity data view, which may include the following steps:

[0063] S201: For multimodal medical and health data of a single diagnosis and treatment activity, the medical and health data of different modalities are correspondingly stored in multiple data storage pools, wherein each data storage pool is used to store medical and health data of one modality.

[0064] In order to improve the input and output IO performance of medical and health data of different modalities, the embodiment of the present application sets up corresponding data storage pools for medical and health data of each modality, and each data storage pool independently stores medical data of a specific modality. For example, image data is stored in a specific image data storage pool, and electronic medical records are stored in a specific document data storage pool. By storing data of different modalities separately, it is possible to improve data access efficiency, reduce the system burden during data processing, and improve the overall speed of data retrieval.

[0065] S202: Establish a single diagnosis and treatment activity data view associated with the patient, where the single diagnosis and treatment activity data view is used to obtain multimodal medical and health data of a single diagnosis and treatment activity from multiple data stores.

[0066] In an embodiment of the present application, a single diagnosis and treatment activity data view refers to a view used to aggregate multimodal medical and health data related to a patient's single diagnosis and treatment activity.

[0067] Specifically, the embodiment of the present application associates all multimodal data in a single diagnosis and treatment activity of the patient into a data view based on the patient's global unique identifier such as the patient ID on multiple data storage pools to form a single diagnosis and treatment activity data view. Through this data view, relevant data can be retrieved from multiple data storage pools when needed to obtain multimodal medical and health data of a single diagnosis and treatment activity corresponding to the data view, such as a series of data such as electronic medical records, radiological images, physical and chemical tests, and hospitalization records generated in a single patient's independent diagnosis and treatment activity.

[0068] By establishing a data view of a single diagnosis and treatment activity, data from different modalities can be effectively organized, unified access to data can be achieved, retrieval efficiency can be improved, and the dispersion of multimodal data during retrieval can be reduced.

[0069] S203: According to the patient's diagnosis and treatment activity time, associate each single diagnosis and treatment activity data view corresponding to the patient to obtain a diagnosis and treatment activity data view time series.

[0070] It is understandable that a patient may have multiple diagnosis and treatment activities, and correspondingly, multiple sets of multimodal medical and health data of single diagnosis and treatment activities will be generated. Furthermore, according to the time sequence of the patient's diagnosis and treatment activities, the multiple single diagnosis and treatment activity data views corresponding to the patient can be sorted and associated by time to form a data structure arranged in time sequence, that is, a diagnosis and treatment activity data view time series, so as to associate medical and health data of different time periods.

[0071] At the same time, semantic associations can be further performed in the time series of the diagnosis and treatment activity data view. For example, the shadow areas of the image data generated by the previous diagnosis and treatment activity and the image data generated by the next diagnosis and treatment activity can be compared to determine expansion or reduction.

[0072] exist Figure 2 In the process shown, multiple data storage pools are involved to store medical and health data of one modality respectively, thereby improving the IO performance of medical and health data and further improving the retrieval rate. By establishing a single diagnosis and treatment activity data view, the medical and health data of different modalities of a single diagnosis and treatment activity are effectively integrated together to achieve unified acquisition of cross-modal data.

[0073] See also Figure 3 As shown, in Figure 1 Based on the process shown, the embodiment of the present application further provides a method for indexing multimodal medical and health data, which obtains medical and health data required by the user based on the user's data retrieval request, and specifically includes the following steps:

[0074] S301: Generate a primary query index based on a data retrieval request sent by a user. When there is one primary query index generated, execute S302; when there are at least two primary query indexes generated, execute S303.

[0075] In an embodiment of the present application, after obtaining the user's retrieval request, the data retrieval request sent by the user is semantically understood and the query intent is identified to obtain query keywords as the primary query index. Among them, the data retrieval request is a query request submitted by the user, which usually contains natural language or keywords. If the user inputs a natural language question, it needs to be parsed. This process mainly relies on a fine-tuned large language model, which has the ability to extract opinions and understand intentions from natural language. The specific steps include: first, extracting opinions; by reading the user's natural language and processing the user input with a large language model, the user's real needs or opinions are extracted. Then, intent recognition is performed, and the large language model understands the user's query intent by analyzing the user's input. Finally, the core keywords or concept entities that can be used for retrieval are further extracted.

[0076] In the embodiment of the present application, after executing S301, the execution of subsequent steps will be determined by the number of generated first-level query indexes. For example, if only one first-level query index is generated, it is directly used by executing S302. If at least two first-level query indexes are generated, further processing is performed by executing S303.

[0077] S302: vectorize the first-level query index to obtain a first-level query index vector; determine a target index vector from multiple first-level index vectors in the vector tag library according to the first-level query index vector; and execute S304.

[0078] The target index vector refers to the index vector in the vector tag library that best meets the user's query conditions.

[0079] In an embodiment of the present application, the primary query index generated based on the data retrieval request is converted into a vector that can be understood by a computer and can reflect the query requirements, thereby obtaining a primary query index vector. Based on the primary query index vector, vector similarity matching is performed in the vector tag library to determine the index vector known in the vector tag library that best meets the user's query requirements as the target index vector.

[0080] S303: Combining at least two first-level query indexes to obtain a second query index, and vectorizing the second-level query index to obtain a second-level query index vector; determining a target index vector in a vector tag library according to the second-level query index vector; and executing S304.

[0081] When a user's query request involves multiple query conditions, the embodiment of the present application will further generate a combined query index, namely a secondary query index, based on the primary index. Afterwards, the secondary query index is vectorized, and a target index vector is determined in the vector tag library based on the secondary query index vector.

[0082] In some implementations of the embodiments of the present application, a primary index vector library and a secondary index vector library are further created; wherein the primary index vector library is used to cache the primary index vectors extracted from the vector tag library, and the secondary index vector library is used to cache the secondary query index vectors generated based on the data retrieval request.

[0083] Among them, the first-level index vector library is mainly used to cache the first-level index vectors extracted from the vector label library. This means that the index vectors matched from the vector label library will be temporarily stored in the first-level index vector library to facilitate the next subsequent use. If there is a similar request next time, the system can read directly from the first-level index vector library without having to extract it from the vector label library again each time, thereby improving the retrieval speed. The function of the second-level index vector library is similar to that of the first-level index vector library, which is used to cache the results of the second-level query and reduce repeated calculations. This ensures that in multiple queries, especially when complex queries are involved, the system can quickly call the previously cached second-level index vectors, further improving the response speed of the system.

[0084] By creating a primary index vector library and a secondary index vector library, the system can effectively reduce the time of repeated calculations and avoid re-extracting index vectors from the vector tag library each time, which not only improves the efficiency of data retrieval, but also speeds up the processing speed of complex user queries. At the same time, the hierarchical cache mechanism can more flexibly handle query requests of different complexities, allowing the system to run efficiently when processing simple and complex queries.

[0085] In some implementations of the embodiments of the present application, the secondary index vector library periodically cleans up secondary query index vectors with low usage frequency.

[0086] When a user performs multiple queries, the system will generate different secondary query index vectors. These vectors may be frequently used over a period of time, but some query indexes may gradually no longer be accessed or used. In order to prevent these infrequently used vectors from occupying a large amount of storage space, the embodiment of the present application periodically cleans up the secondary query index vectors with low usage frequency to free up more space for storing updated query results, thereby avoiding performance degradation caused by overfilling the storage space.

[0087] S304: Acquire target data based on the target index vector.

[0088] In an embodiment of the present application, since the target index vector is a known retrieval vector in the vector tag library, and the retrieval vector is specifically associated with a single diagnosis and treatment activity association map, the target single diagnosis and treatment activity association map can be associated with the target based on the target index vector, and then the corresponding target data can be obtained based on the target single diagnosis and treatment activity association map.

[0089] In some implementations of the embodiments of the present application, acquiring target data based on the target index vector may specifically include the following steps:

[0090] S401: Determine a target primary index associated with a target index vector.

[0091] After the target index vector is determined, the target index vector is matched with its associated primary index by querying the vector tag library.

[0092] S402: Determine a target diagnosis and treatment activity association map associated with the target primary index.

[0093] After matching the target primary index, the target primary index is further used to search the associated single diagnosis and treatment activity association map from the map library as the target diagnosis and treatment activity association map.

[0094] S403: Determine a target diagnosis and treatment activity data view associated with the target diagnosis and treatment activity association map.

[0095] After finding the target diagnosis and treatment activity association map, the corresponding single diagnosis and treatment activity data view is further matched based on the correspondence between the single diagnosis and treatment activity association map and the single diagnosis and treatment activity data view as the target diagnosis and treatment activity data view.

[0096] S404: Acquire target data based on the target diagnosis and treatment activity data view.

[0097] As can be seen from the above, a single diagnosis and treatment activity data view is used to obtain multimodal medical and health data of a single diagnosis and treatment activity from multiple data storages, so the target data can be obtained through the target diagnosis and treatment activity data view.

[0098] S305: De-privacy processing is performed on the target data.

[0099] In the embodiments of the present application, in order to protect user privacy, after the target data is acquired, the target data will be further de-privatized, and only necessary medical information will be retained for further analysis and research. For example, sensitive information such as the patient's name, ID number and other unique identification information is replaced, such as replacing it with a serial number that has no specific reference such as "P0001". This process only needs to ensure the consistency of the replacement. For fields that have no direct meaning for data analysis, such as the patient's home address, random replacement or deletion can be adopted; for gender information, it is replaced with meaningless values ​​such as 0, 1, 2, etc.

[0100] exist Figure 3 In the process shown, generating a primary index based on the user's data retrieval request, or combining multiple primary indexes into a secondary index, can more flexibly handle query requests of different complexities, allowing the system to run efficiently when processing simple and complex queries. In addition, by de-privacy processing of the target data, user information security is further guaranteed.

[0101] In an actual application scenario, the method provided in the embodiment of the present application runs in a distributed cluster computing environment, and each computing node in the cluster must include a central processing unit CPU, memory, and a motherboard. Optional configurations include a hard disk and a deep learning accelerator. When there is a shared storage subsystem in the system, such as a storage area network SAN, a network attached storage NAS, or a storage server array, the computing node may or may not be equipped with a hard disk; when there is no shared storage subsystem, the computing node must include a hard disk. In addition, the deep learning accelerator optionally installed on the computing node can be a homogeneous accelerator or a heterogeneous accelerator. The computing nodes are connected through an Internet network, which includes but is not limited to Ethernet, a high-speed Internet network Infiniband, etc.

[0102] In the specific implementation of the present invention, the index storage server is a large memory server, which is equipped with memory and high-performance hard disk, which together constitute a distributed memory object storage for storing index data. The index storage servers are connected through Infiniband and connected to the query terminal through multi-level routing; the computing server mainly performs semantic understanding, semantic alignment, association analysis and other calculations, and is configured with intelligent computing acceleration devices such as graphics processors GPU; the data storage server provides data storage capabilities, and the server cluster is further divided into relational data storage partitions, document data partitions, file storage partitions, etc.; multiple query access terminals are configured in the environment to receive user queries or calculation inputs, and isolate users from the cluster environment. The specific network topology is as follows Figure 5 shown.

[0103] See also Figure 6 As shown, the system running in the above distributed cluster computing environment mainly involves the following functional modules at the software level:

[0104] 1. Various data repositories.

[0105] Stores various types of medical and health data, including but not limited to: semi-structured document data, such as electronic medical records, medical record homepage data, hospitalization data, and imaging and physical and chemical test interpretation data; unstructured data, such as medical imaging and electrocardiogram, electroencephalogram and other data in physical and chemical tests; structured table data, such as physical and chemical test report data.

[0106] 2. Vector label repository and graph repository.

[0107] The vector tag repository is a vector database that mainly stores conceptual entities identified from medical and health data, such as the International Classification of Diseases ICD, drug names, etc. Based on the tag repository, the embodiment of the present application also constructs an in-memory index library for caching the first-level tags directly extracted from the vector tag library and the second-level tags formed by combining the first-level tags during the query process; the atlas library mainly stores the diagnosis and treatment behavior association atlas constructed from the original data. Among them, the vector tag library can directly index the diagnosis and treatment activity atlas in the atlas library.

[0108] 3. Index building module.

[0109] The core of the index construction module is a multimodal large language model. Preferably, the glm-4V-9B basic large model is used in the embodiment of the present application, and the existing medical field professional knowledge base is plugged in as external knowledge support. In this module, the multimodal large language model is mainly used to construct functional submodules including text concept entity recognition, text semantic understanding, image semantic understanding and test data semantic understanding, and on this basis, a pipeline of semantic alignment, label vectorization and graph construction is constructed. Preferably, the bge-large-zh-v1.5 model is used for label vectorization.

[0110] 4. Data query module.

[0111] The core of the data query module is a large language model with fine-tuning of instructions. Preferably, the embodiment of the present application uses the glm-4V-9B basic large model, and uses a self-built query instruction sample containing about 15,000 sets of instructions and answer pairs to fine-tune the large model. On this basis, a pipeline is constructed that includes user input intent understanding, concept entity recognition, vectorized query, secondary index construction, data query and de-privacy to data set construction. Preferably, the bge-large-zh-v1.5 model is used for label vectorization.

[0112] exist Figure 6 In the software architecture shown for running the method provided by the embodiment of the present application, the overall process described mainly involves two processes, namely the index building process and the data query process.

[0113] Index building process:

[0114] First, we perform text concept entity recognition, extract text content from semi-structured and structured data, and use instructions to fine-tune the large language model and the medical field professional knowledge base to support the extraction of key concepts. We also implement concept alignment to align the aforementioned normative expressions such as medical terminology concepts to eliminate ambiguity.

[0115] Then, semantic understanding is performed, including semantic understanding of text, images, and test data, and semantic information is identified from data of different modalities. This is mainly achieved by combining a large model and a Contrastive Language-Image Pretraining (CLIP) model. The CLIP model mainly extracts semantic information from image data, and the large model extracts semantic information from text. The extracted semantic information is also vectorized, and alignment is achieved after comparing vector similarities.

[0116] Then semantic alignment is carried out to align the semantics in different data and build a unified semantic expression, which is mainly implemented based on the CLIP model.

[0117] Then the concept labels are vectorized, and the concept entities are labeled and vectorized in turn to form a vector label library and a primary index library.

[0118] Finally, the graph is constructed, and the relationship between concepts is extracted by using the concept entity and semantic alignment information to form a diagnostic activity association graph.

[0119] Data query process:

[0120] First, we understand the intent, read the user query, and use the instructions to fine-tune the large model to extract the user's point of view and understand the user's intent. The user's intent is divided into two categories: single case and query data set.

[0121] Then, conceptual entity recognition is performed to identify core concepts based on user queries;

[0122] Then, a vectorized query is performed to vectorize the core concepts of the user's query and query the index library;

[0123] If the user involves multi-condition query, the secondary index will also be built. If the query concept is not hit in the secondary index, and multiple concept entities are hit in the primary index, multiple primary index concepts will be combined and written into the secondary index;

[0124] The query data is then de-privatized, using the index to retrieve data in the diagnostic activity association graph and eliminate private information.

[0125] Finally, the data set is constructed, and the query data is packaged according to the data object to form a data set copy and returned to the user.

[0126] See also Figure 7 The present application embodiment provides a system for constructing an index of multimodal medical and health data, the system comprising:

[0127] The data acquisition module 701 is used to acquire multimodal medical and health data including at least one diagnosis and treatment activity of a single patient;

[0128] A graph generation module 702 is used to perform semantic alignment between different modalities of medical and health data in the multimodal medical and health data of a single diagnosis and treatment activity, and after the semantic alignment, perform semantic association between different modalities of medical and health data on the multimodal medical and health data of a single diagnosis and treatment activity based on the pre-configured medical field knowledge, so as to obtain a single diagnosis and treatment activity association graph;

[0129] The index construction module 703 is used to set labels for the single diagnosis and treatment activity association map based on medical field knowledge, construct multiple first-level indexes of the single diagnosis and treatment activity association map based on the labels, and vectorize the multiple first-level indexes to obtain multiple first-level index vectors;

[0130] The data storage module 704 is used to store multiple primary index vectors in a vector label library, and store a single diagnosis and treatment activity associated map in a map library.

[0131] In some implementations of the embodiments of the present application, the system further includes:

[0132] A data storage module is used to store the multimodal medical and health data of a single diagnosis and treatment activity in multiple data storage pools, wherein each data storage pool is used to store medical and health data of one modality;

[0133] The view establishment module is used to establish a single diagnosis and treatment activity data view associated with the patient. The single diagnosis and treatment activity data view is used to obtain multimodal medical and health data of a single diagnosis and treatment activity from multiple data stores.

[0134] In some implementations of the embodiments of the present application, the system further includes:

[0135] The sequence generation module is used to associate the patient's corresponding single diagnosis and treatment activity data views according to the patient's diagnosis and treatment activity time to obtain the diagnosis and treatment activity data view time sequence.

[0136] In some implementations of the first aspect of the present application, the system further includes:

[0137] A primary index generation module, used to generate a primary query index based on a data retrieval request sent by a user, and vectorize the primary query index to obtain a primary query index vector;

[0138] A target index determination module, used to determine a target index vector from a plurality of first-level index vectors in a vector tag library according to the first-level query index vector;

[0139] The data acquisition module is used to acquire target data based on the target index vector.

[0140] In some implementations of the embodiments of the present application, the system further includes:

[0141] A secondary index generation module, configured to, when generating at least two primary query indexes based on a data retrieval request, combine the at least two primary query indexes to obtain a second query index, and vectorize the secondary query index to obtain a secondary query index vector;

[0142] The target index determination module is also used to determine the target index vector in the vector tag library according to the secondary query index vector.

[0143] In some implementations of the embodiments of the present application, the data acquisition module is specifically used to:

[0144] determining a target primary index associated with the target index vector;

[0145] Determine the target diagnosis and treatment activity association map associated with the target primary index;

[0146] Determine a target diagnosis and treatment activity data view associated with the target diagnosis and treatment activity association map;

[0147] Obtain target data based on target diagnosis and treatment activity data view.

[0148] In some implementations of the embodiments of the present application, the system further includes:

[0149] The privacy processing module is used to de-privacy the target data.

[0150] In some implementations of the embodiments of the present application, the system further includes:

[0151] The vector cache module is used to create a primary index vector library and a secondary index vector library. The primary index vector library is used to cache the primary index vectors extracted from the vector tag library, and the secondary index vector library is used to cache the secondary query index vectors generated based on data retrieval requests.

[0152] In some implementations of the embodiments of the present application, the system further includes:

[0153] The cache cleaning module is used to periodically clean up the secondary query index vectors with low usage frequency in the secondary index vector library.

[0154] Finally, it should be noted that, in the embodiments of the present application, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not clearly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the statement "comprise one..." do not exclude the presence of other identical elements in the process, method, article or equipment including the elements.

[0155] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing an index of multimodal medical and health data, characterized in that: The method comprises: Obtain multimodal medical and health data containing at least one diagnosis and treatment activity of a single patient; In the multimodal medical and health data of a single diagnosis and treatment activity, semantic alignment is performed between medical and health data of different modalities, and after the semantic alignment, based on the pre-configured medical field knowledge, semantic association is performed between the multimodal medical and health data of a single diagnosis and treatment activity, so as to obtain a single diagnosis and treatment activity association map; Based on the medical field knowledge, setting labels for the single diagnosis and treatment activity association map, constructing multiple first-level indexes of the single diagnosis and treatment activity association map based on the labels, and vectorizing the multiple first-level indexes to obtain multiple first-level index vectors; The multiple primary index vectors are stored in a vector label library, and the single diagnosis and treatment activity associated map is stored in a map library.

2. The method according to claim 1, characterized in that The method further comprises: For the multimodal medical and health data of the single diagnosis and treatment activity, the medical and health data of different modalities are correspondingly stored in multiple data storage pools, wherein each data storage pool is used to store medical and health data of one modality; A single diagnosis and treatment activity data view associated with the patient is established, and the single diagnosis and treatment activity data view is used to obtain multimodal medical and health data of the single diagnosis and treatment activity from the multiple data stores.

3. The method according to claim 2, characterized in that The method further comprises: According to the diagnosis and treatment activity time of the patient, the single diagnosis and treatment activity data views corresponding to the patient are associated to obtain a diagnosis and treatment activity data view time series.

4. The method according to claim 1, characterized in that: The method further comprises: Generate a primary query index based on a data retrieval request sent by a user, and vectorize the primary query index to obtain a primary query index vector; According to the first-level query index vector, determining a target index vector from a plurality of the first-level index vectors in the vector tag library; Target data is acquired based on the target index vector.

5. The method according to claim 4, characterized in that The method further comprises: When at least two primary query indexes are generated based on the data retrieval request, combining the at least two primary query indexes to obtain a second query index, and vectorizing the secondary query index to obtain a secondary query index vector; According to the secondary query index vector, a target index vector is determined in the vector tag library.

6. The method according to claims 2 and 4, characterized in that Acquiring target data based on the target index vector includes: determining a target primary index associated with the target index vector; Determine a target diagnosis and treatment activity association map associated with the target primary index; Determine a target diagnosis and treatment activity data view associated with the target diagnosis and treatment activity association map; The target data is acquired based on the target diagnosis and treatment activity data view.

7. The method according to claim 4, characterized in that The method further comprises: The target data is de-privacy processed.

8. The method according to claim 5, characterized in that The method further comprises: A primary index vector library and a secondary index vector library are created, wherein the primary index vector library is used to cache the primary index vector extracted from the vector tag library, and the secondary index vector library is used to cache the secondary query index vector generated based on the data retrieval request.

9. The method according to claim 5, characterized in that The method further comprises: The secondary index vector library periodically clears secondary query index vectors with low usage frequency.

10. A multimodal medical and health data index construction system, characterized in that: The system comprises: A data acquisition module, used to acquire multimodal medical and health data including at least one diagnosis and treatment activity of a single patient; A graph generation module is used to perform semantic alignment between different modalities of medical and health data in the multimodal medical and health data of a single diagnosis and treatment activity, and after the semantic alignment, perform semantic association between different modalities of the multimodal medical and health data of a single diagnosis and treatment activity based on pre-configured medical field knowledge to obtain a single diagnosis and treatment activity association graph; An index construction module, used to set labels for the single diagnosis and treatment activity association map based on the medical field knowledge, construct multiple primary indexes of the single diagnosis and treatment activity association map based on the labels, and vectorize the multiple primary indexes to obtain multiple primary index vectors; A data storage module is used to store the multiple primary index vectors in a vector label library and store the single diagnosis and treatment activity associated map in a map library.

Images