Medical data processing method and device, electronic equipment and computer readable medium

By using a combination of vector library tables and knowledge graphs in medical data processing, the problems of high labor costs, low efficiency and insufficient accuracy in the existing medical data annotation methods are solved, and a more efficient and reliable data annotation process is achieved, improving the The efficiency of user information retrieval.

CN118939804APending Publication Date: 2024-11-12BEIJING JINGDONG TUOXIAN TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411103991.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing medical data annotation methods have problems such as high labor costs, low efficiency, difficulty in model update and maintenance, and insufficient accuracy and professionalism in specific medical fields, resulting in inaccurate annotation affecting the efficiency of user retrieval and information use.

Method used

By receiving data processing requests, abnormal entities in medical text data are extracted, structured data is obtained based on the abnormal entity matching vector library table, combined with knowledge graph and vectorization processing, and used inverted index product quantization index method to match, and weighted features Linear combination calculations are used to obtain comprehensive decision results and improve the accuracy and reliability of labeling.

Benefits of technology

The data processing accuracy and reliability related to data annotation is improved, thereby improving the efficiency of users' annotated data retrieval and information use.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939804A_ABST
    Figure CN118939804A_ABST
Patent Text Reader

Abstract

The invention discloses a medical data processing method and device, electronic equipment and a computer readable medium, and relates to the technical field of artificial intelligence, big data and medical health, one specific implementation mode comprises the steps of receiving a data processing request, obtaining medical text data, and extracting abnormal entities in the medical text data; matching the vector library table based on the abnormal entity to obtain matched structured abnormal data; obtaining first matching result data based on the structured abnormal data and the knowledge graph; performing vectorization processing on a title in the medical text data to obtain a title vector, and matching the title vector with each vector subspace in a vector library table based on an inverted index product quantization index method to obtain second matching result data; based on the first matching result data and the second matching result data, comprehensive decision result data are obtained through linear combination calculation of weighted features. And the accuracy and reliability of data processing related to medical data annotation are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence, big data and medical health technology, and in particular to a medical data processing method, device, electronic device and computer-readable medium. Background Art

[0002] Currently, in the field of medical content labeling, labels such as diseases, departments, symptoms, and drugs are key words used to classify and index medical information. These labels are crucial to improving the efficiency and accuracy of medical information retrieval. First, although traditional manual labeling schemes can achieve high accuracy, they face the problems of high labor costs, low efficiency, and difficulty in quickly expanding to large-scale data sets. Secondly, although existing automatic labeling schemes can reduce costs and improve efficiency to a certain extent, they usually suffer from high training costs, difficulty in model updating and maintenance, and lack of accuracy and professionalism in specific medical fields. Inaccurate labeling will directly affect the efficiency of users in retrieving and using medical information, especially in the decision-sensitive medical field, where this impact is particularly important. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a medical data processing method, device, electronic device and computer-readable medium, which can solve the problem that the existing inaccurate annotations will directly affect the efficiency of users in retrieving and using information.

[0004] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present application, a medical data processing method is provided, including: receiving a data processing request, obtaining medical text data, and extracting abnormal entities in the medical text data; matching a vector library table based on the abnormal entities to obtain matched structured abnormal data; obtaining first matching result data based on the structured abnormal data and the knowledge graph; vectorizing the title in the medical text data to obtain a title vector, matching the title vector with each vector subspace in the vector library table based on an inverted index product quantization index method to obtain second matching result data; based on the first matching result data and the second matching result data, obtaining comprehensive decision result data by calculating a linear combination of weighted features.

[0005] Optionally, extracting abnormal entities from medical text data includes: extracting abnormal content data from the medical text data; generating guiding prompt words based on the abnormal content data; and inputting the guiding prompt words into a large language model to obtain corresponding abnormal entities.

[0006] Optionally, based on the abnormal entity matching vector library table, to obtain matched structured abnormal data, including: vectorizing the abnormal entity to obtain the abnormal entity vector; calculating the similarity between the abnormal entity vector and each abnormal vector in the vector library table, and determining the target abnormal vector based on the similarity and a preset threshold; determining the label corresponding to the target abnormal vector in the vector library table as the structured abnormal data matching the abnormal entity.

[0007] Optionally, based on the structured exception data and the knowledge graph, obtaining first matching result data includes: using the structured exception data as a query condition and applying it to the knowledge graph to obtain corresponding first matching result data.

[0008] Optionally, obtaining the second matching result data includes: clustering the title vector and each vector subspace in the vector library table based on the inverted index to obtain each clustering cluster; for each clustering cluster, performing product quantization on the corresponding title vector and vector subspace to obtain the second matching result data.

[0009] Optionally, based on the first matching result data and the second matching result data, comprehensive decision result data is calculated by linear combination of weighted features, including: obtaining the weight value corresponding to the feature function corresponding to each label in the first matching result data and the second matching result data; for each label, calculating the product of the feature function and the weight value; according to the product, determining the comprehensive decision value of the corresponding label; and determining the comprehensive decision result data according to each comprehensive decision value.

[0010] Optionally, the method also includes: in response to detecting a data change in a relational data temporary storage library, determining the changed data through a logical log mechanism; generating a message queue based on the changed data, and sending the message queue to a message middleware in real time; calling a business application to consume data change messages from the message middleware in real time according to preset message queue consumption rules, executing corresponding business logic processing, and obtaining a business logic processing result; and updating a vector library table according to the business logic processing result.

[0011] In addition, the present application also provides a medical data processing device, including: a receiving unit, configured to receive a data processing request, obtain medical text data, and extract abnormal entities in the medical text data; an acquisition unit, configured to match a vector library table based on the abnormal entity to obtain the matched structured abnormal data; a first matching unit, configured to obtain first matching result data based on the structured abnormal data and the knowledge graph; a second matching unit, configured to vectorize the title in the medical text data to obtain a title vector, and match the title vector with each vector subspace in the vector library table based on an inverted index product quantization index method to obtain second matching result data; a calculation unit, configured to obtain comprehensive decision result data based on the first matching result data and the second matching result data by linear combination calculation of weighted features.

[0012] Optionally, the receiving unit is further configured to: extract abnormal content data from the medical text data; generate guiding prompt words based on the abnormal content data; and input the guiding prompt words into the large language model to obtain corresponding abnormal entities.

[0013] Optionally, the acquisition unit is further configured to: perform vectorization processing on the abnormal entity to obtain an abnormal entity vector; calculate the similarity between the abnormal entity vector and each abnormal vector in the vector library table, and determine the target abnormal vector based on the similarity and a preset threshold; determine the label corresponding to the target abnormal vector in the vector library table as structured abnormal data matching the abnormal entity.

[0014] Optionally, the first matching unit is further configured to: apply the structured abnormal data as a query condition to the knowledge graph to obtain corresponding first matching result data.

[0015] Optionally, the second matching unit is further configured to: cluster the title vector and each vector subspace in the vector library table based on the inverted index to obtain each clustering cluster; for each clustering cluster, perform product quantization on the corresponding title vector and vector subspace to obtain second matching result data.

[0016] Optionally, the calculation unit is further configured to: obtain the weight value corresponding to the feature function corresponding to each label in the first matching result data and the second matching result data; for each label, calculate the product of the feature function and the weight value; based on the product, determine the comprehensive decision value of the corresponding label; and determine the comprehensive decision result data based on each comprehensive decision value.

[0017] Optionally, the medical data processing device also includes an update unit, which is configured to: in response to detecting a data change in a relational data temporary storage library, determine the changed data through a logical log mechanism; generate a message queue based on the changed data, and send the message queue to the message middleware in real time; call the business application to consume data change messages from the message middleware in real time according to preset message queue consumption rules, execute corresponding business logic processing, and obtain business logic processing results; and update the vector library table according to the business logic processing results.

[0018] In addition, the present application also provides a data processing electronic device, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the one or more processors implement the medical data processing method as described above.

[0019] In addition, the present application also provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the medical data processing method as described above is implemented.

[0020] To achieve the above objective, according to another aspect of the embodiments of the present application, a computer program product is provided.

[0021] A computer program product according to an embodiment of the present application includes a computer program, and when the program is executed by a processor, the medical data processing method provided by the embodiment of the present application is implemented.

[0022] One embodiment of the above invention has the following advantages or beneficial effects: the present application receives a data processing request, obtains medical text data, extracts abnormal entities in the medical text data; matches the vector library table based on the abnormal entity to obtain the matched structured abnormal data; obtains the first matching result data based on the structured abnormal data and the knowledge graph; vectorizes the title in the medical text data to obtain the title vector, matches the title vector with each vector subspace in the vector library table based on the inverted index product quantization index method to obtain the second matching result data; based on the first matching result data and the second matching result data, obtains the comprehensive decision result data through the linear combination calculation of weighted features. Improve the accuracy and reliability of data processing related to data annotation, thereby improving the efficiency of users' retrieval and use of information based on the annotated data.

[0023] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to better understand the present application and do not constitute an improper limitation on the present application.

[0025] Figure 1is a schematic diagram of the main process of a medical data processing method according to an embodiment of the present application;

[0026] Figure 2 is a schematic diagram of the main process of a medical data processing method according to an embodiment of the present application;

[0027] Figure 3 It is a schematic diagram of a process of creating a new vector library / migrating a vector library and a process of incrementally maintaining a vector library according to a medical data processing method according to an embodiment of the present application;

[0028] Figure 4 is a schematic diagram of a monitoring process of a medical data processing method according to an embodiment of the present application;

[0029] Figure 5 is a schematic diagram of a label marking process of a medical data processing method according to an embodiment of the present application;

[0030] Figure 6 It is a schematic diagram of the design process of a vector library logic table of a medical data processing method according to an embodiment of the present application;

[0031] Figure 7 is a schematic diagram of a search process of a medical data processing method according to an embodiment of the present application;

[0032] Figure 8 is a schematic diagram of main units of a medical data processing device according to an embodiment of the present application;

[0033] Fig. 9 is an exemplary system architecture diagram to which the embodiments of the present application can be applied;

[0034] Fig.10 It is a structural diagram of a computer system of a terminal device or server suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0035] The following is an explanation of the exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of known functions and structures is omitted in the following description. It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of the present application are in compliance with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of the present application, certain software, components, models, and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or must have used the solution. In the technical solution of the present application, the collection, collection, update, analysis, processing, use, transmission, storage, etc. of the user personal information involved are in compliance with the provisions of relevant laws and regulations, are used for legal and reasonable purposes, and do not violate public order and good customs, are not shared, leaked or sold outside these legal uses, and are subject to supervision and management by regulatory authorities. Take necessary measures to prevent illegal access to such user personal information data, maintain the security of user personal information, network security and national security, ensure that persons who have access to personal information data comply with relevant laws and regulations, and ensure the security of user personal information. Once such user personal information data is no longer needed, the risk should be minimized by limiting or even prohibiting data collection and / or deleting data.

[0036] When used, including in certain related applications, user privacy is protected by de-identifying data, such as by removing specific identifiers when used, controlling the amount or specificity of data stored, controlling how data is stored, and / or other methods of de-identification.

[0037] Figure 1 is a schematic diagram of the main process of a medical data processing method according to an embodiment of the present application, such as Figure 1 As shown, the medical data processing method mainly includes the following steps S101 to S105.

[0038] Step S101, receiving a data processing request, acquiring medical text data, and extracting abnormal entities in the medical text data.

[0039] In this embodiment, the execution subject of the medical data processing method (for example, a server) can receive a data processing request by wired connection or wireless connection. The data processing request can be, for example, a request for labeling the questions raised by the user with preset dimensions. In the scenario of medical content labeling, the preset dimensions can be one or more of departments, diseases, symptoms, drugs, etc. In the exception handling scenario, the preset dimensions can be one or more of exception handling parties, exception judgment data, abnormal performance, and exception solutions. When the execution subject receives the data processing request, it can obtain the link carried in the request and jump to the target position corresponding to the link to obtain the medical text data. Medical text data can be a combination of one or more of pictures and texts. The embodiment of the present application does not specifically limit the form and content of medical text data. For example, the medical text data can be "treatment methods for children's thrush", "what are the causes of chronic cough, sputum and wheezing in winter and spring", "does chronic tonsillitis affect sleep", etc.

[0040] Specifically, extracting abnormal entities from medical text data includes: extracting abnormal content data from the medical text data; generating guiding prompt words based on the abnormal content data; and inputting the guiding prompt words into a large language model to obtain corresponding abnormal entities.

[0041] For example, the abnormal content data may be medical content. The execution subject may replace the corresponding placeholder in the guide prompt word generation template with the abnormal content data to generate the guide prompt word. And the generated guide prompt word is input into the large language model so that the large language model outputs the corresponding result according to the guide prompt word, that is, the abnormal entity. For example, the abnormal entity may be a disease entity, or it may be abnormal software, or it may be an abnormal node. The embodiment of the present application does not specifically limit the type of abnormal entity.

[0042] Step S102, based on the abnormal entity matching vector library table, to obtain matched structured abnormal data.

[0043] Specifically, based on the abnormal entity matching vector library table, to obtain the matched structured abnormal data, including: vectorizing the abnormal entity to obtain the abnormal entity vector; calculating the similarity between the abnormal entity vector and each abnormal vector in the vector library table, and determining the target abnormal vector based on the similarity and a preset threshold; determining the label corresponding to the target abnormal vector in the vector library table as the structured abnormal data matching the abnormal entity.

[0044] The execution subject can vectorize the abnormal entity by word embedding to obtain the abnormal entity vector corresponding to the abnormal entity. The similarity between the calculated abnormal entity vector and each abnormal vector in the vector library table can be cosine similarity. The abnormal vector in the vector library table corresponding to the similarity greater than the preset threshold is determined as the target abnormal vector, and the label corresponding to the target abnormal vector in the vector library table is determined as the structured abnormal data matching the abnormal entity. For example, the structured abnormal data can be structured disease data.

[0045] Step S103, obtaining first matching result data based on the structured exception data and the knowledge graph.

[0046] Specifically, based on the structured exception data and the knowledge graph, first matching result data is obtained, including: using the structured exception data as a query condition and applying it to the knowledge graph to obtain corresponding first matching result data.

[0047] For example, the structured abnormal data can be structured disease data. The structured disease data is used as the query condition of the knowledge graph. Based on the preset query syntax corresponding to the knowledge graph, the labels corresponding to the structured disease data in the knowledge graph, such as department labels, disease labels, symptom labels, drug labels, etc., are obtained as the first matching result data.

[0048] Step S104, vectorize the titles in the medical text data to obtain title vectors, and match the title vectors with each vector subspace in the vector library table based on an inverted index product quantization indexing method to obtain second matching result data.

[0049] The title in the medical text data is vectorized based on the word embedding method. The title can be, for example, a medical content title, such as "How to treat gastric ulcer?", "How to regulate peptic gastric ulcer?". After obtaining the title vector, the title vector can be matched with each vector subspace in the vector library table based on the inverted index product quantization index method to obtain the second matching result data. The second matching result data can be the department label, disease label, symptom label, drug label, etc. matched in each vector subspace in the vector library table. Inverted index product quantization, also known as inverted file product quantization (IVFPQ), is an efficient vector quantization technology used to improve the search efficiency of large-scale vector databases. It combines the two technologies of inverted index and product quantization. By quantizing the vector space and constructing an inverted index, it realizes fast approximate nearest neighbor search.

[0050] Step S105, based on the first matching result data and the second matching result data, obtain comprehensive decision result data by linear combination calculation of weighted features.

[0051] Specifically, based on the first matching result data and the second matching result data, comprehensive decision result data is obtained by calculating the linear combination of weighted features, including: obtaining the weight value corresponding to the feature function corresponding to each label in the first matching result data and the second matching result data; for each label, calculating the product of the feature function and the weight value; according to the product, determining the comprehensive decision value of the corresponding label; and according to each comprehensive decision value, determining the comprehensive decision result data.

[0052] After obtaining the first matching result data and the second matching result data, a comprehensive decision is made to determine the final label annotation. For example, the execution subject can provide the first matching result data obtained based on the knowledge graph matching and the second matching result data obtained based on the vector subspace matching in the vector library table to the comprehensive decision module, so as to obtain the feature function from the first matching result data and the second matching result data through the comprehensive decision logic in the comprehensive decision module. , and determine the weight value corresponding to the acquired feature function , through a comprehensive decision algorithm:

[0053] The comprehensive decision value corresponding to each label is calculated, and the comprehensive decision result data is determined according to the comprehensive decision value of each label. The comprehensive decision result data is the final automatic labeling process result of the department, disease, symptom, and drug, for example, a single department label (id+name), n disease labels (id+name), n symptom labels (id+name), and n drug labels (id+name) corresponding to the medical content and medical content title.

[0054] This embodiment receives a data processing request, obtains medical text data, extracts abnormal entities in the medical text data; matches a vector library table based on the abnormal entities to obtain matched structured abnormal data; obtains first matching result data based on the structured abnormal data and the knowledge graph; vectorizes the title in the medical text data to obtain a title vector, matches the title vector with each vector subspace in the vector library table based on an inverted index product quantization index method to obtain second matching result data; based on the first matching result data and the second matching result data, obtains comprehensive decision result data through a linear combination calculation of weighted features. Improves the accuracy and reliability of data processing related to data annotation, thereby improving the efficiency of users in retrieving and using information based on annotated data.

[0055] Figure 2FIG. 1 is a schematic diagram of the main flow of a medical data processing method according to an embodiment of the present application. Figure 2 As shown, the medical data processing method mainly includes the following steps S201 to S207.

[0056] Step S201, receiving a data processing request, acquiring medical text data, and extracting abnormal entities in the medical text data.

[0057] When receiving a data processing request, the execution subject can obtain the address identifier and location identifier carried in the request, determine the corresponding data source according to the address identifier, and obtain the medical text data at the location corresponding to the location identifier from the determined data source. Medical text data can be a combination of one or more of voice, video, pictures, and documents. The embodiment of the present application does not specifically limit the form and content of medical text data. For example, the medical text data can be "What kind of blood-enriching oral liquid is good for pregnant women", "How to use nasal spray", etc.

[0058] Abnormal entities in the extracted medical text data, such as "anemia in pregnant women", "rhinitis", etc.

[0059] Step S202, based on the abnormal entity matching vector library table, to obtain matched structured abnormal data.

[0060] The vector library table stores structured abnormal data and corresponding structured abnormal labels. After extracting the abnormal entity from the medical text data, the execution subject can convert the abnormal entity into an abnormal entity vector and perform cosine similarity matching with the data in the vector library table, and determine the structured abnormal label corresponding to the matched data as the structured abnormal data corresponding to the abnormal entity. For example, the structured abnormal label can be a structured disease label, and the embodiment of the present application does not specifically limit the type of the structured abnormal label.

[0061] Step S203, obtaining first matching result data based on the structured exception data and the knowledge graph.

[0062] The structured abnormal data is used as a query condition, a query statement is constructed based on the query syntax corresponding to the knowledge graph, and the query statement is executed in the knowledge graph to obtain the first matching result data matched from the knowledge graph. For example, the first matching result data can be a label corresponding to the structured disease data in the knowledge graph, such as a department label, a disease label, a symptom label, a drug label, etc.

[0063] Step S204, vectorizing the titles in the medical text data to obtain title vectors.

[0064] Step S205 , clustering the title vector and each vector subspace in the vector library table based on the inverted index to obtain each cluster.

[0065] The inverted index originates from the need to find records based on attribute values ​​in practical applications. Each item in this index table includes an attribute value and the address of each record with this attribute value. Since the attribute value is not determined by the record, but the location of the record is determined by the attribute value, it is called an inverted index. That is, the title vector (i.e., the attribute value) determines the corresponding vector subspace in the vector library table (i.e., the record corresponding to the attribute value), and the title vector and the corresponding vector subspace are grouped into one category to obtain the corresponding cluster. A title vector can correspond to one or more vector subspaces, and can correspond to multiple clusters.

[0066] Step S206: for each cluster, perform product quantization on the corresponding title vector and vector subspace to obtain second matching result data.

[0067] ‌ The IVFPQ algorithm (i.e., inverted index product quantization algorithm) first performs IVF index clustering, and then uses PQ (Product Quantization) to perform product quantization on the vector. Because PQ performs compression quantization, its index file is relatively small, which enables fast vector retrieval. Product Quantization (PQ) is a data compression and processing technology that is mainly used in high-dimensional data processing and machine learning applications. Its core idea is to divide the high-dimensional space into multiple low-dimensional subspaces and perform quantization processing in each subspace. This method can significantly reduce the storage requirements of data while maintaining the main characteristics of the data.

[0068] When a title vector corresponds to multiple vector subspaces, for each cluster, the corresponding title vector and vector subspace are multiplied and quantized to effectively process high-dimensional data, and then vector retrieval is performed to speed up the retrieval speed, so as to quickly and accurately obtain the second matching result data. The second matching result data can be department labels, disease labels, symptom labels, drug labels, etc. matched in each vector subspace in the vector library table.

[0069] Step S207: Based on the first matching result data and the second matching result data, comprehensive decision result data is calculated by linear combination of weighted features.

[0070] Specifically, the method also includes: in response to detecting a data change in a relational data temporary storage library, determining the changed data through a logical log mechanism; generating a message queue based on the changed data, and sending the message queue to a message middleware in real time; calling a business application to consume data change messages from the message middleware in real time according to a preset message queue consumption rule, executing corresponding business logic processing, and obtaining a business logic processing result; and updating a vector library table according to the business logic processing result.

[0071] A relational database temporary storage library, for example, can be a temporary storage MySQL data table. Use BinLake monitoring to achieve efficient capture and distribution of change data of the temporary storage MySQL data table, and send the change data to the message middleware in real time in the form of MQ messages (i.e., message queues) through the Binlog mechanism. The business code in the business application consumes the messages in the message middleware according to the preset message queue consumption rules to obtain the business logic processing results, and then updates the Vearch vector library medical table (the Vearch vector library medical table in the embodiment of this application is the vector library table) based on the obtained business logic processing results. To ensure that the vector library table is up to date in real time, thereby ensuring the accuracy of the results of data processing based on the vector library table.

[0072] In the embodiments of this application, the terms involved are explained as follows:

[0073] Prompt (prompting) is an important part of the use of large language models. Prompt is an input method that limits the input of a large language model to a specific range, thereby better controlling the output of the model. In a large language model, prompt technology can be used to fine-tune and optimize the large language model to make it more suitable for certain specific tasks.

[0074] BinLake is an internal MySQL binlog synchronization and subscription system of XX. It is secondary developed based on the open source system canal. It unifies and standardizes the technology stack of MySQL binlog synchronization and subscription scenarios.

[0075] Vector Embedding Model is a machine learning model that maps items (such as words, phrases, entire documents, or other types of data) into representations in a continuous vector space. These models are designed to capture semantic relationships between items so that semantically similar items are close in distance in the vector space.

[0076] Vearch is an open source large-scale distributed vector search engine that supports efficient vector retrieval and storage. It can handle neighbor search problems for high-dimensional data and is suitable for various scenarios involving vector search, such as image recognition, recommendation systems, etc.

[0077] When designing a Vearch indexing method: If memory is not a limiting factor, give priority to the graph search HNSW algorithm, because the graph search HNSW algorithm can simultaneously guarantee low latency and recall accuracy. If memory usage needs to be controlled, the inverted index IVF algorithm can be used, which can reduce memory usage while maintaining query speed and quality similar to the graph search HNSW algorithm. However, if memory is a larger limiting factor, PQ encoding can be added to the graph search HNSW algorithm or the inverted index IVF algorithm to achieve product quantization to further reduce memory usage.

[0078] Cluster recall technology is a data retrieval method that clusters large-scale data sets to group similar data together. When querying, it first locates several clusters containing the most relevant data, and then searches within these clusters, thereby improving the efficiency and accuracy of retrieval.

[0079] Inverted File Product Quantization (IVFPQ) is an efficient vector quantization technology used to improve the search efficiency of large-scale vector databases. It combines the inverted index and product quantization techniques to achieve fast approximate nearest neighbor search by quantizing the vector space and building an inverted index.

[0080] Semantic vectors are vectors generated by vector embedding models, which are used to mathematically represent the semantic content of words, phrases, or documents. Semantic vectors can capture the semantic information and contextual relationships in text, enabling computers to understand and process natural language.

[0081] The medical data processing method of the embodiment of the present application can be applied to the field of medical content labeling. For example, XX Health has a large amount of medical content, and each medical content corresponds to more than ten labels, involving multiple categories such as departments, diseases, symptoms, drugs, scenes, festivals, categories, SKUs, etc. The embodiment of the present application simplifies the actual process and mainly describes the processing flow and related systems of department, disease, symptom, and drug labels. The medical data processing method of the embodiment of the present application includes the following processes: 1) A method for synchronizing the upstream database in a full or incremental big data manner, 2) An automatic labeling method for departments, diseases, symptoms, and drugs, and 3) An efficient design and retrieval method for medical vector libraries.

[0082] For medical data, the relationship between 1), 2), and 3) above is: the full or incremental big data method realizes accurate synchronization of medical data to the storage method (vector library table) expected by the embodiment of this application. The automatic annotation process retrieval of departments, diseases, symptoms, and drugs uses these medical data to achieve accurate and professional annotation of medical content. The efficient design and efficient retrieval method of the medical vector library table make it possible to store massive data and conduct millions of parallel retrievals.

[0083] The big data synchronization process and automatic labeling of 1) and 2) above can be as follows: Figure 3 , Figure 4 , Figure 5 As shown in the figure, the efficient design and retrieval method of the medical vector library table mentioned in 3) can be Figure 6 , Figure 7 , Figure 8 shown.

[0084] Specifically, Figure 3 It is a schematic diagram of the process of creating a new / migrating vector library in full and the process of maintaining an incremental vector library in a medical data processing method according to an embodiment of the present application. Figure 4 FIG. 1 is a schematic diagram of a monitoring process of a medical data processing method according to an embodiment of the present application. Figure 3 As shown in the figure, the process of synchronizing the upstream symptom, disease, and drug databases in a full or incremental big data manner is shown. Among them, the full creation / migration of the vector library process includes the big data process and the full insertion process of the vector library. Specifically, the big data process includes: the big data Flink extracts the upstream business library data (that is, the disease, symptom, and drug data in the upstream database) and passes it to the big data OLAP data warehouse, and then executes the full insertion process of the vector library, that is, the big data OLAP data warehouse injects the upstream business library data into and temporarily stores it in the MySQL data table, uses BinLake monitoring to achieve efficient capture and distribution of the change data of the temporary MySQL data table, and sends the change data to the message middleware in real time in the form of MQ messages through the Binlog mechanism. The business code consumes the messages in the message middleware, and then updates the Vearch vector library medical table, BGE (BGE is the strongest semantic vector model under the current Chinese task, and its various semantic representation capabilities comprehensively surpass similar open source models) vector embedding model service Maas.

[0085] Example, the method of creating / migrating a vector library in full:

[0086] Step A1: Real-time data extraction: Use Apache Flink as a real-time data processing framework to extract business data from upstream business databases (such as relational database management systems). Apache Flink was selected for its excellent real-time data processing capabilities, supporting high throughput and low-latency data extraction, and ensuring real-time synchronization with upstream data sources.

[0087] Step A2: Data preprocessing: During the data extraction process, data cleaning, validation, and transformation rules are applied to ensure the quality and format consistency of the data. The preprocessing step ensures that the data meets the quality standards before storage and analysis, reducing the complexity of subsequent processing.

[0088] Step B1: OLAP data warehouse storage: Import the pre-processed data into the OLAP data warehouse and build a large wide table for analysis to facilitate efficient data query and analysis. Technical considerations: The use of OLAP data warehouse supports rapid analysis and processing of massive data. The design of large wide tables optimizes query performance and provides a strong data foundation for subsequent data extraction.

[0089] Step C1: Intermediate data temporary storage: Use big data technology to import the processed data into the MySQL temporary storage library (MySQL is a relational database management system) as the intermediate stage of vector library synchronization. Technical considerations: MySQL is selected as the temporary storage library, considering its wide application and stability, which is convenient for integration with the existing technology stack and for subsequent data operations and management.

[0090] Step D1: Capture and distribute change data: Use the BinLake system to monitor changes in the MySQL temporary repository (including addition, modification, and deletion operations), and send the change data to JMQ in real time through the binlog mechanism (bin log is a logical log that records the logic of the execution statement). The use of BinLake enables efficient capture and distribution of MySQL change data, and JMQ, as a message middleware, ensures efficient and reliable transmission of data change events.

[0091] Step E1: Consume messages and process business logic: Business applications consume data change messages from JMQ in real time according to the preset MQ consumption rules, and perform corresponding business logic processing, such as data conversion, annotation rule application, etc. This link can effectively handle the problem of message backlog during peak hours by flexibly configuring consumption rules, and ensure the real-time and accuracy of data processing.

[0092] Step F1: Update the vector library: Update the vector library according to the results of business logic processing, including the addition, modification and deletion of vector data to keep the data up to date. This step focuses on ensuring the accuracy and timeliness of vector library data, and supports efficient synchronization of full or incremental data through precise data operations to meet dynamic data requirements.

[0093] Specifically, the incremental maintenance process of the vector library is as follows: when the upstream database (disease, symptoms, drugs) is updated, the MySQL data table is temporarily updated, and the vector library incremental update process is executed, that is, BinLake monitoring is used to achieve efficient capture and distribution of changed data in the upstream database, and the changed data is sent to the message middleware in real time in the form of MQ messages through the Binlog mechanism. The business code consumes the messages in the message middleware, and then updates the Vearch vector library medical table, BGE (BGE is the strongest semantic vector model under the current Chinese tasks, and its various semantic representation capabilities comprehensively surpass similar open source models) vector embedding model service Maas.

[0094] Example, method for incremental synchronization of upstream database:

[0095] Step A: Real-time data change capture: Monitor upstream business database changes: Through the customized BinLog monitoring service, the system continuously monitors the change events of the upstream department's business database, including data addition, modification and deletion operations. Customized monitoring services are used instead of general solutions to ensure high sensitivity and low latency response to specific database operations, thereby achieving real-time tracking of business dynamics.

[0096] Step B: Efficient data processing and transmission: Preliminary processing of BinLog data: BinLog data pre-processed by the BinLake system is de-noised, formatted, and compressed to optimize transmission efficiency and reduce the load on the message queue (MQ). The preliminary processing step ensures the quality of data before transmission, reduces network transmission overhead through data compression and optimization, and improves the efficiency of subsequent processing.

[0097] Step C: Flexible and reliable data processing: Highly configurable message consumption: The business logic layer uses highly configurable MQ consumption strategies to customize message monitoring and processing logic according to actual needs to ensure that data changes are effectively and accurately captured and processed. Through configurable consumption strategies, this system can flexibly respond to data change events of different scales and complexities, while ensuring the efficiency and scalability of the data processing process.

[0098] Step D: Precisely controlled data update: Finely managed vector library update operations: Based on the business processing results, perform fine incremental updates of the vector library, including precise data insertion, update, and deletion operations, to ensure the real-time and accuracy of the vector library data. By adopting fine-grained management of update operations, this system can accurately control the quality of data synchronization, minimize data inconsistencies and errors, and thus maintain the high quality status of the vector library data.

[0099] Enhanced features: Data consistency and integrity assurance: Implement data verification mechanism: Introduce data integrity and consistency verification mechanism to ensure that each data point synchronized from the upstream database to the vector library passes a series of verifications to ensure its accuracy and reliability. This enhanced feature improves the system's control over data quality, reduces manual intervention through automated verification processes, and ensures automation and high-quality output of the data synchronization process.

[0100] Figure 4 The BinLake monitoring process (i.e., Binlog to MQ process) in is as follows: When BinLake monitors the change data of the temporary MySQL data table, it performs binary binlog parsing (BinlogWorker decodeBytes) and simple primary key parsing (BinlogWorker simple-parse) through MS SQL (Microsoft's SQLServer database server), and sends the change data in the form of a queue to the message middleware in real time through BinlogHandler. In the message middleware, the sending and receiving of messages depends on the topic. A topic represents a type of message. For example, Figure 4 As shown in the figure, topic1, topic2, ..., topicN represent a type of data change message. Data is distributed according to a certain hash rule (hash), completely parsed, table structure, column name, value processing (complete-parse), and data is encapsulated (formatMessage) as a protobuf (Protobuf uses a syntax for defining message formats, which allows the definition of field types, sequences, and rules, making the message structure clearer and more concise) object, and produced by various producers (producer1, producer2, ..., producerN) to obtain the message queue MQ.

[0101] Figure 5 This is a schematic diagram of the labeling process of the medical data processing method according to an embodiment of the present application. The automatic labeling process of department, disease, symptom, and drug is used as an example to illustrate. The automatic labeling process of department (i.e., abnormal processing party), disease (i.e., abnormal judgment data), symptom (i.e., abnormal manifestation), and drug (i.e., abnormal solution) consists of the following three sub-processes:

[0102] like Figure 5 As shown, 1. Medical content-large language model-knowledge graph sub-process:

[0103] Step 1A: Use a large language model to analyze medical content and extract disease entities. Specifically, the entity extraction prompt containing medical content can be input into the large language model to obtain the disease entity.

[0104] Step 1B: The extracted disease entity matches the existing structured disease label in the vector library. Specifically, the text of the extracted disease entity is obtained, and the BGE vector embedding model service Maas is called to convert the text of the disease entity into a disease vector, and the disease vector is matched with the disease subspace of the medical table in the vector library to obtain structured disease data, i.e., structured disease labels.

[0105] Step 1C: The structured disease labels are used as query conditions and applied to the knowledge graph to match the knowledge graph diseases and obtain the departments, diseases, symptoms, and drugs related to the entity.

[0106] Step 1D: Use the large language model to classify disease entities into corresponding departments. This subprocess combines the deep learning capabilities of the large language model with the rich semantic information of the knowledge graph to ensure accurate identification of medical entities from the text and obtain comprehensive medical information through the knowledge graph.

[0107] The results of the medical content-large language model-knowledge graph sub-process can be shown as follows:

[0108] 2. Medical content title-vectorized matching sub-process (including disease labeling process, symptom labeling process, drug labeling process, and department labeling process):

[0109] Step 2A: Vectorize the title of the medical content to obtain a title vector for matching query in the vector space.

[0110] Step 2B: Match the title vector in the vector subspace of diseases, departments, symptoms and drugs, that is, match the title vector with the disease subspace of the vector library medical table, match it with the symptom subspace of the vector library medical table, and match it with the drug subspace of the vector library medical table, and find the most similar multiple disease, symptom and drug labels, that is, find the disease (structured data) that matches the semantics of the title in the disease library, find the symptoms (structured data) that match the semantics of the title in the symptom library, and find the drugs (structured data) that match the semantics of the title in the drug library. The large language model extracts disease entities from medical content to obtain disease text, generates classification prompts based on the disease text, inputs the classification prompts into the large language model for department inference, and calls the BGE vector embedding model service MaaS to match the vector corresponding to the disease text with the department subspace in the vector library medical table, and obtains the department (structured data) that matches the disease text in the department library. Through the vectorization of the title and the matching query of the subspace, this sub-process supplements the possible information missing of the knowledge graph, improves the matching accuracy and efficiency, and especially shows its advantages when processing large-scale medical data.

[0111] The result of the medical content title-vectorization matching subprocess can be shown as follows:

[0112] Drug Subspace:

[0113] Symptom subspace:

[0114] 3. Comprehensive decision-making process (including department comprehensive decision, disease comprehensive decision, symptom comprehensive decision and drug comprehensive decision): Step 3A: Combining the above two sub-processes, the automatic labeling of departments, diseases, symptoms and drugs is completed through the following comprehensive decision-making algorithm:

[0115]

[0116] in, represents the characteristic functions obtained from the two subprocesses, is the corresponding weight value, which is used to measure the importance of each feature in the final decision. The comprehensive decision algorithm makes full use of the information obtained from the large language model, vectorized matching and knowledge graph through the linear combination of weighted features, and achieves automatic labeling with high accuracy and efficiency.

[0117] The result of the comprehensive decision-making process is the final automatic labeling process result of departments, diseases, symptoms, and drugs, that is, the comprehensive decision result data, for example, a single department label (id+name) corresponding to the medical content and medical content title, n disease labels (id+name), n symptom labels (id+name), and n drug labels (id+name).

[0118] The embodiments of the present application will describe the above three processes starting from the modules.

[0119] Large language model module: The automatic annotation method of the embodiment of the present application first relies on the large language model technology. Specifically, the large language model is used to perform deep semantic understanding and inference on medical texts, including: extracting disease entities and large language model prompts that comply with ICD 10 for the full text of medical content. The Prompt guide words of the large language model here are based on the experience and guidance of "Principled Instructions Are All You Need for Questioning LLaMA-1 / 2, GPT-3.5 / 4", and the results of optimizing multiple boards after business practice are as follows:

[0120] "As a medical NLP content expert, your task is to identify the most relevant disease entity mentioned in the provided article content, according to ICD10 standards. Alongside identifying the disease, assess its relevance to the article content on a scale from 0 (no relevance) to 1 (fullyrelevant). It's imperative to respond concisely in this format in ZH-CN:'DiseaseName,RelevanceScore' without adding extra characters or spacing. Here is the article content: {content}".

[0121] The Prompt guidance prompt words of this large language model use the following prompt word optimization principles:

[0122] Clearly define the task objective: clearly state that the task is to identify the disease entities mentioned in the article that are most relevant to the ICD10 criteria and evaluate their relevance. This helps the model focus precisely on the core requirements of the task.

[0123] Specify output format: Requiring responses in the format of “disease name, relevance score” and emphasizing that no extra characters or spaces are added will help ensure consistency and usability of the model output.

[0124] Emphasize accuracy and conciseness: The prompt words emphasize that the answer needs to be concise and directly related to the question, avoiding unnecessary information filling, making the model output more refined.

[0125] Interactive guidance: Although direct interaction is limited in a single prompt word, by explicitly expressing the task action of "identify and evaluate", an interactive exploration process is simulated, allowing the model to more actively seek relevant information when processing tasks.

[0126] Avoid bias: When assessing the relevance of disease entities, the model is prompted to score based on objective criteria (ICD10) to avoid making judgments based on biases that may be built into the model.

[0127] Save tokens and prevent the illusion of a large language model: Using English instead of Chinese can save about 40% of the tokens (99 for the English version and 144 for the Chinese version).

[0128] The following are examples of extracted diseases and confidence levels:

[0129] After that, a series of methods are used to accurately parse the disease name and confidence level answered by the large language model, and decide whether to accept the extraction result of the large language model based on the confidence level. The specific process can be shown as follows:

[0130] 1. Cleaning and preprocessing: First remove leading and trailing spaces and possible quotes from the input string suggested_result.

[0131] 2. Null value handling: If suggested_result is "None" or an empty string, the function returns None.

[0132] 3. Parsing and processing: Try to split suggested_result into disease name and confidence by comma. If the result of comma splitting exceeds two elements (which may occur when the disease name contains commas), try to parse the confidence from the second element until a floating point number is successfully parsed. If there are only two elements, the second element is directly parsed as the confidence.

[0133] 4. Confidence judgment: If the parsed confidence is greater than or equal to 0.75, the function returns the disease name; otherwise, it returns None.

[0134] 5. Exception handling: If any exception is encountered during the parsing process, the error is recorded and None is returned.

[0135] This process can be simplified by mathematical formula:

[0136]

[0137] The extracted disease entity will be used as input to trigger the other two steps: 1. Infer the prompt guide words of the large language model of the relevant department from the disease entity; 2. Match the disease library according to the disease entity to obtain structured disease data as the input of the knowledge graph.

[0138] After completing these two steps, the following disease and department label matching results are obtained as shown in the following table:

[0139] The large language model Prompt prompt guide words that infer related departments from disease entities. The following is an example:

[0140] "If you are a medical NLP expert, match the most relevant department according to the following medical keywords: {answer}, only one is enough, no need to explain the information, just reply to the specific department, please strictly follow the reply format of "andrology", the department information must come from the following department list, think carefully before answering. The following is the list of departments: fundus department, glaucoma department, eye trauma department, cataract department, optometry center, ophthalmic plastic surgery department, ophthalmology department, cornea department, pediatric ophthalmology department, dermatology department, venereal disease department, neonatology department, pediatric internal medicine department, pediatric hematology department, pediatric respiratory department, pediatric gastroenterology department, pediatric neurology department, pediatric cardiology department, pediatric Nephrology, Pediatric Endocrinology, Pediatric Immunology, Gynecology, Obstetrics, Reproductive Health and Infertility, Family Planning, Otolaryngology, Head and Neck Surgery, Pediatric Otolaryngology, Joint Surgery, Spine Surgery, Traumatology, Foot and Ankle Surgery, Hand Surgery, Orthopedics, Oncology, Oncology Surgery, Interventional and Radiotherapy Center, Gynecology Oncology, Bone Oncology, Internal Medicine of Traditional Chinese Medicine, Ophthalmology of Traditional Chinese Medicine, Gynecology and Obstetrics of Traditional Chinese Medicine, Orthopedics of Traditional Chinese Medicine, Pediatrics of Traditional Chinese Medicine, Acupuncture and Massage, Dermatology of Traditional Chinese Medicine, Oncology of Traditional Chinese Medicine, Rehabilitation of Traditional Chinese Medicine, Surgery of Traditional Chinese Medicine, Obstetrics of Traditional Chinese Medicine, Preventive Medicine of Traditional Chinese Medicine, Nephrology of Traditional Chinese Medicine, Endocrinology of Traditional Chinese Medicine, Geriatrics of Traditional Chinese Medicine, Peripheral Vascular Disease of Traditional Chinese Medicine, Traditional Chinese Medicine Department of Psychosomatic Medicine, Department of Hepatobiliary Medicine, Department of Breast Medicine, Department of Urology, Department of Otolaryngology, Department of Anorectal Medicine, Department of Digestive Medicine, Department of Respiratory Medicine, Department of Rheumatology and Immunology, Department of Cardiology, Department of Neurology, Department of Hematology, Department of Andrology, Department of Respiratory Medicine, Department of Cardiovascular Medicine, Department of Neurology, Department of Digestive Medicine, Department of Nephrology, Department of Hematology, Department of Endocrinology, Department of Rheumatology and Immunology, Department of Geriatrics, Department of General Medicine, Department of Andrology, Department of Maxillofacial Surgery, Department of Prosthodontics, Department of Orthodontics, Department of Implantology, Department of Endodontics, Department of Periodontology, Department of Oral Mucosa, Department of Pediatric Dentistry, Department of General Dentistry, Department of Psychiatry, Department of Clinical Psychology, Department of Sleep Medicine, Department of Addiction Medicine, Department of Youth Department of Child Psychology, Geriatric Psychiatry, Pediatric Health Care, Pediatric Rehabilitation, Pediatric Surgery, Pediatric Cardiothoracic Surgery, Pediatric Orthopedics, Pediatric Urology, Pediatric Neurosurgery, Radiology, Ultrasound, Nuclear Medicine, Pathology, Pain Medicine, Hepatology, Infectious Diseases, Medical Cosmetology, Tuberculosis, Hyperbaric Oxygen, Sports Medicine, Occupational Diseases, Nursing Consultation, Health Management, Pharmacy, Emergency Department, Anesthesiology, Hepatobiliary Surgery, Urology, General Surgery, Neurosurgery, Cardiovascular Surgery, Thoracic Surgery, Breast Surgery, Vascular Surgery, Anorectal Surgery, Thyroid Surgery, Burns, Gastrointestinal Surgery, Minimally Invasive Surgery, Plastic Surgery.

[0141] The purpose of this prompt is to let the model match the most relevant department from a preset list of departments based on the given medical keywords. It uses the following principles:

[0142] Clarity and conciseness: The prompts are clear and directly point to the required task.

[0143] Specify output format: Emphasize the importance of output format to ensure the consistency of model output.

[0144] Avoid bias and errors: Make sure department lists are accurate and do not contain incorrect characters.

[0145] In the embodiment of the present application, the large model is a large language model, and the results obtained are shown in the column "Department inference answer of the large model" in the following table:

[0146] Vector search module: The functions of the vector search module are as follows: 1. Medical content-large language model-knowledge graph sub-process, calibrate and ensure the output of the large language model: match the disease vector library according to the disease entity, and obtain structured disease data as the input of the knowledge graph; 2. Medical content title-vectorized matching sub-process, as the second sub-process and auxiliary method outside the medical content-large language model-knowledge graph sub-process, uses the vectorized medical content title to match in each vector subspace of the disease, department, symptom and drug. The matching results are finally input into the comprehensive decision-making module, and the final labeling result is comprehensively decided together with the results obtained from the medical content-large language model-knowledge graph sub-process.

[0147] Sub-process 1: Vector library matching of disease entities and knowledge graph input preparation:

[0148] Step 1a: Disease entity vector matching: Use the disease entity output by the large language model to perform a matching query in the disease vector library to obtain the vector entry that is most similar to the disease entity. This step uses advanced vector search technology to ensure that the disease entity output by the large language model is highly matched with the entity in the disease vector library.

[0149] Step 1b: Structured disease data extraction: Based on the matching results, structured disease data is extracted from the disease vector library to provide input for subsequent knowledge graph construction and updating. This step emphasizes the importance of inputting high-quality, structured disease information into the knowledge graph, ensuring the accuracy and depth of knowledge graph analysis and providing reliable input for comprehensive decision-making.

[0150] Sub-process 2: Medical content title-vectorized matching:

[0151] Step 2a: Vectorization of medical content titles: Vectorize the medical content titles and convert them into mathematical expressions that can be matched and queried in the vector space. The vectorization step uses advanced NLP technology to ensure that the titles of medical content accurately reflect their semantic content.

[0152] Step 2b: Vector subspace matching query: Perform matching query in the vector subspace of diseases, departments, symptoms and drugs to find the most relevant entries. This step demonstrates the ability of the vector search module to perform efficient and accurate queries in multi-dimensional vector spaces. It is worth noting that the present invention ensures the efficiency, relevance and accuracy of the query results through the efficient design of the medical vector library table and the retrieval method.

[0153] Step 2c: Preparation of input for the comprehensive decision module: Provide the matching query results as input to the comprehensive decision module, and make a comprehensive decision together with the results of the large language model-knowledge graph sub-process to finally decide on the label annotation of the content. This step integrates the matching results of the vector search module with the analysis results of the large language model and knowledge graph, and uses the comprehensive decision module to ensure the high accuracy and reliability of the final annotation results.

[0154] Similarity calculation of vector matching: The inner product measurement method is used to calculate the matching degree between the entity and the vector in the vector library. The formula is as follows:

[0155]

[0156] Here, (A) and (B) are two vectors, and (A i ) and (B i ) are the elements of vectors (A) and (B) at the (i)th position, respectively. Symbols It means to sum all the elements (i) from 1 to the dimension (n) of the vector. The product of each item ( ) are added together, and the resulting sum is the inner product of vectors (A) and (B). This inner product measures how much the two vectors point in the same direction.

[0157] Method for filtering labels matched by vectors: is the original label set in the vector table subspace, where each label There is a corresponding similarity score For each label, a threshold is defined , used to filter similarity scores. The formal representation is as follows:

[0158]

[0159] Here The operation represents the process of selecting the top N elements with the highest similarity. The condition Ensure that only the similarity is not less than the threshold The tag is selected . Representation Tags The similarity score of is a preset similarity threshold.

[0160] Knowledge Graph Module: In the medical content - large language model - knowledge graph sub - process, embodiments of the present application use a vector library to extract disease entities and query a medical knowledge graph to obtain labels to be annotated, such as departments, diseases, symptoms, medications, etc. This step utilizes the rich associated data of the knowledge graph, which can provide more comprehensive and in - depth medical information, thereby enhancing the accuracy and richness of the annotation results. The query syntax based on the knowledge graph is as follows:

[0161] In the above query syntax, the MATCH clause is used to specify a pattern, which is the relationship from the disease node (d:Disease) to the symptom node (s:Symptom), the medication node (m:Medication), and the department node (dpt:Department). {name: 'Cold'} is an attribute filter used to match only disease entities whose name attribute is "Cold". -> represents a directional relationship, from the disease to the symptom, medication, and department. [:CAUSES], [:TREATED_BY], and [:BELONGS_TO] are relationship types, representing the symptoms caused by the disease, the medications used to treat the disease, and the department to which the disease belongs, respectively. The RETURN clause specifies the return format of the query result, which is to return the name of the disease, the list of related symptoms, the list of related medications, and the name of the department to which it belongs. The collect() function is an aggregation function used to combine the names of multiple nodes into a list.

[0162] Exemplarily, the finally obtained query results are as follows:

[0163] As can be seen from the table, since the knowledge graph is still under construction and there is a certain lack of data, the results obtained from the medical content - large language model - knowledge graph sub - process may not be precise enough. The application of the medical content title - vectorized matching sub - process provides an efficient and precise matching method, compensating for possible information gaps in the knowledge graph, ensuring the comprehensiveness of the matching results, and the appropriateness with the original content.

[0164] Figure 6 is a schematic diagram of the design process of the vector library logic table for the medical data processing method according to an embodiment of the present application.

[0165] Among them, efficient design of medical vector library table: When facing the massive content data of XX Health and various medical libraries, the traditional table storage design method will face multiple challenges, including data island problems, increased complexity of searching across tables, increased maintenance costs, and decreased search performance. In order to overcome these problems, the embodiment of the present application proposes an innovative non-table vector table design method, which can achieve efficient data retrieval and management: the unified vector table design greatly simplifies the retrieval process, because all subspaces share the same vector table index, and there is no need to perform association queries between multiple tables, thereby speeding up the cross-subspace search. Optimized memory usage and computing resource utilization can be achieved: by using product quantization (IVFPQ) technology, the memory usage is significantly reduced, and all data sets are compressed and stored in the same structure, thereby improving the efficiency of data processing and reducing storage costs. Flexible data expansion and update can be achieved: the non-table design allows new data categories or dimensions to be added quickly without redesigning the entire database architecture, improving dynamic data adaptability and system scalability. Simplified system maintenance and development can be achieved: the unified structure reduces the maintenance workload and potential error risks, simplifies subsequent development and system upgrades, and reduces the complexity and cost of long-term maintenance.

[0166] The vector library logic table design method (i.e., non-table design) of the embodiment of the present application comprises the following steps:

[0167] Step A: Vector table structure definition: Define a unified vector table structure that can store the vectorized representations of all medical entities, regardless of the entity category.

[0168] Step B: Multi-dimensional data integration: Use the embeddingType field to distinguish different subspaces, that is, use the embeddingType field to distinguish different categories of medical entities, such as 10000L corresponding to content (such as symptoms) CONTENT, 20000L corresponding to tags TAG, 30000L corresponding to diseases DISEASE, etc. For example, the content subspace is 10000L-19999L; the tag subspace is 20000L and above. There are multiple subspaces (Collections) under a vector table (Space). For example, CONTENT_DEFAULT(10000L, "Medical popular science graphic content", "content"), ... n content subspaces, CONTENT_QA(10013L, "Medical question and answer content", "content"), DEPART(20000L, "Department label", "depart"), DISEASE(30000L, "Label subspace corresponding to the drug library", "desease"), ..., ... m independent label subspaces, DRUG(60000L, "Label subspace corresponding to the drug library", "drug"), SYMPTOM(70000L, "Symptom library label", "symptom").

[0169] Step C: Index construction and management: non-table design, so that all subspaces share a vector index (such as IVFPQ, HNSW, greatly improving the speed of cross-subspace search), near real-time millisecond-level million concurrency, low vector library memory requirements, use a common indexing strategy, such as IVFPQ, to build an inverted index applicable to all medical entities to achieve fast vector retrieval.

[0170] Using the embeddingType field to distinguish different subspaces and share a unified index can optimize engine cache usage, avoid engine context switching, improve resource utilization, and achieve multi-subspace joint recall. In addition, the vector table field has strong verification.

[0171] Step D: Unified search interface design: Design a unified API layer so that upper-level applications can query through a public interface without having to worry about specific data categories.

[0172] Step E: Dynamic Data Update and Extension: When different categories of data need to be updated or new data is added, the unified vector table structure allows seamless extension without modifying the existing data structure.

[0173] Through the above-mentioned design method, the embodiment of the present application can effectively manage and retrieve XX Health's huge and diverse medical data set, providing a scalable, high-performance and cost-effective solution in a big data environment.

[0174] Design of efficient retrieval method for medical vector library table: IVFPQ algorithm first performs IVF index clustering, and then uses PQ (Product Quantization) to perform product quantization on the vector. Because PQ performs compression quantization, its index file is relatively small, which will also cause corresponding accuracy loss.

[0175] The embodiment of the present application adopts the inverted index product quantization (IVFPQ) indexing method as the core technical route, mainly based on the following advantages: Memory efficiency: By compressing each text vector to the extreme storage size, IVFPQ achieves a substantial optimization of memory usage. For example, for 100 million 768-dimensional vectors (768 (dimensions) X4 (float occupies 4 bytes)), the IVF-Flat algorithm requires a memory size of (100 million X768X4) / (1024X1024X1024), while IVFPQ only requires a memory size of (100 million X6X1) / (1024X1024X1024), reducing the amount of data that originally required 286G of memory to 0.56G. Balance between query performance and accuracy: IVFPQ provides an effective balance between high query speed and good recall effect, meeting the needs of XX Health to ensure the accuracy of search results while ensuring search efficiency. Adaptability and scalability: IVFPQ technology supports GPU acceleration and shows strong adaptability when processing dynamically updated data. It can quickly respond to data changes and ensure the timeliness and accuracy of the search system.

[0176] The parameters of IVFPQ of Vearch in the embodiment of the present application are as follows - indexing parameters:

[0177] Query parameters:

[0178] IVFPQ: It is described as having medium accuracy, the fastest speed, and low cost in large-scale high-dimensional data compression and search. It has high adaptability to high dimensions because quantization technology can effectively process high-dimensional data. Application scenarios: large-scale high-dimensional data search that requires compressed storage. It has high query speed and good recall effect. It is suitable for scenarios with cluster data of billions and single-machine data volume of more than 10 million. If the query speed and recall effect are required to be high, it can be used in combination with HNSW and OPQ. It combines IVF and quantization technology to reduce storage and computing through quantization, which is suitable for scenarios with limited storage space and computing resources. It supports GPU.

[0179] Figure 7 FIG. 1 is a schematic diagram of a search process of a medical data processing method according to an embodiment of the present application. Figure 7 As shown, the acquired type + title + medical content Id is embedded into the BGE vector, and then the type vector is input into the embeddingType mapper, and inserted into the vector library table 10000L to query and obtain the corresponding vector table doc id; the acquired medical content id + type is input into the medical content id to vector table id queryer, and the corresponding vector table id is queried and obtained; according to the above-obtained two vector table doc id + the subspace range to be matched (ie, the embedding Type range), for example, the searched vector table doc id: -18291211L, the target subspace embedding Type is greater than or equal to 20000L and less than or equal to 29999L, and the matching TopN department labels are quickly and accurately queried (Query).

[0180] The embodiment of the present application introduces the design of a general large language model and a medical knowledge graph to achieve the integration of extensive general knowledge and deep professional medical knowledge. The introduction of a full or incremental data synchronization mechanism, as well as the design of a vector embedding model and a clustering recall technology for a vector library, achieves a simplified model training and maintenance process, and also ensures that the annotation system is automatically synchronized with the latest medical knowledge. The optimized medical vector library table design and IVFPQ vector search solution design achieve high performance and high efficiency in the annotation process. The design of the vector embedding model and Vearch vector library improves the accuracy of the semantic understanding of complex and ambiguous medical terms, and solves the shortcomings of traditional methods in the processing of professional medical terms.

[0181] The embodiment of the present application realizes automatic labeling based on the multi-dimensional medical content of the large language model, vector embedding model, Vearch vector library and XX Health Medical Knowledge Graph, and synchronizes the upstream symptom, disease, and drug database to the Vearch vector library in full or incremental manner, realizing efficient design and efficient retrieval of medical vector library tables. Fusion of general and professional knowledge: By combining the extensive general knowledge of ChatGPT 3.5 Turbo and the deep professional knowledge of XX Health Medical Knowledge Graph, it can ensure that the annotation not only accurately reflects the general understanding of medical content, but also conforms to the professional standards and latest developments in the medical field.

[0182] The embodiment of the present application realizes low-cost machine learning annotation of large-scale data sets. By automating the annotation process, the need for manual participation is significantly reduced, thereby reducing manpower and time costs, making it possible to quickly annotate tens of millions of data. Automatically internalize upstream information sources: including methods for automatically synchronizing upstream symptom, disease, and drug databases through full or incremental big data methods, ensuring that the annotation system is always synchronized with the latest medical knowledge, enhancing the adaptability and accuracy of the system. Easy to maintain: The automatic annotation system based on vector embedding and clustering recall technology simplifies the model training and update process and reduces maintenance costs. High accuracy: Using the clustering recall algorithm of the vector embedding model and the Vearch vector library, the accuracy of semantic understanding in the annotation process is improved when dealing with complex and ambiguous medical terms. High performance and high efficiency: By adopting the IVFPQ vector recall algorithm and the optimized medical vector library table design, high-speed data processing and retrieval are achieved, ensuring high performance and high efficiency on large-scale data sets. It solves the problems of accuracy, efficiency, cost, and maintenance difficulty of automatic annotation of medical content tags, and provides an efficient, low-cost and easy-to-maintain solution for the field of medical information management and retrieval.

[0183] Figure 8 Schematic diagram of the main units of the medical data processing device according to an embodiment of the present application. Figure 8 As shown, the medical data processing device 800 includes a receiving unit 801 , an acquiring unit 802 , a first matching unit 803 , a second matching unit 804 and a calculating unit 805 .

[0184] The receiving unit 801 is configured to receive a data processing request, obtain medical text data, and extract abnormal entities in the medical text data.

[0185] The acquisition unit 802 is configured to acquire matched structured abnormal data based on the abnormal entity matching vector library table.

[0186] The first matching unit 803 is configured to obtain first matching result data based on the structured abnormal data and the knowledge graph.

[0187] The second matching unit 804 is configured to vectorize the titles in the medical text data to obtain a title vector, and match the title vector with each vector subspace in the vector library table based on an inverted index product quantization index method to obtain second matching result data.

[0188] The calculation unit 805 is configured to obtain comprehensive decision result data by linear combination calculation of weighted features based on the first matching result data and the second matching result data.

[0189] In some embodiments, the receiving unit 801 is further configured to: extract abnormal content data from the medical text data; generate guiding prompt words based on the abnormal content data; and input the guiding prompt words into the large language model to obtain corresponding abnormal entities.

[0190] In some embodiments, the acquisition unit 802 is further configured to: perform vectorization processing on the abnormal entity to obtain an abnormal entity vector; calculate the similarity between the abnormal entity vector and each abnormal vector in the vector library table, and determine the target abnormal vector based on the similarity and a preset threshold; determine the label corresponding to the target abnormal vector in the vector library table as structured abnormal data matching the abnormal entity.

[0191] In some embodiments, the first matching unit 803 is further configured to: apply the structured exception data as a query condition to the knowledge graph to obtain corresponding first matching result data.

[0192] In some embodiments, the second matching unit 804 is further configured to: cluster the title vector and each vector subspace in the vector library table based on the inverted index to obtain each clustering cluster; for each clustering cluster, perform product quantization on the corresponding title vector and vector subspace to obtain second matching result data.

[0193] In some embodiments, the calculation unit 805 is further configured to: obtain the weight value corresponding to the feature function corresponding to each label in the first matching result data and the second matching result data; for each label, calculate the product of the feature function and the weight value; based on the product, determine the comprehensive decision value of the corresponding label; and determine the comprehensive decision result data based on each comprehensive decision value.

[0194] In some embodiments, the medical data processing apparatus further includes Figure 8 The update unit not shown in the figure is configured to: in response to detecting a data change in the relational data temporary storage library, determine the changed data through the logical log mechanism; generate a message queue based on the changed data, and send the message queue to the message middleware in real time; call the business application to consume the data change message from the message middleware in real time according to the preset message queue consumption rules, execute the corresponding business logic processing, and obtain the business logic processing result; update the vector library table according to the business logic processing result.

[0195] It should be noted that the medical data processing method and the medical data processing device of the present application have a corresponding relationship in terms of specific implementation contents, so the repeated contents will not be described again.

[0196] Fig. 9 An exemplary system architecture 900 is shown to which the medical data processing method or medical data processing device according to the embodiments of the present application can be applied.

[0197] like Fig. 9 As shown, system architecture 900 may include terminal devices 901, 902, 903, network 904 and server 905. Network 904 is used to provide a medium for communication links between terminal devices 901, 902, 903 and server 905. Network 904 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0198] The user can use the terminal devices 901, 902, 903 to interact with the server 905 through the network 904 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 901, 902, 903, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0199] The terminal devices 901 , 902 , and 903 may be various electronic devices having a data processing screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0200] Server 905 may be a server that provides various services, such as a background management server that provides support for data processing requests submitted by users using terminal devices 901, 902, and 903 (only as an example). The background management server may receive data processing requests, obtain medical text data, and extract abnormal entities in the medical text data; match the vector library table based on the abnormal entity to obtain the matched structured abnormal data; obtain the first matching result data based on the structured abnormal data and the knowledge graph; vectorize the title in the medical text data to obtain the title vector, match the title vector with each vector subspace in the vector library table based on the inverted index product quantization index method to obtain the second matching result data; based on the first matching result data and the second matching result data, obtain the comprehensive decision result data through the linear combination calculation of weighted features. Improve the accuracy and reliability of data processing related to data annotation, thereby improving the efficiency of users' retrieval and use of information based on the annotated data.

[0201] It should be noted that the medical data processing method provided in the embodiment of the present application is generally executed by the server 905 , and accordingly, the medical data processing device is generally set in the server 905 .

[0202] It should be understood that Fig. 9 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0203] Reference below Fig.10, which shows a schematic diagram of the structure of a computer system 1000 of a terminal device suitable for implementing an embodiment of the present application. Fig.10 The terminal device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0204] like Fig.10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage part 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the computer system 1000 are also stored. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0205] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed, so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0206] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1009, and / or installed from the removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the above-mentioned functions defined in the system of the present application are executed.

[0207] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable storage media may include, for example, but are not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0208] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0209] The units involved in the embodiments described in the present application may be implemented by software or hardware. The units described may also be set in a processor, for example, it may be described as: a processor includes a receiving unit, an acquisition unit, a first matching unit, a second matching unit, and a computing unit. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0210] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device receives a data processing request, obtains medical text data, extracts abnormal entities in the medical text data; matches the vector library table based on the abnormal entity to obtain the matched structured abnormal data; obtains the first matching result data based on the structured abnormal data and the knowledge graph; vectorizes the title in the medical text data to obtain the title vector, matches the title vector with each vector subspace in the vector library table based on the inverted index product quantization index method to obtain the second matching result data; based on the first matching result data and the second matching result data, obtains the comprehensive decision result data through the linear combination calculation of weighted features.

[0211] The computer program product of the present application includes a computer program, and when the computer program is executed by a processor, the medical data processing method in the embodiment of the present application is implemented.

[0212] According to the technical solution of the embodiment of the present application, the accuracy and reliability of data processing related to data annotation can be improved, thereby improving the efficiency of users' retrieval and use of information based on annotated data.

[0213] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A medical data processing method, characterized in that: include: receiving a data processing request, acquiring medical text data, and extracting abnormal entities from the medical text data; Based on the abnormal entity matching vector library table, to obtain matched structured abnormal data; Based on the structured abnormal data and the knowledge graph, obtaining first matching result data; Vectorizing the titles in the medical text data to obtain title vectors, matching the title vectors with each vector subspace in the vector library table based on an inverted index product quantization indexing method to obtain second matching result data; Based on the first matching result data and the second matching result data, comprehensive decision result data is obtained by calculating a linear combination of weighted features.

2. The method according to claim 1, characterized in that: The extracting of abnormal entities from the medical text data comprises: Extracting abnormal content data from the medical text data; Based on the abnormal content data, generating a guiding prompt word; The guiding prompt word is input into the large language model to obtain the corresponding abnormal entity.

3. The method according to claim 1, characterized in that The matching vector library table based on the abnormal entity to obtain the matched structured abnormal data includes: Performing vectorization processing on the abnormal entity to obtain an abnormal entity vector; Calculate the similarity between the abnormal entity vector and each abnormal vector in the vector library table, and determine the target abnormal vector based on the similarity and a preset threshold; The label corresponding to the target abnormal vector in the vector library table is determined as the structured abnormal data matching the abnormal entity.

4. The method according to claim 1, characterized in that The obtaining of first matching result data based on the structured abnormal data and the knowledge graph includes: The structured abnormal data is used as a query condition and applied to the knowledge graph to obtain corresponding first matching result data.

5. The method according to claim 1, characterized in that The obtaining of the second matching result data includes: Clustering the title vector and each vector subspace in the vector library table based on the inverted index to obtain each clustering cluster; For each cluster, product quantization is performed on the corresponding title vector and vector subspace to obtain second matching result data.

6. The method according to claim 1, characterized in that The step of obtaining comprehensive decision result data by calculating a linear combination of weighted features based on the first matching result data and the second matching result data includes: Obtaining a weight value corresponding to a feature function corresponding to each label in the first matching result data and the second matching result data; For each of the labels, calculating the product of the feature function and the weight value; Determine the comprehensive decision value of the corresponding label according to the product; According to each of the comprehensive decision values, comprehensive decision result data is determined.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: In response to detecting a data change in the relational data temporary repository, determining the changed data through a logical log mechanism; Based on the changed data, a message queue is generated, and the message queue is sent to the message middleware in real time; Calling the business application to consume data change messages from the message middleware in real time according to the preset message queue consumption rules, executing corresponding business logic processing, and obtaining business logic processing results; According to the business logic processing result, the vector library table is updated.

8. A medical data processing device, characterized in that: include: A receiving unit configured to receive a data processing request, obtain medical text data, and extract abnormal entities from the medical text data; An acquisition unit is configured to acquire matched structured abnormal data based on the abnormal entity matching vector library table; A first matching unit is configured to obtain first matching result data based on the structured abnormal data and the knowledge graph; A second matching unit is configured to vectorize the title in the medical text data to obtain a title vector, and match the title vector with each vector subspace in the vector library table based on an inverted index product quantization index method to obtain second matching result data; The calculation unit is configured to obtain comprehensive decision result data by linear combination calculation of weighted features based on the first matching result data and the second matching result data.

9. A data processing electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • Knowledge base updating and deleting method and system based on industry large model

    CN119807214A

  • Medical term matching method and device based on double paths

    CN121786502A