Database construction method for national medicine characteristic multi-diagnosis combined clinical auxiliary diagnosis

Abstract

The invention discloses a database construction method for national medicine characteristic multi-diagnosis combined clinical auxiliary diagnosis, and relates to the field of database construction. According to the method, before the semantic features of the Transformer-style model are expressed, the modal conversion algorithm is constructed based on the deep network to carry out modal conversion, dimensionality reduction and feature expression are carried out on the multi-modal data, and meanwhile, the application field of the deep learning theory is also expanded; on the basis of a traditional character recognition and data structuring algorithm, a vector representation encoder of diagnosis and treatment data is reused, and vectorized feature expression with semantic information is generated; the recognized characters are matched with corresponding key fields and contents based on a certain rule, and a structured ethnic medicine knowledge base is generated; the data is text content and can reuse a vector feature encoder of doctor seeing data, key fields and content are vectorized after training, then the key fields and the content are combined and stored in a vector database, and a national medical knowledge base is formed.

Description

Technical Field

[0001] The present invention relates to the field of database construction, and in particular to a method for constructing a database for multi-diagnosis and clinical auxiliary diagnosis with ethnic medicine characteristics. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0003] Due to the complexity and diversity of business in the medical field, the multi-diagnosis and multi-reference data obtained are multi-source, heterogeneous, multi-type, and multi-modal. Traditional relational databases can only store text information, such as picture names, picture addresses, text content, etc., and for unstructured or semi-structured data (such as text, images, etc.), relational databases are not the best storage option. In addition, with the rapid development of artificial intelligence, extracting semantic information of multimodal data in the same representation space is extremely important for model construction and training. However, relational databases cannot meet these characteristics and requirements of multimodal data.

[0004] In recent years, with the continuous development of deep models, the models have gradually transitioned from the initial RNN and LSTM to the current Transformer-style models based on Attention and Transformer architecture. The semantics have also changed from the early word meaning expression to the current joint expression of lexical, word meaning, syntax, and sentence meaning. The modal type has also been upgraded from processing text to processing image data, which provides a new idea for constructing semantic information fusion storage of multi-diagnosis data.

[0005] However, there are many types of information systems in the medical field, and multi-diagnosis data are not just text and image modalities, but also include physiological signals and other types. It is still a bit difficult for a simple Transformer-style model to directly process this data.

[0006] Some data modalities can be represented in the same representation space through spatial transformation methods. For example, spectrograms and visual diagnosis images can be processed as image modalities, and diagnostic results can be processed as text models.

[0007] In addition to diagnosis and treatment data, the electronic storage and access of sorted and interpreted ethnic medical literature (including ancient books, classics, prescriptions, medical records of famous doctors, etc.) is also an important part of medical data governance. Traditional literature directly stores image addresses or links, or first converts image data into text data and then stores it in a relational database. Similar to the problems faced by multi-diagnosis and treatment data, original or initially converted data is difficult to effectively cope with the training of complex auxiliary diagnosis models. Summary of the invention

[0008] The purpose of the present invention is to provide a method for constructing a database for multi-diagnosis and clinical auxiliary diagnosis with ethnic medicine characteristics in view of the problems existing in the prior art, thereby solving the above problems.

[0009] The technical solution of the present invention is as follows:

[0010] A method for constructing a database for clinical auxiliary diagnosis with ethnic medicine characteristics, the database comprising: a vector database formed by historical multi-diagnosis data and a vectorized ethnic medicine diagnosis and treatment knowledge base formed by ethnic medicine literature;

[0011] The method for constructing the vector database comprises:

[0012] Step S1: converting the three modalities of text data, physiological signal data, and image data in the historical multi-diagnosis combined data into two modalities of text data and image data;

[0013] Step S2: For the converted text data and image data, use the constructed Transformer-style model to extract the semantic information represented by the vector;

[0014] Step S3: storing the semantic information in combination with the metadata into a vector database;

[0015] The method for constructing the vectorized ethnic medicine diagnosis and treatment knowledge base includes:

[0016] Step I: Use text recognition algorithms to identify text information in ethnic medicine literature;

[0017] Step II: Use data structuring algorithms to process text information and generate structured ethnic medical diagnosis and treatment knowledge;

[0018] Step III: For the generated structured ethnic medicine diagnosis and treatment knowledge, reuse the constructed Transformer-style model to extract the semantic information of the vector representation to form a vectorized ethnic medicine diagnosis and treatment knowledge base.

[0019] Furthermore, the step S1 comprises:

[0020] Step S11: collect text data, physiological signal data and image data, and perform data cleaning on them to eliminate noise and outliers in the data;

[0021] Step S12: Convert the physiological signal data into image data through a data modality conversion model.

[0022] Furthermore, the data cleaning includes:

[0023] For text data and physiological signal data, firstly, data with missing field values ​​and duplicate data are deleted, and then the minimum and maximum normalization is used to standardize the values ​​of each field to ensure the accuracy and reliability of the data;

[0024] For image data, we first delete images with a lot of noise information, and then scale the sizes of each image to a uniform size.

[0025] Furthermore, the step S12 includes:

[0026] For physiological signal data, fast Fourier transform is used to transform the time domain signal into the frequency domain signal to generate the corresponding spectrum image, and then the size of each spectrum image is scaled to a uniform size.

[0027] Furthermore, the step S2 comprises:

[0028] Step S21: constructing a medical image-text bimodal Embedding model based on the CLIP model, text data, and image data;

[0029] Step S22: In the reasoning stage, the constructed medical image-text bimodal Embedding model is used to vectorize the input text data and image data to extract the semantic information represented by the vectors.

[0030] Furthermore, the construction process of the medical image-text bimodal Embedding model is as follows:

[0031] Encode part of the text content on the patient's medical record into a 512-dimensional One-Hot vector, i.e., a text vector;

[0032] At the same time, the inspection image corresponding to the text content is cut into 512 image blocks, and the median pixel value of each image block is taken to form a total of 512 pixel values, and then the pixel values ​​are flattened into a 1*512-dimensional vector, namely the image vector;

[0033] A text-image pair is constructed based on the text vector and the image vector;

[0034] The text vector and image vector in each text-image pair are then concatenated and input into the CLIP model for training until the model converges to obtain a medical image-text bimodal Embedding model.

[0035] Furthermore, the metadata includes: patient name, gender, and age.

[0036] Furthermore, for the heterogeneous data in historical multi-diagnosis data and ethnic medicine literature, they are input into the encoder for vectorization to generate multi-dimensional vector features; and then the decoder is used for image reconstruction or text reconstruction.

[0037] Furthermore, the encoder is an Embedding model.

[0038] Furthermore, the decoder adopts the Transformer architecture in the MAE model for image reconstruction, that is, generating a large amount of high-quality data with normalized pixel values ​​or text reconstruction, that is, filling in missing keywords and correcting text semantics. At the same time, it can generate a large number of medical record description contents with different expressions of the same semantics.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. Before expressing the semantic features of the Transformer-style model, the present invention constructs a modal conversion algorithm based on a deep network to perform modal conversion, reduce the dimension and express the features of multimodal data, and also expand the application field of deep learning theory.

[0041] 2. Based on traditional text recognition and data structuring algorithms, the present invention reuses the vector representation encoder of medical data to generate vectorized feature expressions with semantic information; the recognized text is based on certain rules, such as prescription name, medicine name, symptoms, dosage method and other information, and the corresponding key fields and content are matched to generate a structured ethnic medicine knowledge base; the data is text content, and the vector feature encoder of medical data can be reused. After training, the key fields and content are vectorized, and then combined and stored in a vector database to form an ethnic medicine knowledge base.

[0042] 3. The present invention also proposes a deep network model based on adversarial generation and few-sample learning, which aims to use the encoder-decoder method to represent the semantic features of multimodal data, restore the original multimodal data or virtualize a large amount of high-quality labeled data, so that the generative adversarial network can automatically learn the distribution representation of different modal data from fewer labeled samples, and generate a mapping relationship between heterogeneous data and homogeneous data after quality unification. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flowchart of a method for constructing a vector database;

[0044] Figure 2 This is a flowchart of the method for constructing a quantitative ethnic medicine diagnosis and treatment knowledge base;

[0045] Figure 3 Flowchart for heterogeneous data processing. DETAILED DESCRIPTION

[0046] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0047] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0048] Embodiment 1

[0049] A method for constructing a database for multi-diagnosis and clinical auxiliary diagnosis with ethnic medicine characteristics, a method for constructing a database for multi-diagnosis and clinical auxiliary diagnosis with ethnic medicine characteristics, the database comprising: a vector database formed by historical multi-diagnosis and clinical auxiliary data and a vectorized ethnic medicine diagnosis and treatment knowledge base formed by ethnic medicine literature materials;

[0050] See also Figure 1 , the method for constructing the vector database comprises:

[0051] Step S1: converting the three modalities of text data, physiological signal data, and image data in the historical multi-diagnosis combined data into two modalities of text data and image data;

[0052] Step S2: For the converted text data and image data, use the constructed Transformer-style model to extract the semantic information represented by the vector;

[0053] Step S3: The semantic information is combined with metadata (which can be defined according to business needs, such as patient name, gender, age, etc.) and stored in a vector database; it should be noted that the Transformer-style model has achieved good results in expressing semantic information of text and image modal data, but cannot process medical data of more modalities. Therefore, this embodiment introduces the modal conversion algorithm into the semantic feature expression of the Transformer-style model, thereby solving the difficulty of "not being able to express semantic information in the same representation space" encountered when storing multimodal medical data;

[0054] See also Figure 2 The method for constructing the vectorized ethnic medicine diagnosis and treatment knowledge base includes:

[0055] Step I: Use text recognition algorithms to identify text information in ethnic medicine literature;

[0056] Step II: Use data structuring algorithms to process text information and generate structured ethnic medical diagnosis and treatment knowledge;

[0057] Step III: For the generated structured ethnic medicine diagnosis and treatment knowledge, reuse the constructed Transformer-style model to extract the semantic information of the vector representation to form a vectorized ethnic medicine diagnosis and treatment knowledge base.

[0058] In this embodiment, specifically, step S1 includes:

[0059] Step S11: collect text data, physiological signal data and image data, and perform data cleaning on them to eliminate noise and outliers in the data; improve data quality so that subsequent deep network model training can converge quickly;

[0060] Step S12: Convert the physiological signal data into image data through a data modality conversion model.

[0061] In this embodiment, specifically, the data cleaning includes:

[0062] For text data and physiological signal data, firstly, data with missing field values ​​(field values ​​are empty or NaN) and duplicate data are deleted, and then the minimum and maximum normalization is used to standardize the values ​​of each field to ensure the accuracy and reliability of the data;

[0063] For image data, first delete images with a lot of noise information (blurred, blocked, missing, etc.), and then scale the size of each image to a uniform size (such as 512*512).

[0064] In this embodiment, specifically, step S12 includes:

[0065] For physiological signal data, Fast Fourier Transform (FFT) is used to transform the time domain signal into the frequency domain signal to generate the corresponding spectrum image, and then the size of each spectrum image is scaled to a uniform size (such as 512*512).

[0066] In this embodiment, specifically, step S2 includes:

[0067] Step S21: constructing a medical image-text bimodal Embedding model based on the CLIP (Contrastive Language-Image Pre-training) model, text data, and image data;

[0068] Step S22: In the reasoning stage, the constructed medical image-text bimodal Embedding model is used to vectorize the input text data and image data to extract the semantic information represented by the vectors.

[0069] In this embodiment, specifically, the construction process of the medical image-text bimodal Embedding model is as follows:

[0070] Encode part of the text content in the patient's medical record (e.g., white and greasy tongue coating) into a 512-dimensional One-Hot vector, i.e., a text vector;

[0071] At the same time, the inspection image corresponding to the text content (such as the greasy moss image) is cut into 512 image blocks, and the median pixel value of each image block is taken to form a total of 512 pixel values, and then the pixel values ​​are flattened into a 1*512-dimensional vector, namely the image vector;

[0072] A text-image pair is constructed based on the text vector and the image vector;

[0073] The text vector and image vector in each text-image pair are then concatenated and input into the CLIP model for training until the model converges to obtain a medical image-text bimodal Embedding model (i.e., the loss function no longer decreases significantly and the model tends to be stable. The convergence state varies depending on factors such as data and hyperparameters and is not limited).

[0074] Medical images for clinical applications are usually multi-source heterogeneous data. Even for the same modality data, the medical data generated are not of the same quality due to factors such as imaging mechanism, configuration differences, and environmental influences. For example, the spatial resolution and density resolution of medical image data vary greatly. Such heterogeneous data make it difficult to train auxiliary diagnosis models with high generalization capabilities, thus affecting the ultimate use effect of the auxiliary diagnosis model.

[0075] Therefore, it is extremely important to reconstruct multi-source heterogeneous data and keep the data quality at the same standard. Traditional image interpolation algorithms and data missing value filling algorithms process data consistently under certain rules and cannot adaptively distinguish and treat data of different modalities. At the same time, there are few high-quality medical annotations, which makes it difficult to promote deep feature learning.

[0076] Therefore, this embodiment also proposes a deep network model based on adversarial generation and few-sample learning, which aims to use the encoder-decoder method to represent the semantic features of multimodal data, restore the original multimodal data or virtualize a large amount of high-quality annotated data, so that the generative adversarial network can automatically learn the distribution representation of different modal data from fewer annotated samples, and generate a mapping relationship between heterogeneous data and homogeneous data after quality unification, such as Figure 3 As shown, the model can be applied to data of different modalities such as images and texts;

[0077] Specifically, for the heterogeneous data in historical multi-diagnosis data and ethnic medicine literature (data quality is not uniform, such as different image resolutions, missing words in medical record text descriptions causing ambiguity, etc.), they are input into the encoder for vectorization to generate 512-dimensional vector features; among them, the encoder is an Embedding model, and multimodal models such as CLIP (Contrastive Language-Image Pre-training) can be used without limitation;

[0078] The decoder is then used to reconstruct the image or text. The decoder uses the Transformer architecture (8 layers, 512 nodes) in the MAE (Masked Auto Encoder) model for image reconstruction, i.e., generating a large amount of high-quality data with normalized pixel values ​​or text reconstruction, i.e., filling in missing keywords and correcting text semantics. At the same time, it can generate a large amount of medical record descriptions with different expressions of the same semantics.

[0079] It should be noted that heterogeneous and small-sample data make it difficult to train auxiliary diagnosis models with high generalization capabilities. The use of adversarial sample generation technology can expand the number of samples, unify the sample quality distribution, and train models in a targeted manner. The introduction of vector-represented semantic feature information can give full play to the advantages of multi-diagnosis data feature fusion, and is expected to overcome the difficulty of unifying data quality using the same model for multimodal data.

[0080] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

[0081] This background section is provided to generally present the context of the invention, and the work of the presently named inventors, the work to the extent described in this background section, and aspects of this section that did not constitute prior art at the time of application are neither explicitly nor implicitly admitted to be prior art to the present invention.

Claims

1. A method for constructing a database for multi-diagnosis and clinical auxiliary diagnosis with ethnic medicine characteristics, characterized in that: The database includes: a vector database formed by historical multi-diagnosis and reference data and a vectorized ethnic medicine diagnosis and treatment knowledge base formed by ethnic medicine literature; The method for constructing the vector database comprises: Step S1: converting the three modalities of text data, physiological signal data, and image data in the historical multi-diagnosis combined data into two modalities of text data and image data; Step S2: For the converted text data and image data, use the constructed Transformer-style model to extract the semantic information represented by the vector; Step S3: storing the semantic information in combination with the metadata into a vector database; The method for constructing the vectorized ethnic medicine diagnosis and treatment knowledge base includes: Step I: Use text recognition algorithms to identify text information in ethnic medicine literature; Step II: Use data structuring algorithms to process text information and generate structured ethnic medical diagnosis and treatment knowledge; Step III: For the generated structured ethnic medicine diagnosis and treatment knowledge, reuse the constructed Transformer-style model to extract the semantic information of the vector representation to form a vectorized ethnic medicine diagnosis and treatment knowledge base.

2. The method for constructing a database for multi-diagnosis and clinical auxiliary diagnosis with ethnic medicine characteristics according to claim 1, characterized in that: The step S1 comprises: Step S11: collect text data, physiological signal data and image data, and perform data cleaning on them to eliminate noise and outliers in the data; Step S12: Convert the physiological signal data into image data through a data modality conversion model.

3. The method for constructing a database for multi-diagnosis and clinical auxiliary diagnosis with ethnic medicine characteristics according to claim 2, characterized in that: The data cleaning includes: For text data and physiological signal data, first delete the data with missing field values ​​and duplicate data, and then use minimum and maximum normalization to standardize the values ​​of each field to ensure the accuracy and reliability of the data; For image data, we first delete images with a lot of noise information, and then scale the sizes of each image to a uniform size.

4. The method for constructing a database for multi-diagnosis and clinical auxiliary diagnosis with ethnic medicine characteristics according to claim 2, characterized in that: The step S12 comprises: For physiological signal data, fast Fourier transform is used to transform the time domain signal into the frequency domain signal to generate the corresponding spectrum image, and then the size of each spectrum image is scaled to a uniform size.

5. The method for constructing a database for multi-diagnosis and clinical auxiliary diagnosis with ethnic medicine characteristics according to claim 1, characterized in that: The step S2 comprises: Step S21: constructing a medical image-text bimodal Embedding model based on the CLIP model, text data, and image data; Step S22: In the reasoning stage, the constructed medical image-text bimodal Embedding model is used to vectorize the input text data and image data to extract the semantic information represented by the vectors.

6. The method for constructing a database for multi-diagnosis and clinical auxiliary diagnosis with ethnic medicine characteristics according to claim 5, characterized in that: The construction process of the medical image-text bimodal Embedding model is as follows: Encode part of the text content on the patient's medical record into a 512-dimensional One-Hot vector, i.e., a text vector; At the same time, the inspection image corresponding to the text content is cut into 512 image blocks, and the median pixel value of each image block is taken to form a total of 512 pixel values, and then the pixel values ​​are flattened into a 1*512-dimensional vector, namely the image vector; A text-image pair is constructed based on the text vector and the image vector; The text vector and image vector in each text-image pair are then concatenated and input into the CLIP model for training until the model converges to obtain a medical image-text bimodal Embedding model.

7. The method for constructing a database for multi-diagnosis and clinical auxiliary diagnosis with ethnic medicine characteristics according to claim 1, characterized in that: The metadata includes: patient name, gender, and age.

8. The method for constructing a database for multi-diagnosis and clinical auxiliary diagnosis with ethnic medicine characteristics according to claim 1, characterized in that: For the heterogeneous data in historical multi-diagnosis data and ethnic medicine literature, they are input into the encoder for vectorization to generate multi-dimensional vector features; The decoder is then used to reconstruct the image or text.

9. The method for constructing a database for multi-diagnosis and clinical auxiliary diagnosis with ethnic medicine characteristics according to claim 8, characterized in that: The encoder is an Embedding model.

10. The method for constructing a database for multi-diagnosis and clinical auxiliary diagnosis with ethnic medicine characteristics according to claim 8, characterized in that: The decoder adopts the Transformer architecture in the MAE model for image reconstruction, that is, generating a large amount of high-quality data with normalized pixel values ​​or text reconstruction, that is, filling in missing keywords and correcting text semantics. At the same time, it can generate a large number of medical record description contents with different expressions of the same semantics.

Images