Multi-diagnosis combined clinical auxiliary diagnosis method for national medicine characteristics

Abstract

The invention discloses a national medicine characteristic multi-diagnosis combined clinical auxiliary diagnosis method, and relates to the field of medical diagnosis, and the method comprises the steps: S1, forming a vector database through historical multi-diagnosis combined parameter data, and forming a vectorized national medicine diagnosis and treatment knowledge base through national medicine literature data; s2, on the basis of the vector database and the vectorized ethnic medicine diagnosis and treatment knowledge base, constructing an ethnic medicine characteristic multi-diagnosis combined clinical auxiliary diagnosis model; and S3, inputting the acquired text data or image data into a national medicine characteristic multi-diagnosis combined clinical auxiliary diagnosis model for diagnosis. According to the method, the problems of subjectivity, information integration, knowledge utilization, report generation and the like in national medicine clinical diagnosis can be solved, and national medicine is promoted to develop towards the directions of standardization, objectification and intelligence.

Description

Technical Field

[0001] The present invention relates to the field of medical diagnosis, and in particular to a multi-diagnosis and clinical auxiliary diagnosis method with ethnic medicine characteristics. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0003] In the current field of medical diagnosis, ethnic medicine, as an important traditional medical system, has unique diagnosis and treatment methods and rich clinical experience. However, the diagnosis process of ethnic medicine often relies on the personal experience and subjective judgment of doctors, and lacks standardized and objective diagnostic tools, which to a certain extent limits the inheritance and development of ethnic medicine. Specifically, there are the following technical problems:

[0004] 1. Subjectivity in clinical diagnosis of ethnic medicine: Traditional ethnic medicine diagnosis is highly dependent on the doctor’s personal experience and intuition, which results in differences in diagnostic results between different doctors and lacks uniform standards and objectivity.

[0005] 2. Inadequate processing of text and image data: In the diagnosis process of ethnic medicine, medical records and medical images are important diagnostic bases. However, existing technologies lack effective means to convert text and image data into high-dimensional vector representations that can be used for in-depth analysis, which limits the application of computer-aided diagnosis systems.

[0006] 3. Difficulty in integrating information from multiple diagnoses: Ethnic medicine emphasizes multiple diagnoses, that is, combining multiple diagnostic methods to improve the accuracy of diagnosis. However, how to effectively integrate qualitative and quantitative information from different diagnostic methods is a problem that has not yet been fully solved.

[0007] 4. Insufficient utilization of knowledge base: Ethnic medicine has a large amount of diagnostic and treatment knowledge and historical cases, but this knowledge has not been fully utilized in the auxiliary diagnosis process and lacks an effective retrieval and matching mechanism.

[0008] 5. Automation and standardization of diagnostic report generation: Currently, diagnostic reports are mostly written manually by doctors, which is not only inefficient but also difficult to ensure the standardization and quality of the reports. Summary of the invention

[0009] The purpose of the present invention is to provide a clinical auxiliary diagnosis method with multi-diagnosis and combined reference with ethnic medicine characteristics to solve the above problems in view of the problems existing in the prior art.

[0010] The technical solution of the present invention is as follows:

[0011] A clinical auxiliary diagnosis method with ethnic medicine characteristics of multiple diagnosis and reference, including:

[0012] Step S1: forming a vector database from historical multi-diagnosis data and forming a vectorized ethnic medicine diagnosis and treatment knowledge base from ethnic medicine literature;

[0013] Step S2: Based on the vector database and the vectorized ethnic medicine diagnosis and treatment knowledge base, a clinical auxiliary diagnosis model with ethnic medicine characteristics and multi-diagnosis is constructed;

[0014] Step S3: input the acquired text data or image data into the ethnic medicine characteristic multi-diagnosis and reference clinical auxiliary diagnosis model for diagnosis.

[0015] Furthermore, the method for constructing the vector database includes:

[0016] Step A: converting the three modalities of text data, physiological signal data, and image data in the historical multi-diagnosis data into two modalities of text data and image data;

[0017] Step B: For the converted text data and image data, use the constructed Transformer-style model to extract the semantic information represented by the vector;

[0018] Step C: storing the semantic information combined with metadata into the vector database;

[0019] The method for constructing the vectorized ethnic medicine diagnosis and treatment knowledge base includes:

[0020] Step I: Use text recognition algorithms to identify text information in ethnic medicine literature;

[0021] Step II: Use data structuring algorithms to process text information and generate structured ethnic medical diagnosis and treatment knowledge;

[0022] Step III: For the generated structured ethnic medicine diagnosis and treatment knowledge, reuse the constructed Transformer-style model to extract the semantic information of the vector representation to form a vectorized ethnic medicine diagnosis and treatment knowledge base.

[0023] Furthermore, the step A comprises:

[0024] Step A1: collect text data, physiological signal data and image data, and perform data cleaning to eliminate noise and outliers in the data;

[0025] Step A2: converting physiological signal data into image data through a data modality conversion model;

[0026] The data cleaning includes:

[0027] For text data and physiological signal data, first delete the data with missing field values ​​and duplicate data, and then use minimum and maximum normalization to standardize the values ​​of each field to ensure the accuracy and reliability of the data;

[0028] For image data, first delete the images with a lot of noise information, and then scale the size of each image to a uniform size;

[0029] The step A2 comprises:

[0030] For physiological signal data, fast Fourier transform is used to transform the time domain signal into the frequency domain signal to generate the corresponding spectrum image, and then the size of each spectrum image is scaled to a uniform size.

[0031] Furthermore, the step B comprises:

[0032] Step B1: Construct a medical image-text bimodal Embedding model based on the CLIP model, text data, and image data;

[0033] Step B2: In the reasoning stage, the constructed medical image-text bimodal Embedding model is used to vectorize the input text data and image data to extract the semantic information represented by their vectors;

[0034] The construction process of the medical image-text bimodal Embedding model is as follows:

[0035] Encode part of the text content on the patient's medical record into a 512-dimensional One-Hot vector, i.e., a text vector;

[0036] At the same time, the inspection image corresponding to the text content is cut into 512 image blocks, and the median pixel value of each image block is taken to form a total of 512 pixel values, and then the pixel values ​​are flattened into a 1*512-dimensional vector, namely the image vector;

[0037] A text-image pair is constructed based on the text vector and the image vector;

[0038] Then, the text vector and image vector in each text-image pair are concatenated and input into the CLIP model for training until the model converges to obtain a medical image-text bimodal embedding model.

[0039] Furthermore, the metadata includes: patient name, gender, and age.

[0040] Furthermore, the ethnic medicine characteristic multi-diagnosis and clinical auxiliary diagnosis model includes:

[0041] The text embedding model and image embedding model can respectively vectorize text data and image data in the same representation space;

[0042] The Transformer encoder, the classification head and regression head of the fine-grained model, and the Transformer decoder are used to extract qualitative and quantitative information of the examination site from the vectorized data, and then embedded in the Transformer decoder to decode the preliminary diagnosis results; based on the preliminary diagnosis results, relevant diagnosis and treatment information is retrieved from the vector database and the vectorized ethnic medicine diagnosis and treatment knowledge base through similarity metric learning;

[0043] The language model of the National Medical University inputs the preliminary diagnosis results and related diagnosis and treatment information into the language model of the National Medical University and outputs a standardized diagnosis report.

[0044] Furthermore, the ethnic medicine-specific multi-diagnosis and integrated clinical auxiliary diagnosis model adopts a grouped multi-head attention calculation method to reduce the complexity of attention mechanism calculation.

[0045] Furthermore, the grouped multi-head attention includes:

[0046] For each group X, three 1*3 matrices WQ, WK, and WV are randomly initialized, and X*WQ, X*WK, and X*WV are calculated respectively to obtain Q, K, and V; then according to the formula Calculate the self-attention, and the obtained vector is flattened to Z; finally, add the four sets of results Z1, Z2, Z3, and Z4 to get Z sum .

[0047] Furthermore, heterogeneous data in historical multi-diagnosis data, ethnic medicine literature, text data and image data are input into the encoder for vectorization to generate multi-dimensional vector features; then the decoder is used to perform image reconstruction or text reconstruction; the encoder is an Embedding model.

[0048] Furthermore, the decoder adopts the Transformer architecture in the MAE model for image reconstruction, that is, generating a large amount of high-quality data with normalized pixel values ​​or text reconstruction, that is, filling in missing keywords and correcting text semantics. At the same time, it can generate a large number of medical record description contents with different expressions of the same semantics.

[0049] Compared with the prior art, the present invention has the following beneficial effects:

[0050] 1. The present invention uses Transformer and CNN to mine the key relationships between multimodal medical data, and realizes the consistency of data input and the guidance of image features in the model to text content in a single unified model architecture front end. At the same time, the construction of a group attention mechanism greatly reduces the auxiliary diagnosis model parameters, thereby reducing resource consumption; and it also integrates the unique diagnosis and treatment knowledge of ethnic medicine to simulate a multi-diagnosis diagnosis method.

[0051] 2. Before expressing the semantic features of the Transformer-style model, the present invention constructs a modal conversion algorithm based on a deep network to perform modal conversion, reduce the dimension and express the features of multimodal data, and also expand the application field of deep learning theory.

[0052] 3. Based on traditional text recognition and data structuring algorithms, the present invention reuses the vector representation encoder of medical data to generate vectorized feature expressions with semantic information; the recognized text is based on certain rules, such as prescription name, medicine name, symptoms, dosage method and other information, and the corresponding key fields and content are matched to generate a structured ethnic medicine knowledge base; the data is text content, and the vector feature encoder of medical data can be reused. After training, the key fields and content are vectorized, and then combined and stored in a vector database to form an ethnic medicine knowledge base.

[0053] 4. The present invention also proposes a deep network model based on adversarial generation and few-sample learning, which aims to use the encoder-decoder method to represent the semantic features of multimodal data, restore the original multimodal data or virtualize a large amount of high-quality labeled data, so that the generative adversarial network can automatically learn the distribution representation of different modal data from fewer labeled samples, and generate a mapping relationship between heterogeneous data and homogeneous data after quality unification. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of a clinical auxiliary diagnosis method with multi-diagnosis and combined reference in ethnic medicine;

[0055] Figure 2 is a flowchart of a method for constructing a vector database;

[0056] Figure 3 This is a flowchart of the method for constructing a quantitative ethnic medicine diagnosis and treatment knowledge base;

[0057] Figure 4 This is the principle diagram of the multi-diagnosis and clinical auxiliary diagnosis model with ethnic medicine characteristics;

[0058] Figure 5 This is the schematic diagram of the grouped multi-head attention mechanism;

[0059] Figure 6 Flowchart for heterogeneous data processing. DETAILED DESCRIPTION

[0060] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0061] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0062] Embodiment 1

[0063] See also Figure 1 , a clinical auxiliary diagnosis method with ethnic medicine characteristics, specifically comprising the following steps:

[0064] Step S1: Form a vector database from historical multi-diagnosis data and a vectorized ethnic medicine diagnosis and treatment knowledge base from ethnic medicine literature; it should be noted that the vector database and the vectorized ethnic medicine diagnosis and treatment knowledge base together constitute Figure 1 The vectorized ethnomedicine knowledge base shown in ;

[0065] Step S2: Based on the vector database and the vectorized ethnic medicine diagnosis and treatment knowledge base, a clinical auxiliary diagnosis model with ethnic medicine characteristics and multi-diagnosis is constructed;

[0066] Step S3: input the acquired text data or image data into the ethnic medicine characteristic multi-diagnosis and reference clinical auxiliary diagnosis model for diagnosis.

[0067] In this embodiment, please refer to Figure 2 , the method for constructing the vector database comprises:

[0068] Step A: converting the three modalities of text data, physiological signal data, and image data in the historical multi-diagnosis data into two modalities of text data and image data;

[0069] Step B: For the converted text data and image data, use the constructed Transformer-style model to extract the semantic information represented by the vector;

[0070] Step C: storing the semantic information in combination with metadata (which can be defined according to business needs, such as patient name, gender, age, etc.) into a vector database; it should be noted that the Transformer-style model has achieved good results in expressing semantic information of text and image modal data, but cannot process medical data of more modalities. Therefore, this embodiment introduces the modal conversion algorithm into the semantic feature expression of the Transformer-style model, thereby solving the difficulty of "not being able to express semantic information in the same representation space" encountered when storing multimodal medical data;

[0071] See also Figure 3 The method for constructing the vectorized ethnic medicine diagnosis and treatment knowledge base includes:

[0072] Step I: Use text recognition algorithms to identify text information in ethnic medicine literature;

[0073] Step II: Use data structuring algorithms to process text information and generate structured ethnic medical diagnosis and treatment knowledge;

[0074] Step III: For the generated structured ethnic medicine diagnosis and treatment knowledge, reuse the constructed Transformer-style model to extract the semantic information of the vector representation to form a vectorized ethnic medicine diagnosis and treatment knowledge base.

[0075] In this embodiment, specifically, step A includes:

[0076] Step A1: Collect text data, physiological signal data, and image data, and perform data cleaning to eliminate noise and outliers in the data; improve data quality so that subsequent deep network model training can converge quickly;

[0077] Step A2: Convert physiological signal data into image data through a data modality conversion model.

[0078] In this embodiment, specifically, the data cleaning includes:

[0079] For text data and physiological signal data, firstly, data with missing field values ​​(field values ​​are empty or NaN) and duplicate data are deleted, and then the minimum and maximum normalization is used to standardize the values ​​of each field to ensure the accuracy and reliability of the data;

[0080] For image data, first delete images with a lot of noise information (blurred, blocked, missing, etc.), and then scale the size of each image to a uniform size (such as 512*512).

[0081] In this embodiment, specifically, step A2 includes:

[0082] For physiological signal data, Fast Fourier Transform (FFT) is used to transform the time domain signal into the frequency domain signal to generate the corresponding spectrum image, and then the size of each spectrum image is scaled to a uniform size (such as 512*512).

[0083] In this embodiment, specifically, the step B includes:

[0084] Step B1: Construct a medical image-text bimodal Embedding model based on the CLIP (Contrastive Language-Image Pre-training) model, text data, and image data;

[0085] Step B2: In the reasoning stage, the constructed medical image-text bimodal Embedding model is used to vectorize the input text data and image data to extract the semantic information represented by their vectors.

[0086] In this embodiment, specifically, the construction process of the medical image-text bimodal Embedding model is as follows:

[0087] Encode part of the text content in the patient's medical record (e.g., white and greasy tongue coating) into a 512-dimensional One-Hot vector, i.e., a text vector;

[0088] At the same time, the inspection image corresponding to the text content (such as the greasy moss image) is cut into 512 image blocks, and the median pixel value of each image block is taken to form a total of 512 pixel values, and then the pixel values ​​are flattened into a 1*512-dimensional vector, namely the image vector;

[0089] A text-image pair is constructed based on the text vector and the image vector;

[0090] The text vector and image vector in each text-image pair are then concatenated and input into the CLIP model for training until the model converges to obtain a medical image-text bimodal Embedding model (i.e., the loss function no longer decreases significantly and the model tends to be stable. The convergence state varies depending on factors such as data and hyperparameters and is not limited).

[0091] Medical images for clinical applications are usually multi-source heterogeneous data. Even for the same modality data, the generated medical data are not of the same quality due to factors such as imaging mechanism, configuration differences, and environmental influences. For example, the spatial resolution and density resolution of medical image data vary greatly. Such heterogeneous data make it difficult to train auxiliary diagnosis models with high generalization capabilities, thus affecting the ultimate use of auxiliary diagnosis models.

[0092] It should be noted that the construction methods of the text embedding model and the image embedding model are the same as above.

[0093] It should be noted that the encoder in the Transformer architecture can realize feature extraction, the decoder is often used to generate text data, the discriminant model in the Convolutional Neural Networks (CNN) can determine the category of the image, and the large language model may combine with the local knowledge base to comprehensively analyze the text content from both breadth and depth.

[0094] Therefore, this embodiment proposes a multi-diagnosis and clinical auxiliary diagnosis model with ethnic medicine characteristics based on feature significance guidance, multi-domain knowledge union, and ethnic medicine characteristics. The model intends to mine the key relationships between multimodal medical data based on Transformer, and use the results of CNN feedforward layer, classification head, and regression head to guide the Transformer-style model to enhance significant feature information, so that the model can "read text" while "looking at pictures", and combine the ethnic medicine knowledge auxiliary model to give a comprehensive diagnosis result, so as to achieve consistency of data input in a single unified model architecture front end.

[0095] In this embodiment, specifically, Figure 4 As shown, the ethnic medicine characteristic multi-diagnosis and combined clinical auxiliary diagnosis model includes:

[0096] The text embedding model and image embedding model can respectively vectorize text data and image data in the same representation space;

[0097] The Transformer encoder, the classification head and regression head of the fine-grained model, and the Transformer decoder are used to extract qualitative and quantitative information of the examination site from the vectorized data, and then embedded in the Transformer decoder to decode the preliminary diagnosis results; based on the preliminary diagnosis results, relevant diagnosis and treatment information is retrieved from the vector database and the vectorized ethnic medicine diagnosis and treatment knowledge base through similarity metric learning;

[0098] The National Medical University Language Model (LLM) inputs the preliminary diagnosis results and related diagnosis and treatment information into the National Medical University Language Model to output a standardized diagnosis report; it should be noted that the LLM adopts large language models that are open source or commercial on the market, including but not limited to: Qwen2-72B-Instruct, Meta-Llama-3-70B-Instruct, Baichuan4, etc.

[0099] The complexity of multi-diagnosis auxiliary diagnosis models will inevitably lead to great consumption of computing power and more time for training and inference, which may lead to high computing costs and energy consumption. Moreover, complex models usually require more data for training, especially in the medical field, where obtaining large-scale, high-quality data can be challenging. Therefore, reducing the computational complexity of the model and retaining sufficient model performance is an inevitable difficulty in building large-scale auxiliary diagnosis models.

[0100] Therefore, this embodiment adopts the calculation method of grouped multi-head attention in the multi-diagnosis and clinical auxiliary diagnosis model with ethnic medicine characteristics to reduce the complexity of attention mechanism calculation.

[0101] In this embodiment, specifically, Figure 5 As shown, the grouped multi-head attention includes:

[0102] For each group X, three 1*3 matrices WQ, WK, and WV are randomly initialized, and X*WQ, X*WK, and X*WV are calculated respectively to obtain Q, K, and V; then according to the formula Calculate the self-attention, and the obtained vector is flattened to Z; finally, add the four sets of results Z1, Z2, Z3, and Z4 to get Z sum ; It can compress column vectors by rows and perform group multiplication and summation, which can greatly reduce the video memory capacity required for traditional multi-head attention calculations.

[0103] Medical images for clinical applications are usually multi-source heterogeneous data. Even for the same modality data, the medical data generated are not of the same quality due to factors such as imaging mechanism, configuration differences, and environmental influences. For example, the spatial resolution and density resolution of medical image data vary greatly. Such heterogeneous data make it difficult to train auxiliary diagnosis models with high generalization capabilities, thus affecting the ultimate use effect of the auxiliary diagnosis model.

[0104] Therefore, it is extremely important to reconstruct multi-source heterogeneous data and keep the data quality at the same standard. Traditional image interpolation algorithms and data missing value filling algorithms process data consistently under certain rules and cannot adaptively distinguish and treat data of different modalities. At the same time, there are few high-quality medical annotations, which makes it difficult to promote deep feature learning.

[0105] Therefore, this embodiment also proposes a deep network model based on adversarial generation and few-sample learning, which aims to use the encoder-decoder method to represent the semantic features of multimodal data, restore the original multimodal data or virtualize a large amount of high-quality annotated data, so that the generative adversarial network can automatically learn the distribution representation of different modal data from fewer annotated samples, and generate a mapping relationship between heterogeneous data and homogeneous data after quality unification, such as Figure 6As shown, the model can be applied to data of different modalities such as images and texts;

[0106] Specifically, for the heterogeneous data in historical multi-diagnosis data and ethnic medicine literature (data quality is not uniform, such as different image resolutions, missing words in medical record text descriptions causing ambiguity, etc.), they are input into the encoder for vectorization to generate 512-dimensional vector features; among them, the encoder is an Embedding model, and multimodal models such as CLIP (Contrastive Language-Image Pre-training) can be used without limitation;

[0107] The decoder is then used to reconstruct the image or text. The decoder uses the Transformer architecture (8 layers, 512 nodes) in the MAE (Masked Auto Encoder) model for image reconstruction, i.e., generating a large amount of high-quality data with normalized pixel values ​​or text reconstruction, i.e., filling in missing keywords and correcting text semantics. At the same time, it can generate a large amount of medical record descriptions with different expressions of the same semantics.

[0108] It should be noted that heterogeneous and small-sample data make it difficult to train auxiliary diagnosis models with high generalization capabilities. The use of adversarial sample generation technology can expand the number of samples, unify the sample quality distribution, and train models in a targeted manner. The introduction of vector-represented semantic feature information can give full play to the advantages of multi-diagnosis data feature fusion, and is expected to overcome the difficulty of unifying data quality using the same model for multimodal data.

[0109] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

[0110] This background section is provided to generally present the context of the invention, and the work of the presently named inventors, the work to the extent described in this background section, and aspects of this section that did not constitute prior art at the time of application are neither explicitly nor implicitly admitted to be prior art to the present invention.

Claims

1. A clinical auxiliary diagnosis method with multi-diagnosis and combined reference of ethnic medicine, characterized in that: include: Step S1: forming a vector database from historical multi-diagnosis data and forming a vectorized ethnic medicine diagnosis and treatment knowledge base from ethnic medicine literature; Step S2: Based on the vector database and the vectorized ethnic medicine diagnosis and treatment knowledge base, a clinical auxiliary diagnosis model with ethnic medicine characteristics and multi-diagnosis is constructed; Step S3: input the acquired text data or image data into the ethnic medicine characteristic multi-diagnosis and reference clinical auxiliary diagnosis model for diagnosis.

2. The clinical auxiliary diagnosis method with multi-diagnosis and combined reference of ethnic medicine characteristics according to claim 1 is characterized in that: The method for constructing the vector database comprises: Step A: converting the three modalities of text data, physiological signal data, and image data in the historical multi-diagnosis data into two modalities of text data and image data; Step B: For the converted text data and image data, use the constructed Transformer-style model to extract the semantic information represented by the vector; Step C: storing the semantic information combined with metadata into the vector database; The method for constructing the vectorized ethnic medicine diagnosis and treatment knowledge base includes: Step I: Use text recognition algorithms to identify text information in ethnic medicine literature; Step II: Use data structuring algorithms to process text information and generate structured ethnic medical diagnosis and treatment knowledge; Step III: For the generated structured ethnic medicine diagnosis and treatment knowledge, reuse the constructed Transformer-style model to extract the semantic information of the vector representation to form a vectorized ethnic medicine diagnosis and treatment knowledge base.

3. The clinical auxiliary diagnosis method with multi-diagnosis and combined reference of ethnic medicine characteristics according to claim 2 is characterized in that: The step A comprises: Step A1: collect text data, physiological signal data and image data, and perform data cleaning to eliminate noise and outliers in the data; Step A2: converting physiological signal data into image data through a data modality conversion model; The data cleaning includes: For text data and physiological signal data, first delete the data with missing field values ​​and duplicate data, and then use minimum and maximum normalization to standardize the values ​​of each field to ensure the accuracy and reliability of the data; For image data, first delete the images with a lot of noise information, and then scale the size of each image to a uniform size; The step A2 comprises: For physiological signal data, fast Fourier transform is used to transform the time domain signal into the frequency domain signal to generate the corresponding spectrum image, and then the size of each spectrum image is scaled to a uniform size.

4. The clinical auxiliary diagnosis method with multi-diagnosis and combined reference of ethnic medicine characteristics according to claim 3 is characterized in that: The step B comprises: Step B1: Construct a medical image-text bimodal Embedding model based on the CLIP model, text data, and image data; Step B2: In the reasoning stage, the constructed medical image-text bimodal Embedding model is used to vectorize the input text data and image data to extract the semantic information represented by their vectors; The construction process of the medical image-text bimodal Embedding model is as follows: Encode part of the text content on the patient's medical record into a 512-dimensional One-Hot vector, i.e., a text vector; At the same time, the inspection image corresponding to the text content is cut into 512 image blocks, and the median pixel value of each image block is taken to form a total of 512 pixel values, and then the pixel values ​​are flattened into a 1*512-dimensional vector, namely the image vector; A text-image pair is constructed based on the text vector and the image vector; The text vector and image vector in each text-image pair are then concatenated and input into the CLIP model for training until the model converges to obtain a medical image-text bimodal Embedding model.

5. The clinical auxiliary diagnosis method with multi-diagnosis and combined reference of ethnic medicine characteristics according to claim 2 is characterized in that: The metadata includes: patient name, gender, and age.

6. The clinical auxiliary diagnosis method with multi-diagnosis and combined reference of ethnic medicine characteristics according to claim 2 is characterized in that: The multi-diagnosis and clinical auxiliary diagnosis model with ethnic medicine characteristics includes: The text embedding model and image embedding model can respectively vectorize text data and image data in the same representation space; The Transformer encoder, the classification head and regression head of the fine-grained model, and the Transformer decoder are used to extract qualitative and quantitative information of the examination site from the vectorized data, and then embedded in the Transformer decoder to decode the preliminary diagnosis results; based on the preliminary diagnosis results, relevant diagnosis and treatment information is retrieved from the vector database and the vectorized ethnic medicine diagnosis and treatment knowledge base through similarity metric learning; The language model of the National Medical University inputs the preliminary diagnosis results and related diagnosis and treatment information into the language model of the National Medical University and outputs a standardized diagnosis report.

7. The clinical auxiliary diagnosis method with multi-diagnosis and combined reference of ethnic medicine characteristics according to claim 6 is characterized in that: The ethnic medicine-specific multi-diagnosis and clinical auxiliary diagnosis model adopts a grouped multi-head attention calculation method to reduce the complexity of attention mechanism calculation.

8. The clinical auxiliary diagnosis method with multi-diagnosis and combined reference of ethnic medicine characteristics according to claim 7 is characterized in that: The grouped multi-head attention includes: For each group X, three 1*3 matrices WQ, WK, and WV are randomly initialized, and X*WQ, X*WK, and X*WV are calculated respectively to obtain Q, K, and V; then according to the formula Calculate the self-attention, and the obtained vector is flattened to Z; finally, add the four sets of results Z1, Z2, Z3, and Z4 to get Z sum .

9. The clinical auxiliary diagnosis method with multi-diagnosis and combined reference of ethnic medicine characteristics according to claim 1, characterized in that: For the heterogeneous data in historical multi-diagnosis data, ethnic medicine literature, text data and image data, they are input into the encoder for vectorization to generate multi-dimensional vector features; A decoder is then used to reconstruct the image or text; the encoder is an Embedding model.

10. The clinical auxiliary diagnosis method with multi-diagnosis and combined reference of ethnic medicine characteristics according to claim 9, characterized in that: The decoder adopts the Transformer architecture in the MAE model for image reconstruction, that is, generating a large amount of high-quality data with normalized pixel values ​​or text reconstruction, that is, filling in missing keywords and correcting text semantics. At the same time, it can generate a large number of medical record description contents with different expressions of the same semantics.

Images