A training method and device for a multimodal large model to assist traditional Chinese medicine diagnosis

Abstract

The present invention discloses a training method and device for a multimodal large model for assisting TCM diagnosis, and belongs to the field of intelligent TCM medical technology. The method comprises: obtaining continuous blood flow data and vascular image data of an artery examined by ultrasound; performing time series processing on the blood flow data to obtain time series data; performing Doppler effect staining processing on the vascular image data to obtain Doppler effect staining video data; using the time series data and the Doppler effect staining video data to perform parameter fine-tuning on the multimodal large model, so that the multimodal large model outputs data related to the TCM diagnosis result. The scheme provided by the present invention is adopted, and data is collected from multiple angles, and image data and video data are comprehensively processed for fine-tuning the multimodal large model, and the obtained multimodal large model is used to assist TCM diagnosis. The shortcomings of the prior art in TCM application, data processing depth and model training limitations are effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent traditional Chinese medical technology, and in particular to a training method and device for a multimodal large model for assisting traditional Chinese medical diagnosis. Background Art

[0002] In today's medical field, especially in the practice of TCM diagnosis, ultrasound technology has been widely used in a variety of diagnostic and treatment monitoring due to its non-invasive, strong real-time and relatively low cost characteristics. Especially in the diagnostic process of TCM, such as the study of pulse, the traditional reliance on the experience of clinicians for hand pulse diagnosis has certain subjectivity and uncertainty. In recent years, with the development of medical technology, especially the advancement of medical imaging and artificial intelligence technology, ultrasound images have begun to be explored for simulating and enhancing the traditional diagnostic methods of TCM.

[0003] In the prior art, ultrasound is used to obtain real-time images of the internal structure of the human body, which is crucial for the diagnosis of organ diseases in the field of Western medicine. For example, in cardiology, echocardiography is an important means of evaluating the structure and function of the heart. Similarly, this technology has been gradually introduced into the diagnostic practice of traditional Chinese medicine, attempting to observe physiological parameters such as blood vessels and blood flow through ultrasound images, and providing more objective data support for traditional Chinese medicine diagnosis. Although ultrasound technology has broad application prospects in clinical applications, its application in the field of traditional Chinese medicine still faces many challenges. First, the capture and interpretation of information in traditional Chinese medicine diagnostic methods such as pulse diagnosis is highly dependent on the experience and skills of clinicians. This high degree of individual dependence makes it complicated to use ultrasound technology to standardize the traditional Chinese medicine diagnostic process. Secondly, most of the existing ultrasound equipment and technologies are designed to analyze anatomical structures rather than meet the specific needs of traditional Chinese medicine pulse diagnosis, such as pulse fluctuation analysis.

[0004] In response to the above challenges, technicians have integrated ultrasound imaging technology with advanced large-model data analysis methods. This includes using artificial intelligence algorithms to analyze ultrasound images and automatically identify and interpret key medical information in images. For example, the vascular pulse fluctuation data collected by ultrasound images, combined with machine learning technology for in-depth analysis, can simulate and optimize the traditional pulse diagnosis process and improve the accuracy and repeatability of diagnosis. For example, the patent application with publication number CN117114072A records a method for using ultrasound images for simulation system training, which mainly includes: obtaining ultrasound image data and training through an AI large model to simulate and predict the position of the heart in a virtual system. This technology improves clinicians' understanding and application efficiency of ultrasound images by combining ultrasound images with virtual heart models, and reduces the learning curve. However, these technologies generally have the following technical problems: First, they have application limitations. The existing technology mainly focuses on the simulation training of cardiac ultrasound images, and has few applications in other fields such as traditional Chinese medicine diagnostic methods such as pulse diagnosis. This limits the possibility of wider application of ultrasound technology in the field of traditional Chinese medicine. Second, there is a lack of data processing. Although ultrasound images are analyzed through AI model training, there is no clear solution for how to process and analyze more detailed data such as vascular pulsation frequency, amplitude, and blood flow rate required for TCM pulse diagnosis. These technical details are crucial factors in TCM diagnosis. Third, there are limitations in model training. The Can model is mainly based on static image feature training, and may not be able to fully capture dynamically changing physiological signals, such as pulse fluctuations, which poses certain challenges to accurately simulate and predict TCM pulse. Summary of the invention

[0005] In order to solve the problems existing in the prior art, the present invention provides the following technical solutions.

[0006] The first aspect of the present invention provides a training method for a multimodal large model for assisting TCM diagnosis, comprising:

[0007] Acquire continuous blood flow data and vascular image data of ultrasound-examined arteries;

[0008] Performing time series processing on the blood flow data to obtain time series data; performing Doppler effect staining processing on the blood vessel image data to obtain Doppler effect staining video data;

[0009] The parameters of the multimodal large model are fine-tuned using time-serialized data and Doppler effect stained video data, so that the multimodal large model outputs data related to the TCM diagnosis results for assisting TCM diagnosis; the fine-tuning includes: first, converting the time-serialized data into a time-serialized multidimensional vector corresponding to the timestamp, and each time-serialized multidimensional vector is processed by a linear layer and passed as an input to the natural language processing module of the multimodal large model; at the same time, for the Doppler effect stained video data, the key frames under each timestamp are extracted, and the key frames are processed by the visual processing module of the multimodal large model and converted into key frame multidimensional vectors; each key frame multidimensional vector is processed by a full parameter layer and passed as an input to the natural language processing module of the multimodal large model; the time-serialized multidimensional vector under each timestamp is spliced ​​with the key frame to form a corresponding comprehensive input; all the comprehensive inputs are spliced ​​together according to the continuity of the timestamps and passed as a whole input to the natural language processing module of the multimodal large model; and fine-tuning is performed in the natural language processing module.

[0010] Preferably, the acquisition of blood flow data and vascular image data of an ultrasonically examined artery includes: using an array ultrasound probe, setting an angle of 60° between the ultrasound probe and the skin, using the Doppler effect to locate the artery to be examined, and collecting blood flow data and vascular image data of the artery.

[0011] Preferably, the blood flow data includes: the frequency and amplitude of blood vessel pulsation and the speed and direction of blood flow; and the blood vessel image data is a Doppler effect image of the blood vessel.

[0012] Preferably, the time series processing of the blood flow data includes: sampling the blood flow data once every preset time period.

[0013] Preferably, performing Doppler effect staining processing on the blood vessel image data includes: color coding the blood vessel image data to obtain Doppler effect staining video data.

[0014] The second aspect of the present invention provides a training device for a multi-modal large model for assisting TCM diagnosis, comprising:

[0015] A data acquisition module, used to acquire continuous blood flow data and blood vessel image data of an ultrasound-examined artery;

[0016] The data processing module is used to perform time series processing on the blood flow data to obtain time series data; perform Doppler effect staining processing on the blood vessel image data to obtain Doppler effect staining video data;

[0017] A large model fine-tuning module is used to use time-serialized data and Doppler effect stained video data to fine-tune the parameters of the multimodal large model, so that the multimodal large model outputs data related to the TCM diagnosis results for assisting TCM diagnosis; the fine-tuning includes: first, converting the time-serialized data into a time-serialized multidimensional vector corresponding to the timestamp, and each time-serialized multidimensional vector is processed by a linear layer and passed as an input to the natural language processing module of the multimodal large model; at the same time, for the Doppler effect stained video data, extracting the key frames under each timestamp, and converting the key frames into key frame multidimensional vectors after processing by the visual processing module of the multimodal large model; each key frame multidimensional vector is processed by a full parameter layer and passed as an input to the natural language processing module of the multimodal large model; splicing the time-serialized multidimensional vector under each timestamp with the key frame to form a corresponding comprehensive input; splicing all the comprehensive inputs together according to the continuity of the timestamp, and passing them as a whole input to the natural language processing module of the multimodal large model; fine-tuning is performed in the natural language processing module.

[0018] The third aspect of the present invention provides a method for providing auxiliary data for TCM diagnosis using a multimodal large model, comprising:

[0019] Acquiring continuous blood flow data and vascular image data of an ultrasound-examined artery of a patient to be diagnosed by traditional Chinese medicine;

[0020] Performing time series processing on the blood flow data to obtain time series data; performing Doppler effect staining processing on the blood vessel image data to obtain Doppler effect staining video data;

[0021] Input the time series data and Doppler effect dyed video data into the trained multimodal large model, and output data related to the TCM diagnosis results to assist TCM diagnosis;

[0022] Among them, the trained multimodal large model is pre-trained using the training method of the multimodal large model for assisting TCM diagnosis described in the first aspect.

[0023] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the training method of a multimodal large model for assisting TCM diagnosis as described in the first aspect, and / or the method of providing auxiliary data for TCM diagnosis using a multimodal large model as described in the third aspect.

[0024] The present invention also provides an electronic device, comprising a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, which can be loaded and executed by the processor, so that the processor can execute the training method of a multimodal large model for assisting TCM diagnosis as described in the first aspect, and / or the method of providing auxiliary data for TCM diagnosis using a multimodal large model as described in the third aspect.

[0025] The beneficial effects of the present invention are as follows: the training method and device of the multimodal large model for assisting TCM diagnosis provided by the present invention. The method first obtains continuous blood flow data and vascular image data of the ultrasound examination artery; then performs time series processing on the blood flow data to obtain time series data; performs Doppler effect staining processing on the vascular image data to obtain Doppler effect staining video data; finally, the parameters of the multimodal large model are fine-tuned using the time series data and the Doppler effect staining video data, so that the multimodal large model outputs data related to the TCM diagnosis result. Through fine-tuning, it is not only ensured that the time series data can be integrated into the multimodal large model in an appropriate format, thereby retaining its time dependency and multidimensional information. Moreover, the dynamic information and Doppler effect features in the video can be effectively embedded in the multimodal large model. At the same time, it can also ensure that the multimodal large model can capture the dynamic changes in the time series and the correlation between the multimodal data, thereby improving the accuracy and reliability of the diagnosis result. By adopting the above method, the present invention effectively solves the shortcomings of the prior art in terms of TCM application, data processing depth and model training limitations, and provides a more scientific and accurate technical solution for TCM diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of the flow chart of the training method of the multimodal large model for assisting TCM diagnosis according to the present invention;

[0027] Figure 2 This is a schematic diagram of the functional structure of the training device for the multimodal large model for assisting TCM diagnosis according to the present invention. DETAILED DESCRIPTION

[0028] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0029] The method provided by the present invention can be implemented in the following terminal environment, and the terminal may include one or more of the following components: a processor, a memory, and a display screen. The memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method described in the following embodiment.

[0030] The processor may include one or more processing cores. The processor uses various interfaces and lines to connect various parts of the entire terminal, and executes various functions of the terminal and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory, and calling data stored in the memory.

[0031] The memory may include random access memory (RAM) or read-only memory (ROM). The memory may be used to store instructions, programs, codes, code sets or instructions.

[0032] The display screen is used to display the user interface of each application.

[0033] In addition, those skilled in the art can understand that the structure of the above terminal does not constitute a limitation on the terminal, and the terminal may include more or fewer components, or combine certain components, or arrange the components differently. For example, the terminal also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, and a power supply, which will not be described in detail here.

[0034] Embodiment 1

[0035] like Figure 1 As shown, an embodiment of the present invention provides a training method for a multimodal large model to assist TCM diagnosis, including: S101, obtaining continuous blood flow data and vascular image data of an ultrasonically examined artery; S102, performing time serialization processing on the blood flow data to obtain time serialization data; performing Doppler effect staining processing on the vascular image data to obtain Doppler effect staining video data; S103, using the time serialization data and the Doppler effect staining video data to fine-tune the parameters of the multimodal large model, so that the multimodal large model outputs data related to the TCM diagnosis results.

[0036] The fine-tuning includes: first, converting the time-serialized data into a time-serialized multidimensional vector corresponding to the timestamp, and each time-serialized multidimensional vector is processed by a linear layer and passed as an input to the natural language processing module of the multimodal large model; at the same time, for the Doppler effect stained video data, extracting the key frames at each timestamp, and converting the key frames into key frame multidimensional vectors after processing by the visual processing module of the multimodal large model; each key frame multidimensional vector is processed by a full parameter layer and passed as an input to the natural language processing module of the multimodal large model; splicing the time-serialized multidimensional vector at each timestamp with the key frame to form a corresponding comprehensive input; splicing all the comprehensive inputs together according to the timestamp continuity, and passing them as a whole input to the natural language processing module of the multimodal large model; and performing fine-tuning in the natural language processing module.

[0037] In the present invention, the above method is used for fine-tuning, which can not only ensure that the time series data can be integrated into the multimodal large model in an appropriate format, thereby retaining its time dependency and multidimensional information. At the same time, the dynamic information and Doppler effect characteristics in the Doppler effect stained video can be effectively embedded in the multimodal large model. In addition, by passing the data spliced ​​with the multidimensional vector and the key frame as the overall input to the multimodal large model, it is ensured that the multimodal large model can capture the dynamic changes in the time series and the correlation between the multimodal data, thereby improving the accuracy and reliability of the auxiliary Chinese medicine diagnosis results.

[0038] In a preferred embodiment of the present invention, the acquisition of blood flow data and vascular image data of an ultrasonically examined artery may include: using an array ultrasound probe, setting an angle of 60° between the ultrasound probe and the skin, using the Doppler effect to locate the artery to be examined, and collecting blood flow data and vascular image data of the artery.

[0039] The blood flow data includes: the frequency and amplitude of blood vessel pulsation and the blood flow speed and direction; and the blood vessel image data is a Doppler effect image of the blood vessel.

[0040] In another preferred embodiment of the present invention, the time series processing of the blood flow data includes: sampling the blood flow data once every preset time period. The Doppler effect staining processing of the blood vessel image data includes: color coding the blood vessel image data to obtain Doppler effect staining video data.

[0041] In the specific application process, the following steps can be used for implementation:

[0042] Step 1: Use ultrasound to locate the pulse.

[0043] Select an array ultrasound probe with high resolution and depth adjustment function, which can be adjusted to a 60° angle with the skin to maximize the detection efficiency of arterial blood flow. Probe position adjustment: Ensure that the probe can accurately collect signals through a mechanical or electric adjustment system.

[0044] Step 2: Data collection.

[0045] The collected raw ultrasound data is converted in real time into quantifiable data such as the frequency and amplitude of vascular pulsation. This part of the data mainly focuses on quantitative parameters extracted from the blood flow, such as the frequency and amplitude of vascular pulsation and blood flow rate. These data are usually numerical and can be organized into time series data, that is, arranged in chronological order and collected every 10 milliseconds to form continuous data points. This data format is convenient for mathematical and statistical analysis, such as calculating the characteristics of time series, modeling, or monitoring changes in blood flow status.

[0046] Using the Doppler effect, the probe emits ultrasonic waves and receives reflected waves, and accurately calculates the blood flow velocity and blood vessel position based on the change in wave frequency. This part of the data shows the changes in blood flow velocity and direction through color coding, forming a visually intuitive representation. This video information is easier for the human eye to observe and understand the dynamics of blood flow. Video information is usually used to intuitively display the dynamic changes of data, allowing doctors to observe blood flow in real time, providing auxiliary data display for doctors to make diagnoses and evaluations, and helping to diagnose and evaluate diseases.

[0047] Step 3: Data processing.

[0048] Time series data generation: The continuously collected blood flow data (frequency, amplitude and speed) are processed into time series, sampled every 10 milliseconds, and form a data sequence of 100 frames per second.

[0049] Data synchronization and integration: Doppler image data and serialized linear data are processed synchronously to provide a complete data set for subsequent analysis.

[0050] Step 4: Multimodal large model training and fine-tuning.

[0051] Model input configuration: Take the Doppler effect stained image and the linear information of vascular pulsation as input, and configure appropriate model prompts to adjust the model's response to specific data.

[0052] Fine-tuning of the multimodal large model: Use the collected comprehensive data to fine-tune all parameters of the multimodal large model (combining vision and natural language processing modules) to enhance the model's predictive ability for TCM pulse diagnosis characteristics.

[0053] Step 5: Implementation and feedback.

[0054] Application implementation: Deploy the multimodal large model trained by the present invention in an actual TCM diagnosis environment, and verify the effectiveness and accuracy of the algorithm through actual cases.

[0055] Performance evaluation and optimization: Based on feedback from doctors and patients, the performance of trained multimodal large models is evaluated, key operational data is collected, and the algorithm is continuously optimized.

[0056] Through the above steps and technical arrangements, the solution of the present invention not only optimizes the traditional Chinese medicine pulse diagnosis method, but also significantly improves the scientificity and accuracy of diagnosis by using modern technology, providing strong technical support for the modernization and internationalization of Chinese medicine.

[0057] Compared with the prior art, the technical solution provided by the present invention has made technical improvements in the following aspects:

[0058] Multi-angle data acquisition: The present invention adopts an array ultrasound probe to detect special signals of the human body at a specific angle and uses the Doppler effect to accurately locate blood vessels, thereby overcoming the shortcomings of traditional technology in capturing dynamic physiological signals.

[0059] Comprehensive data analysis: Through waveform transformation and time series sampling, the present invention not only obtains static image data, but also continuously records dynamic parameters such as vascular pulsation frequency, amplitude and blood flow rate, providing more comprehensive data support for traditional Chinese medicine pulse diagnosis.

[0060] Fine-tuning of the multimodal large model: Combining the Doppler effect stained image with serialized linear information, the present invention fine-tunes all parameters of the multimodal large model, enabling it to conduct in-depth analysis and prediction based on comprehensive visual and textual information, thereby effectively improving the accuracy and scientificity of the auxiliary process of TCM pulse diagnosis.

[0061] Through these technical improvements, the present invention effectively solves the shortcomings of the existing technology in terms of TCM application, data processing depth and model training limitations, and provides a more scientific and accurate technical support for auxiliary TCM diagnosis.

[0062] The present invention significantly improves the scientificity and accuracy of auxiliary Chinese medicine pulse diagnosis by combining advanced ultrasonic technology with multimodal large model processing. The specific beneficial effects are as follows:

[0063] Improved diagnostic accuracy: The use of array ultrasound probes and the Doppler effect can accurately locate signal acquisition and accurately measure vascular pulsation frequency, amplitude, and blood flow rate. This precise measurement of physiological parameters provides more objective and reliable data support than traditional methods, thus providing assistance in improving the accuracy of traditional Chinese medicine pulse diagnosis.

[0064] Optimized data processing and analysis: By time-series the collected data and analyzing it using natural language processing technology, this solution can comprehensively consider the multi-dimensional data of vascular pulsation and provide continuous and detailed blood flow dynamic analysis. This method not only improves the efficiency of data processing, but also enhances the depth and breadth of data analysis.

[0065] Enhanced predictive power of diagnostic models: By fine-tuning all parameters of the multimodal large model and integrating Doppler images with linear data, the model can more comprehensively understand and predict pulse characteristics. This approach takes advantage of the complementarity of images and data, enhancing the predictive power and adaptability of the model in practical applications.

[0066] Improved operational convenience and lowered diagnostic threshold: The present invention reduces the need for traditional manual operations through automated data collection and analysis processes, and makes traditional Chinese medicine pulse diagnosis more convenient and user-friendly by assisting Chinese medicine pulse diagnosis. In addition, through real-time image and data feedback, doctors can instantly obtain auxiliary diagnosis results, improving the overall diagnosis and treatment efficiency.

[0067] In summary, the present application is not only innovative in technology, but also provides significant improvements in practical applications. By assisting TCM pulse diagnosis, the accuracy of TCM pulse diagnosis is improved, which has important significance and value for the overall development of TCM diagnostic technology.

[0068] Embodiment 2

[0069] like Figure 2 As shown, another aspect of the present invention also includes a functional module architecture that is completely consistent with the aforementioned method flow, that is, the embodiment of the present invention also provides a training device for a multimodal large model assisting TCM diagnosis, including: a data acquisition module 201, used to acquire continuous blood flow data and vascular image data of an ultrasonically examined artery; a data processing module 202, used to perform time series processing on the blood flow data to obtain time series data; perform Doppler effect staining processing on the vascular image data to obtain Doppler effect staining video data; a large model fine-tuning module 203, used to use the time series data and the Doppler effect staining video data to perform parameter fine-tuning on the multimodal large model, so that the multimodal large model outputs data related to the TCM diagnosis result; the fine-tuning includes: first, converting the time series data into a time series data; The data is converted into a time-serialized multidimensional vector corresponding to the timestamp, and each time-serialized multidimensional vector is processed by the linear layer and passed as input to the natural language processing module of the multimodal large model; at the same time, for the Doppler effect stained video data, the key frames at each timestamp are extracted, and the key frames are processed by the visual processing module of the multimodal large model and converted into key frame multidimensional vectors; each key frame multidimensional vector is processed by the full parameter layer and passed as input to the natural language processing module of the multimodal large model; the time-serialized multidimensional vector at each timestamp is spliced ​​with the key frame to form the corresponding comprehensive input; all the comprehensive inputs are spliced ​​together according to the timestamp continuity and passed to the natural language processing module of the multimodal large model as the overall input; and fine-tuning is performed in the natural language processing module.

[0070] Furthermore, in the data acquisition module, the acquisition of blood flow data and vascular image data of the ultrasonically examined artery includes: using an array ultrasound probe, setting an angle of 60° between the ultrasound probe and the skin, using the Doppler effect to locate the artery to be examined, and collecting the blood flow data and vascular image data of the artery.

[0071] The blood flow data includes: the frequency and amplitude of blood vessel pulsation and the blood flow speed and direction; and the blood vessel image data is a Doppler effect image of the blood vessel.

[0072] Furthermore, in the data processing module, the time series processing of the blood flow data includes: sampling the blood flow data once every preset time period. The Doppler effect staining processing of the blood vessel image data includes: color coding the blood vessel image data to obtain Doppler effect staining video data.

[0073] The device can be implemented through the training method of the multimodal large model for assisting TCM diagnosis provided in the above-mentioned embodiment 1. The specific implementation method can be found in the description of embodiment 1 and will not be repeated here.

[0074] Embodiment 3

[0075] An embodiment of the present invention provides a method for providing auxiliary data for TCM diagnosis using a multimodal large model, comprising: obtaining continuous blood flow data and vascular image data of an ultrasonically examined artery of a patient to be diagnosed by TCM; performing time serialization processing on the blood flow data to obtain time serialization data; performing Doppler effect staining processing on the vascular image data to obtain Doppler effect staining video data; inputting the time serialization data and the Doppler effect staining video data into a trained multimodal large model, and outputting data related to the TCM diagnosis result for assisting TCM diagnosis; wherein the trained multimodal large model is pre-trained using the training method of the multimodal large model for assisting TCM diagnosis described in Example 1.

[0076] The present invention also provides a memory storing a plurality of instructions, wherein the instructions are used to implement the training method of a multimodal large model for assisting TCM diagnosis as described in Example 1, and / or the method of using a multimodal large model to provide auxiliary data for TCM diagnosis as described in Example 3.

[0077] The present invention also provides an electronic device, comprising a processor and a memory connected to the processor, wherein the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor, so that the processor can execute the training method of a multimodal large model for assisting TCM diagnosis as described in Example 1, and / or the method of providing auxiliary data for TCM diagnosis using a multimodal large model as described in Example 3.

[0078] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A training method for a multimodal large model to assist TCM diagnosis, characterized in that: include: Acquire continuous blood flow data and vascular image data of ultrasound-examined arteries; Performing time series processing on the blood flow data to obtain time series data; Performing Doppler effect staining processing on the blood vessel image data to obtain Doppler effect staining video data; The parameters of the multimodal large model are fine-tuned using time-serialized data and Doppler effect stained video data, so that the multimodal large model outputs data related to the TCM diagnosis results for assisting TCM diagnosis; the fine-tuning includes: first, converting the time-serialized data into a time-serialized multidimensional vector corresponding to the timestamp, and each time-serialized multidimensional vector is processed by a linear layer and passed as an input to the natural language processing module of the multimodal large model; at the same time, for the Doppler effect stained video data, the key frames under each timestamp are extracted, and the key frames are processed by the visual processing module of the multimodal large model and converted into key frame multidimensional vectors; each key frame multidimensional vector is processed by a full parameter layer and passed as an input to the natural language processing module of the multimodal large model; the time-serialized multidimensional vector under each timestamp is spliced ​​with the key frame to form a corresponding comprehensive input; all the comprehensive inputs are spliced ​​together according to the continuity of the timestamps and passed as a whole input to the natural language processing module of the multimodal large model; and fine-tuning is performed in the natural language processing module.

2. The training method of a multimodal large model for assisting TCM diagnosis as claimed in claim 1, characterized in that: The method of obtaining the blood flow data and vascular image data of the artery under ultrasonic examination includes: using an array ultrasonic probe, setting an angle of 60° between the ultrasonic probe and the skin, locating the artery to be examined by using the Doppler effect, and collecting the blood flow data and vascular image data of the artery.

3. The training method of a multimodal large model for assisting TCM diagnosis as claimed in claim 1, characterized in that: The blood flow data includes: the frequency and amplitude of blood vessel pulsation and the blood flow speed and direction; the blood vessel image data is a Doppler effect image of the blood vessel.

4. The training method of a multimodal large model for assisting TCM diagnosis as claimed in claim 1, characterized in that: The time series processing of the blood flow data includes: sampling the blood flow data once every preset time period.

5. The training method of a multimodal large model for assisting TCM diagnosis as claimed in claim 1, characterized in that: The performing Doppler effect staining processing on the blood vessel image data includes: color coding the blood vessel image data to obtain Doppler effect staining video data.

6. A training device for a multimodal large model to assist TCM diagnosis, characterized in that: include: A data acquisition module, used to acquire continuous blood flow data and blood vessel image data of an ultrasound-examined artery; The data processing module is used to perform time series processing on the blood flow data to obtain time series data; perform Doppler effect staining processing on the blood vessel image data to obtain Doppler effect staining video data; A large model fine-tuning module is used to use time-serialized data and Doppler effect stained video data to fine-tune the parameters of the multimodal large model, so that the multimodal large model outputs data related to the TCM diagnosis results for assisting TCM diagnosis; the fine-tuning includes: first, converting the time-serialized data into a time-serialized multidimensional vector corresponding to the timestamp, and each time-serialized multidimensional vector is processed by a linear layer and passed as an input to the natural language processing module of the multimodal large model; at the same time, for the Doppler effect stained video data, extracting the key frames under each timestamp, and converting the key frames into key frame multidimensional vectors after processing by the visual processing module of the multimodal large model; each key frame multidimensional vector is processed by a full parameter layer and passed as an input to the natural language processing module of the multimodal large model; splicing the time-serialized multidimensional vector under each timestamp with the key frame to form a corresponding comprehensive input; splicing all the comprehensive inputs together according to the continuity of the timestamp, and passing them as a whole input to the natural language processing module of the multimodal large model; fine-tuning is performed in the natural language processing module.

7. A method for providing auxiliary data for TCM diagnosis using a multimodal large model, characterized in that: include: Acquiring continuous blood flow data and vascular image data of an ultrasound-examined artery of a patient to be diagnosed by traditional Chinese medicine; Performing time series processing on the blood flow data to obtain time series data; performing Doppler effect staining processing on the blood vessel image data to obtain Doppler effect staining video data; Input the time series data and Doppler effect dyed video data into the trained multimodal large model, and output data related to the TCM diagnosis results to assist TCM diagnosis; Among them, the trained multimodal large model is pre-trained using the training method of the multimodal large model for assisting TCM diagnosis described in any one of claims 1-5.

8. A memory, characterized in that: Multiple instructions are stored, and the instructions are used to implement the training method of a multimodal large model for assisting TCM diagnosis as described in any one of claims 1-5, or the method of using a multimodal large model to provide auxiliary data for TCM diagnosis as described in claim 7.

9. An electronic device, characterized in that: It includes a processor and a memory connected to the processor, the memory stores a plurality of instructions, and the instructions can be loaded and executed by the processor, so that the processor can execute the training method of a multimodal large model for assisting TCM diagnosis as described in any one of claims 1 to 5, or the method of providing auxiliary data for TCM diagnosis using a multimodal large model as described in claim 7.

Images