Auxiliary diagnosis and treatment system and method based on multi-modal fusion large model

Abstract

The application discloses an auxiliary diagnosis and treatment system and method based on a multi-modal fusion large model, relates to the technical field of medical artificial intelligence, adopts a dynamic time warping algorithm to align a voice feature sequence and a physiological feature sequence, then maps non-text modal semantic features to a unified text semantic space and combines a text semantic vector to construct a normalized three-dimensional feature matrix, finally extracts keywords to match a knowledge graph through vector retrieval and keyword retrieval, so as to solve the problems of subjectification of traditional diagnosis and treatment and fragmentation of multi-modal data, realize the objectification and precision of auxiliary diagnosis and treatment of mental diseases, and generate an authoritative auxiliary diagnosis and treatment scheme.

Description

Technical Field

[0001] This invention relates to the field of medical artificial intelligence technology, and in particular to an auxiliary diagnosis and treatment system and method based on a multimodal fusion large model. Background Technology

[0002] Mental illness has become a major global public health issue. Currently, mental health diagnosis and treatment mainly rely on three types of technical approaches: traditional diagnosis and treatment techniques are based on doctors' clinical experience, combined with standardized scales such as PHQ-9 and HAM-D to assess symptoms, and rely on patient self-reporting and human interpretation; existing digital therapies are mostly limited to the digitization of scales, converting paper scales into mobile tools, and some introduce simple NLP techniques to analyze the emotions in the text.

[0003] Therefore, existing technologies suffer from numerous key shortcomings, making it difficult to support the needs for objective diagnosis, early identification, and precise intervention of mental illnesses. Traditional diagnosis and treatment rely solely on text-based scales and subjective experience, failing to integrate multimodal information such as voice, emotion, physiological signals, and behavioral data. This results in diagnostic results being heavily influenced by doctors' subjective interpretations and patients' biases in their statements. While existing digital therapies have digitized scales, they have not resolved the issues of spatiotemporal alignment and semantic consistency of multimodal data, making it impossible to capture early, hidden symptoms, resulting in low early identification rates. Furthermore, the data exhibits broken correlations, hindering comprehensive analysis. General-purpose large-scale models, lacking support from a mental health specialty knowledge graph, are prone to producing hallucinations, such as incorrect medication recommendations, and are unable to adapt to multimodal data input, leading to insufficient safety and accuracy in treatment plans. These shortcomings collectively result in the long-term limitations of mental illness diagnosis and treatment due to subjectivity, fragmented data, and delayed intervention, making it difficult to adapt to primary healthcare scenarios and data privacy protection requirements. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies in the diagnosis and treatment of mental illness, which are long limited by subjectivity, fragmented data, and delayed intervention, and are also difficult to adapt to the requirements of primary healthcare scenarios and data privacy protection. This invention provides an auxiliary diagnosis and treatment system and method based on a multimodal fusion model. It employs a dynamic time warping algorithm to align speech feature sequences and physiological feature sequences, then maps non-textual modal semantic features to a unified textual semantic space and combines textual semantic vectors to construct a normalized three-dimensional feature matrix. Finally, it extracts keywords and matches them with a knowledge graph through vector retrieval and keyword retrieval. This achieves the goal of solving the problems of subjectivity and fragmented multimodal data in traditional diagnosis and treatment, realizing objectivity and precision in auxiliary diagnosis and treatment of mental illness, and generating authoritative auxiliary diagnosis and treatment plans.

[0005] The objective of this invention is achieved through the following technical solution:

[0006] The auxiliary diagnosis and treatment method based on a multimodal fusion large model includes the following steps:

[0007] Step 1: Acquire speech data, physiological signal data, behavioral data, and text data;

[0008] Step 2: Extract the peak values ​​from the speech data as the speech feature sequence and the abnormal points of the physiological signal in the physiological signal data as the physiological feature sequence. Use the dynamic time warping algorithm to dynamically adjust and align the time of the speech feature sequence and the physiological feature sequence.

[0009] Step 3: Map the aligned speech feature sequence, physiological feature sequence, and behavioral data—three non-textual modalities—to a unified text semantic space. Simultaneously, extract the text semantic vectors from the text data. The speech semantic vectors and text semantic vectors are fused to form the language modality. Construct a three-dimensional feature matrix of language, physiology, and behavior according to the three dimensions of time, modality, and semantics. Normalize the three-dimensional feature matrix and verify the semantic correlation between modalities. Output a standardized three-dimensional feature matrix.

[0010] Step 4: Extract search keywords based on the standardized three-dimensional feature matrix, and match the semantic vectors of the knowledge graph from the knowledge graph through vector retrieval and keyword retrieval to generate corresponding auxiliary diagnosis and treatment plans.

[0011] Preferably, step 2 specifically includes the following sub-steps:

[0012] Sub-step 2.1: Convert the timestamps of the speech data and the timestamps of the physiological signals to the same time coordinate system, extract the peak values ​​in the speech data as the speech feature sequence, and extract the abnormal points of the physiological signals in the physiological signal data as the physiological feature sequence;

[0013] Sub-step 2.2: Identify peak points in the speech feature sequence and abnormal points in the physiological feature sequence using the threshold method;

[0014] Sub-step 2.3: Construct a distance matrix by calculating the Euclidean distance between peak points and outliers using time difference and feature correlation as metrics.

[0015] Sub-step 2.4: Use the dynamic time warping algorithm to traverse the distance matrix and find the path with the minimum cumulative distance between peak points and outliers as the optimal matching relationship;

[0016] Sub-step 2.5: Calculate the global offset compensation value based on the optimal matching relationship, and correct the timestamp of the physiological feature sequence according to the global offset compensation value.

[0017] As a preferred option, the aligned speech feature sequence and physiological feature sequence are also verified. If the average time difference between the peak points in the speech feature sequence and the outliers in the physiological feature sequence exceeds the set threshold, the distance weight and path constraint parameters of the dynamic time warping algorithm are readjusted, and sub-steps 2.4 and 2.5 are repeated until the average time difference between the peak points in the speech feature sequence and the outliers in the physiological feature sequence does not exceed the set threshold.

[0018] Preferably, in step 3, the aligned speech feature sequence, physiological feature sequence, and behavioral data—three non-textual modalities—are mapped to a unified text semantic space, specifically as follows:

[0019] The speech feature sequence is input into the speech branch of Q-Former (query vector converter), and attention matching is performed between the pre-trained query vector group and the speech features to output 100 768-dimensional speech semantic vectors.

[0020] The physiological signal sequence is input into the physiological branch of Q-Former, and matched with the physiological signal sequence through a dedicated query vector group, outputting 100 physiological semantic vectors of 768 dimensions.

[0021] Input the behavioral data into the behavioral branch of Q-Former, and output 100 768-dimensional behavioral semantic vectors after querying and matching the behavioral vectors.

[0022] Preferably, in step 3, extracting the text semantic vector of the text data specifically involves:

[0023] The BERT model (a bidirectional encoder representation model of Transformer) is used to process text data, and 100 768-dimensional text semantic vectors are output, with a one-to-one correspondence between the text and non-text modal time frames.

[0024] Preferably, in step 3, verifying the semantic correlation between modalities specifically involves:

[0025] Calculate the cosine similarity of the semantic vectors of the three modalities—speech feature sequence, physiological feature sequence, and behavioral data—within the same time frame. If the similarity within a certain time frame is less than a set threshold, re-execute the semantic feature mapping until the similarity threshold requirement is met.

[0026] Preferably, the standardized three-dimensional feature matrix is ​​a weighted standardized three-dimensional feature matrix. The weighted standardized three-dimensional feature matrix is ​​used to construct a modal reliability assessment model, dynamically adjusting the modal weight factors based on historical data. Specifically:

[0027] Acquire multimodal historical data from different population groups and extract modal reliability features;

[0028] An evaluation model was constructed, with modal reliability characteristics as input and the actual contribution of the modality to the diagnosis and treatment outcome as the output label. The evaluation model was trained to quantify the reliability of different populations and different modalities.

[0029] Establish a weight factor mapping rule to map the reliability score output by the evaluation model to the modality weight factor according to the normalization ratio;

[0030] The real-time reliability index of the modal data is calculated in real time, and the real-time reliability index is input into the evaluation model to obtain the real-time reliability score.

[0031] The weighting factors of the modality are dynamically adjusted based on the real-time reliability score.

[0032] Preferably, in step 4, semantic vectors of the knowledge graph are matched from the knowledge graph through vector retrieval and keyword retrieval, specifically as follows:

[0033] The search keywords are converted into 768-dimensional semantic vectors. The cosine similarity between the keywords and the semantic vectors of the knowledge graph is calculated in the vector database, and candidate knowledge with similarity greater than a set threshold is selected.

[0034] Filter knowledge entries containing core terms by keyword search, and exclude irrelevant knowledge;

[0035] Candidate knowledge is prioritized, sorted, and integrated in a structured manner to generate an association table, clarifying the correspondence between the semantic vector of the knowledge graph and the candidate knowledge.

[0036] The auxiliary diagnosis and treatment system based on a multimodal fusion large model is applicable to auxiliary diagnosis and treatment methods based on a multimodal fusion large model, including:

[0037] The data acquisition layer receives voice, text, physiological signals, and behavioral data.

[0038] The model computation layer includes a full-modal data fusion module, a knowledge-enhanced large model module, and a personalized solution generation module; the model computation layer undertakes the core algorithm calculations and generates auxiliary diagnosis and treatment plans based on the data from the data acquisition layer.

[0039] The application service layer outputs the auxiliary diagnosis and treatment plan generated by the model calculation layer to the user terminal.

[0040] The security layer employs federated learning, differential privacy, and hierarchical access control to enhance data security.

[0041] Preferably, the model computation layer also includes a lightweight deployment module for compressing the models in the full-modal data fusion module, the knowledge-enhanced large model module, and the personalized solution generation module.

[0042] The beneficial effects of this invention are: by integrating multimodal data of speech, physiological signals, behavior and text, it breaks the reliance of traditional diagnosis and treatment on subjective experience and single text scales, provides a comprehensive and objective reference for symptom recognition, effectively reduces the problems of misdiagnosis and missed diagnosis caused by subjective bias or one-sided data, and improves the consistency of diagnostic results in different scenarios.

[0043] Overcoming the limitations of existing technologies that process only single data points and cannot capture hidden symptoms, this technology can integrate and analyze early signs such as abnormal speech and tone, fluctuations in physiological signals, and changes in behavioral patterns, to more accurately identify early mental illnesses such as mild depression and anxiety, providing technical support for early detection and intervention.

[0044] Knowledge graphs, combined with retrieval enhancement technology, enable the generated auxiliary diagnosis and treatment plans to be traceable to professional medical evidence, effectively suppressing large model illusions, reducing the risk of erroneous diagnosis and treatment recommendations, and increasing doctors' trust in and adoption of the plans.

[0045] It supports lightweight deployment, can adapt to the hardware conditions of primary medical institutions and remote areas, lowers the threshold for technology implementation, helps to make up for the uneven distribution of high-quality mental health resources, and makes auxiliary diagnosis and treatment services available to more people in areas lacking professional diagnosis and treatment resources. Attached Figure Description

[0046] Figure 1 This is a flowchart of the present invention;

[0047] Figure 2 This is a flowchart of the dynamic time warping algorithm of the present invention;

[0048] Figure 3 This is a flowchart of the multimodal data fusion method of the present invention;

[0049] Figure 4 This is the flowchart of the knowledge-enhanced reasoning process of this invention. Detailed Implementation

[0050] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0051] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0052] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0053] Example: Assisted diagnosis and treatment methods based on multimodal fusion large models, such as... Figure 1 As shown, it includes the following steps:

[0054] Step 1: Acquire speech data, physiological signal data, behavioral data, and text data;

[0055] Step 2: Extract the peak values ​​from the speech data as the speech feature sequence and the abnormal points of the physiological signal in the physiological signal data as the physiological feature sequence. Use the dynamic time warping algorithm to dynamically adjust and align the time of the speech feature sequence and the physiological feature sequence.

[0056] Step 3: Map the aligned speech feature sequence, physiological feature sequence, and behavioral data—three non-textual modalities—to a unified text semantic space. Simultaneously, extract the text semantic vectors from the text data. The speech semantic vectors and text semantic vectors are fused to form the language modality. Construct a three-dimensional feature matrix of language, physiology, and behavior according to the three dimensions of time, modality, and semantics. Normalize the three-dimensional feature matrix and verify the semantic correlation between modalities. Output a standardized three-dimensional feature matrix.

[0057] Step 4: Extract search keywords based on the standardized three-dimensional feature matrix, and match the semantic vectors of the knowledge graph from the knowledge graph through vector retrieval and keyword retrieval to generate corresponding auxiliary diagnosis and treatment plans.

[0058] Specifically, such as Figure 2 , Figure 3 As shown, in step 2, the alignment prerequisites are first set, including:

[0059] Input data requirements: Speech data sampling rate ≥ 16kHz, physiological signal (heart rate variability, skin conductance) sampling rate ≥ 100Hz, both carrying precise timestamps (error ≤ 1ms).

[0060] Data preprocessing: Noise reduction and endpoint detection are performed on the speech data, and emotion-related acoustic features (fundamental frequency, energy, speech rate) are extracted; physiological signals are filtered (motion artifact removal) and normalized, and abnormal fluctuation segments are preserved.

[0061] The specific alignment steps are as follows:

[0062] Unified initialization of the time axis: Based on the system's local high-precision clock, the timestamps of the speech data and the physiological signal are converted to the same time coordinate system, generating two independent time-series data sequences: speech feature sequence S and physiological signal sequence P.

[0063] Key event point extraction: Identify voice emotion peaks (such as sudden energy increases, abnormal fluctuations in fundamental frequency) in S using a threshold method, and label them as S1, S2, ..., Sn (each point contains a timestamp Ts and feature intensity); Identify abnormal physiological signal points (such as sudden drops in heart rate variability, peak skin conductance) in P, and label them as P1, P2, ..., Pm (each point contains a timestamp Tp and signal amplitude).

[0064] DTW (Dynamic Time Warping) Algorithm Offset Calculation: A distance matrix is ​​constructed, using time difference and feature correlation as dual metrics to calculate the Euclidean distance between each Si and Pj, forming an n×m dimensional distance matrix. The optimal path is found by traversing the distance matrix using dynamic programming to solve for the path with the minimum cumulative distance (constraints are set: path slope range [-2,2] to avoid excessive time distortion). This path corresponds to the optimal matching relationship between Si and Pj. Offset compensation is calculated based on the optimal matching path. The time difference Δt for each matching pair (Si, Pj) is statistically analyzed, and the global offset compensation value ΔT is calculated using a weighted average (feature strength / signal amplitude as weights).

[0065] Dynamic alignment adjustment: The timestamp of the physiological signal sequence P is offset and corrected as a whole according to ΔT (Tp'=Tp+ΔT). For isolated signal points that are not matched (such as a single physiological abnormality point without corresponding speech features), the time correlation information is supplemented by linear interpolation.

[0066] Alignment verification and iteration: Calculate the average time difference between the peak of speech emotion and the abnormal point of physiological signal after alignment (target ≤ 5ms). If the average exceeds the threshold, readjust the distance weight and path constraint parameters of the DTW algorithm until the millisecond-level alignment requirement is met (maximum time difference ≤ 10ms).

[0067] In this embodiment, an alignment effect guarantee mechanism is also set up. A local alignment result is generated every 500ms, and the offset compensation value is updated through a sliding window (window length 1s) to adapt to dynamic changes in the signal. Matching pairs that exceed the normal time difference range (Δt>50ms) are marked as abnormal and do not participate in the global offset calculation to avoid interfering with the overall alignment accuracy.

[0068] In step 3, the aligned speech feature sequence, physiological feature sequence, and behavioral data—three non-textual modalities—are mapped to a unified text semantic space. Specifically:

[0069] The aligned speech feature sequence (e.g., 100 time frames, 128-dimensional features per frame) is input into the speech branch of Q-Former. Attention matching is performed between the pre-trained query vector group (containing 128 query vectors) and the speech features, outputting 100 768-dimensional speech semantic vectors (denoted as V_voice=[v_v1,v_v2,...,v_v100], each vector corresponding to the speech semantics of one time frame).

[0070] Similarly, the physiological signal sequence (100 time frames, 64-dimensional features per frame) is input into the physiological branch of Q-Former, and matched through a dedicated query vector group (64 query vectors) to output 100 physiological semantic vectors of 768 dimensions (denoted as V_physio=[v_p1,v_p2,...,v_p100]).

[0071] The behavior data sequence (100 time frames, 32-dimensional features per frame) is input into the behavior branch of Q-Former. After query vector matching, 100 768-dimensional behavior semantic vectors are output (denoted as V_behavior=[v_b1,v_b2,...,v_b100]).

[0072] In step 3, the text semantic vector of the text data is extracted, specifically as follows:

[0073] The BERT model is directly used to process the spatiotemporally aligned text data (e.g., each time frame corresponds to one medical consultation text segment), and 100 768-dimensional text semantic vectors are output (denoted as V_text=[v_t1,v_t2,...,v_t100]), ensuring that the text and non-text modal time frames correspond one-to-one.

[0074] The aforementioned method involves constructing a three-dimensional feature matrix of language, physiology, and behavior based on the three dimensions of time, modality, and semantics. This three-dimensional feature matrix is ​​then normalized, and the semantic correlation between modalities is verified to output a standardized three-dimensional feature matrix. Specifically:

[0075] Three-dimensional dimensions determined:

[0076] Time dimension (T): The unit is the time frame after spatiotemporal alignment, set to 100 (corresponding to 100 consecutive time frames, each frame interval is 500ms, covering 50 seconds of diagnosis and treatment data), and the dimension index T∈[1,100].

[0077] Modality dimension (M): includes three modalities: language (text), physiology, and behavior. The dimension index M∈[1 (language), 2 (physiology), 3 (behavior)].

[0078] Semantic Dimension (YS): Unified as 768-dimensional semantic features output by Q-Former and BERT models, with dimension index WS∈[1,768].

[0079] Matrix element filling:

[0080] Construct an empty three-dimensional matrix M_total∈R^(T×M×WS) (i.e., a matrix of 100×3×768).

[0081] Fill in according to the time frame-modal-semantic correspondence:

[0082] When M=1 (language modality), M_total[T,1,YS]=V_text[T][WS] (the WS-th dimension value of the language semantic vector of the T-th time frame);

[0083] When M=2 (physiological modality), M_total[T,2,YS]=V_physio[T][WS] (the WS-th dimension value of the physiological semantic vector of the T-th time frame);

[0084] When M=3 (behavioral modality), M_total[T,3,YS]=V_behavior[T][WS] (the WS dimension value of the behavioral semantic vector of the Tth time frame).

[0085] Normalization: L2 normalization is performed on the three modal semantic vectors (language, physiology, and behavior) for each time frame in the matrix to ensure that the magnitudes of the semantic features of each modality are consistent, and to prevent a particular modality feature from dominating subsequent calculations due to excessively large values. The formula is as follows: ( (where L2 norm is the vector).

[0086] Validity verification: Calculate the cosine similarity of the semantic vectors of the three modalities within the same time frame (target similarity ≥ 0.3, to ensure reasonable semantic association between modalities). If the similarity of a certain time frame is < 0.3, re-execute Q-Former mapping (adjust the query vector weights) until the similarity requirement is met, and finally output the standardized three-dimensional feature matrix.

[0087] Constructing a three-dimensional feature matrix of language, physiology, and behavior has four core significances: First, it can solve the problem of multimodal semantic fragmentation. By using Q-Former technology, it unifies non-textual modalities such as speech, physiology, and behavior with textual modalities into a 768-dimensional textual semantic space. This breaks through the limitations of traditional solutions where different modal feature dimensions and semantic spaces are independent and cannot be directly fused and calculated, providing a calculable foundational data for subsequent adaptive weight allocation. Second, it can preserve spatiotemporal correlation information. The temporal dimension of the matrix fully presents the dynamic changes of each modality feature during the diagnosis and treatment process, avoiding the information loss of traditional static fusion. Combined with modal dimension features, it can accurately capture early hidden symptoms, improving the accuracy of mild depression identification. The accuracy rate has been improved from 60% in traditional single-modality models to ≥75%; thirdly, it can adapt to the input requirements of large-scale mental health models, serving as direct input to support subsequent diagnostic and treatment decisions. It supports the model in quickly matching language-physiological-behavioral feature combinations with the disease-symptom triad in the knowledge graph, and provides real-time feature support for dynamic plan adjustments, ultimately reducing the risk of hallucinations in large-scale models and increasing the accuracy rate of diagnostic suggestions to ≥85%; fourthly, it can ensure data consistency. Its standardized structure with fixed T×M×S dimensions can adapt to the output data of different acquisition devices, eliminating the need to redesign the fusion algorithm due to device differences. When deployed in a lightweight manner in primary healthcare institutions, the compressed matrix data can be directly transmitted, reducing the difficulty of cross-scenario deployment.

[0088] The standardized three-dimensional feature matrix is ​​a weighted standardized three-dimensional feature matrix. This weighted standardized three-dimensional feature matrix is ​​used to construct a modal reliability assessment model, dynamically adjusting the modal weight factors based on historical data. Specifically:

[0089] Three types of core historical data were collected: ① Multimodal data (voice, physiology, behavior) of different populations (adolescents / elderly / adults); ② Diagnosis and treatment results of the corresponding data (such as DSM-5 diagnostic conclusions and treatment effectiveness rates); ③ Correlation labels between modal data and diagnosis and treatment results (such as a high correlation between adolescent voice features and depressive symptoms, labeled as 0.8, and a high correlation between elderly physiological signals and anxiety symptoms, labeled as 0.9).

[0090] Extract modal reliability features, including data integrity (e.g., the proportion of speech frames without missing frames), feature discrimination (e.g., the difference in modal features under different disease states), and diagnostic relevance (e.g., the contribution of modal data to the diagnostic results), to form feature vectors (e.g., the feature vector of adolescent speech is [0.95, 0.88, 0.82]).

[0091] An evaluation model was constructed using gradient boosting trees (XGBoost): taking modal reliability features as input and the actual contribution of the modality to the diagnosis and treatment results as output labels, the model was trained to quantify the reliability of different populations and different modalities (the score range is 0-1, with 1 representing the highest reliability).

[0092] Establish a weighting factor mapping rule: Map the reliability score output by the model to a weighting factor according to a normalized ratio. The formula is: Weighting factor for a certain modality = Reliability score of that modality / Sum of reliability scores of all modalities. For example, in the adolescent group, the speech reliability score is 0.7, physiological reliability is 0.4, behavioral reliability is 0.3, and the total score is 1.4. Then, the speech weight = 0.7 / 1.4 = 50%? This needs to be corrected by combining historical annotations. Finally, through training, initial weights specific to each population are fitted: 35% for adolescent speech, 30% for physiological reliability, and 35% for behavioral reliability; and 25% for elderly speech, 40% for physiological reliability, and 35% for behavioral reliability.

[0093] Real-time reliability metrics for modal data are calculated, such as the signal-to-noise ratio (SNR) of speech data, the signal fluctuation stability of physiological data (e.g., the standard deviation of heart rate variability), and the recording integrity of behavioral data (e.g., the missing rate of sleep cycle data). These real-time metrics are then input into a trained evaluation model to obtain a real-time reliability score.

[0094] Dynamically update weighting factors: The weights are adjusted every 5 minutes (or 1 treatment segment) based on the real-time reliability score. If the real-time score of a certain modality decreases (e.g., the signal-to-noise ratio of adolescent speech drops from 30dB to 15dB, and the reliability score drops from 0.7 to 0.4), its weight is reduced synchronously according to the mapping rules (e.g., from 35% to 20%) to ensure that the weight matches the actual reliability of the current modality data.

[0095] Noise data assessment is achieved through a dual mechanism of multi-dimensional threshold detection and feature anomaly identification, with specific assessment criteria developed for different noise modalities:

[0096] Speech modal noise detection (such as slurred speech and ambient noise).

[0097] Key performance indicators: ① Signal-to-noise ratio (SNR): The ratio of speech signal energy to noise energy is calculated. When SNR < 10dB, it is considered noise data. ② Frame loss rate: The proportion of invalid frames (such as silence frames and plosive distortion frames) in the total number of frames is counted. If the proportion is > 15%, it is considered noise data. ③ Semantic integrity: The semantic coherence of speech-to-text is detected by a lightweight NLP model (such as Mini-BERT). If the proportion of semantically fragmented sentences (such as illogical and scattered words) is > 20%, it is considered noise data.

[0098] Physiological modal noise assessment (such as motion artifacts and sensor interference).

[0099] Key judgment indicators: ① Signal fluctuation range: Set the normal fluctuation threshold of physiological signals (e.g., normal range of heart rate variability is 50-150ms, normal range of skin conductance is 1-5μS). When the signal exceeds the normal range and the duration is >3 seconds, it is judged as noise data; ② Signal mutation rate: Calculate the signal difference between adjacent sampling points. If the difference is >2 times the normal fluctuation range and there is no reasonable clinical explanation (e.g., the patient has not engaged in strenuous exercise), it is judged as noise data; ③ Sensor status: Detect whether there is poor contact of the sensor (e.g., the wearable device heart rate monitoring electrode falls off) and data transmission interruption through the device feedback signal. Data with such conditions is judged as noise data.

[0100] Behavioral modality noise detection (such as incorrect activity recording or mislabeling of sleep data).

[0101] Key judgment indicators: ① Data consistency: Compare the behavioral data of different devices within the same time period (such as the activity records of mobile phone motion sensors and smart bracelets). If the difference is greater than 30% and there is no device malfunction, it is judged as noise data; ② Logical rationality: For example, if the sleep data shows that the nighttime awake time is greater than 8 hours and the activity level is 0, but there are voice interaction records at the corresponding time, such data that does not conform to physiological logic is judged as noise data.

[0102] In this embodiment, a method for determining the adjustment range of automatic weight reduction is also provided. The adjustment range of automatic weight reduction is determined based on a triple logic of noise level classification, preset weight gradient, and real-time reliability compensation, ensuring that the range is reasonable and does not affect the robustness of the fusion result.

[0103] Noise Level Classification:

[0104] Based on the degree to which noise disrupts modal information, noise data is divided into three levels:

[0105] Mild noise: Only some features are affected (e.g., speech SNR 10-15dB, frame loss rate 5%-15%), and the modality still retains more than 60% of the effective information;

[0106] Moderate noise: Core features are affected (e.g., speech SNR 5-10dB, frame loss rate 15%-30%), and modalities retain only 30%-60% of the effective information;

[0107] Heavy noise: Most features are invalid (e.g., speech SNR < 5dB, frame loss rate > 30%), and effective modal information < 30%.

[0108] Preset weight gradient correspondence rules:

[0109] Based on the noise level, a basic reduction in weight is set. Referring to the optimal matching relationship between noise level and weight adjustment in historical data, the following rules are formulated:

[0110] Mild noise: Reduce the weight by 20%-30% based on the current weight. For example, if the original speech weight is 35%, it will be reduced to 24.5%-28%.

[0111] Moderate noise: Reduce the weighting by 50%-70% based on the current weighting. For example, if the original speech weight is 35%, the weighting will be reduced to 10.5%-17.5%.

[0112] Heavy noise: Reduced to the lowest weight (5%). Regardless of the original weight, the weight of heavy noise mode is uniformly set to 5% (to avoid invalid data interfering with the fusion results).

[0113] Real-time reliability compensation adjustment:

[0114] If reducing the weight of a certain modality improves the real-time reliability score of other modalities (e.g., speech is reduced to 5% due to heavy noise, but the physiological signal reliability score increases from 0.4 to 0.6), then a fine-tuning adjustment is made through weight redistribution: the weight reduced by the reduced-weight modality is allocated to the effective modalities according to the reliability scores of other modalities. The formula is: supplementary weight allocation for an effective modality = (weight reduced by the reduced-weight modality) × (reliability score of that effective modality / sum of reliability scores of all effective modalities). For example, if speech is reduced by 30% (from 35% to 5%), physiological reliability is 0.6, behavioral reliability is 0.4, and the total effective score is 1.0, then physiological supplementary allocation is 30% × 0.6 = 18%, and behavioral supplementary allocation is 30% × 0.4 = 12%. The final weights are speech 5%, physiological 48%, and behavioral 47%, ensuring that the total weight sum is 100% and the fusion result is robust.

[0115] like Figure 4 As shown, in step 4, semantic vectors of the knowledge graph are matched from the knowledge graph through vector retrieval and keyword retrieval, specifically as follows:

[0116] By introducing Retrieval Enhanced Generation (RAG) technology, authoritative evidence is obtained from the knowledge graph in real time when the model generates diagnoses / solutions, avoiding fabricated content. RAG technology is embedded in the entire process of diagnostic reasoning and solution generation, and is divided into four steps: retrieval triggering, multi-dimensional matching, evidence integration, and constraint generation, as detailed below:

[0117] Triggered Search—Extracting Search Keywords Based on a 3D Feature Matrix

[0118] After the system receives the spatiotemporally aligned and feature-mapped 3D feature matrix (containing language-physiological-behavioral semantic features), it first extracts the core search keywords through the feature-text conversion module:

[0119] Extract keywords of patient complaints and symptom descriptions (such as persistent low mood for 2 weeks and early nighttime awakening) from language modal semantic vectors.

[0120] Extract abnormal indicator keywords from physiological modality semantic vectors (such as abnormal peak skin signal with heart rate variability ≤50ms).

[0121] Extract behavioral feature keywords from behavioral modality semantic vectors (e.g., 80% reduction in daily activity and sleep cycle disorder).

[0122] Keywords are merged to form a search query (e.g., depressed mood for 2 weeks + early awakening + heart rate variability ≤50ms + reduced activity level by 80%), while patient population attributes are labeled (e.g., adolescents and the elderly) for precise matching of population-specific diagnosis and treatment rules in the knowledge graph.

[0123] Multi-dimensional retrieval and matching—obtaining authoritative evidence from the knowledge graph, and matching relevant authoritative knowledge from the knowledge graph based on the retrieval query through a dual approach of vector retrieval and keyword retrieval, specifically including:

[0124] Vector semantic matching: The search query is converted into a 768-dimensional semantic vector. The cosine similarity between the query and the semantic vector of the knowledge graph is calculated in the Milvus vector database. Candidate knowledge with a similarity of ≥0.85 is selected (e.g., adolescent + low mood for 2 weeks + early awakening → moderate depressive tendency, refer to Article 3.2.1 of the "Guidelines for the Prevention and Treatment of Depression in China").

[0125] Precise keyword matching: Filter knowledge entries containing core terms such as early awakening, abnormal heart rate variability, and adolescents through keyword indexing, and exclude irrelevant knowledge (such as excluding entries related to hypertension and emotional problems in the elderly).

[0126] Prioritization: Candidate knowledge is sorted according to the priority of guideline basis > case basis > literature basis (e.g., first match the explicit clauses of DSM-5 and Chinese guidelines, then match the treatment plans of similar cases, and finally supplement the literature to support the evidence). The top 5 most relevant authoritative evidence is taken as the reference for generation.

[0127] Based on the integration—generating a correlation table of diagnostic and treatment conclusions and authoritative evidence, the top 5 authoritative evidence retrieved are structurally integrated to generate a correlation table, clarifying the correspondence between diagnostic conclusions / treatment plans and authoritative sources.

[0128] Generate constraints—Based on authoritative evidence, the model output is limited. The integrated diagnosis and treatment conclusion-authoritative evidence association table is used as a constraint and input into the generation module of the Jingwei Big Model. The following rules are used to avoid fabricated content:

[0129] Output content binding basis: When the model generates diagnostic reports or treatment plans, it must mark the corresponding authoritative basis for each core conclusion (such as the sertraline medication recommendation for moderate depression) (such as according to Article 3.2.1 of the 2023 edition of the "Guidelines for the Prevention and Treatment of Depression in China"). Conclusions that do not match authoritative basis (such as the medication recommendation for a niche drug generated by the model) are prohibited from being output.

[0130] Conflict correction: If the content initially generated by the model conflicts with the authoritative evidence retrieved (e.g., the model recommends that adolescents use monoamine oxidase inhibitors, but the guidelines clearly state that such drugs are contraindicated for adolescents), automatic correction will be triggered to adjust the output content according to the authoritative evidence.

[0131] Supplementing basis for fuzzy content: If the model generates a fuzzy statement (such as recommending the use of antidepressants), then supplement specific information based on the search criteria (such as recommending the use of the antidepressant sertraline, according to DSM-54.1.2) to ensure that the output content is specific and traceable.

[0132] In this embodiment, the hallucination suppression effect of RAG technology is guaranteed by the following design: Search coverage verification: After each generation, the proportion of authoritative evidence bound in the model output conclusion is automatically counted, and the coverage is required to be ≥95%. If it is lower than this threshold, the search is retried (adjusting the search query keywords, such as adding treatments specifically for adolescents), until the coverage reaches the target.

[0133] Based on validity verification: The knowledge graph is updated regularly (monthly), and outdated guidelines (such as old versions of depression prevention and treatment guidelines) and low-quality literature (impact factor < 2.0) are removed to ensure the timeliness and authority of the search criteria; at the same time, two chief psychiatrists review the accuracy of the matching of 100 randomly selected generated contents, requiring an accuracy rate of ≥ 98%, otherwise the search matching algorithm is optimized (such as adjusting the semantic similarity threshold).

[0134] User feedback iteration: A questioning function is set up in the doctor assistant. If doctors believe that the authoritative basis and conclusion of the model annotation do not match (such as incorrect basis annotation), they can submit feedback. The system records such cases and uses them to optimize RAG search rules (such as adjusting keyword matching weights) to continuously reduce the risk of hallucination.

[0135] In this embodiment, a corresponding auxiliary diagnosis and treatment plan is generated, specifically as follows:

[0136] Based on patient baseline data and multimodal targets, the system invokes knowledge graph diagnostic rules to generate auxiliary treatment plans. The core process consists of four steps: First, the input data is preprocessed, extracting key information such as age and medical history from the patient baseline data, and extracting quantitative features (such as abnormal indicators in voice, physiology, and behavior dimensions) from the multimodal targets to form structured matching data. Next, rule matching is performed, firstly screening out inapplicable rules based on baseline data (e.g., excluding rules specific to the elderly based on adolescent data), then finely screening out rules that meet the conditions based on multimodal targets, and finally determining the optimal rule based on the priority of core rules > supplementary rules > alternative rules. Subsequently, rule reasoning is used for verification, retrieving historical cases in the knowledge graph with a similarity of ≥90% to the current patient data to verify the treatment effectiveness of the rule plan, while also identifying hidden contraindications (such as the risk of drug allergies). Finally, the plan is assembled, extracting drug and psychological intervention elements from the optimal rule, supplementing details such as frequency of administration and training duration, and annotating authoritative evidence to form a complete integrated drug + psychological intervention plan.

[0137] The aforementioned auxiliary treatment plan can be dynamically adjusted. It adopts reinforcement learning (DQN algorithm) and uses the task completion rate of symptom improvement as the reward signal. If the patient fails to complete cognitive behavioral therapy (CBT) exercises for 3 consecutive days, it will automatically switch to 5 minutes of lightweight mindfulness training. If the sleep duration is shortened by ≥30 minutes, a bedtime relaxation audio module will be added.

[0138] The specific process of dynamic adjustment of the DQN algorithm:

[0139] The DQN algorithm achieves dynamic adjustment of the scheme through four steps: state awareness, action decision-making, reward calculation, and policy update.

[0140] State Awareness: Real-time Acquisition and Quantification of Environmental State The system collects two types of data in real time through the patient's end (APP / wearable device), converting the environmental state into a computable feature vector: ① Task completion rate data (e.g., CBT practice completion status is recorded as 0 = not completed, 1 = completed, and if it is not completed for 3 consecutive days, it is marked as task completion rate feature = 0); ② Symptom improvement data (e.g., sleep duration is calculated as the difference between actual duration and baseline duration, and if it is shortened by ≥30 minutes, it is marked as sleep feature = -30, and at the same time, it combines multimodal data such as voice emotion features and heart rate variability to form a state vector with unified dimensions (e.g., [task completion rate 0, sleep feature -30, emotion feature 0.2]).

[0141] Action Decision: The agent selects the optimal adjustment strategy based on a deep Q-network. The deep Q-network of the agent consists of a state feature extraction layer, a fully connected hidden layer, and a Q-value output layer: ① The input layer receives the above state vector and extracts key state features such as low compliance and sleep abnormalities through convolutional layers; ② The hidden layer processes the features through the ReLU activation function and learns the mapping relationship between the state and the action; ③ The output layer outputs the Q-value corresponding to each adjustment action (representing the expected long-term reward after selecting the action). The agent selects the action with the highest Q-value to execute (e.g., for the state of not completing CBT + sleep shortening for 3 consecutive days, switching to lightweight mindfulness training + adding pre-sleep relaxation audio has the highest Q-value, so this combination action is executed), which is completely consistent with the adjustment logic in the example in the handover document.

[0142] Reward Calculation: After the quantified feedback action based on symptom improvement and task completion rate (e.g., switching to lightweight mindfulness training), the system collects new patient data within 1-3 days and calculates the reward value: ① If the patient completes lightweight mindfulness training (task completion rate = 1) and sleep duration returns to the baseline level (symptom improvement meets the standard), a positive reward of +8 is given; ② If the training is not completed but the reduction in sleep duration decreases (e.g., from a reduction of 40 minutes to a reduction of 10 minutes), a neutral to positive reward of +2 is given; ③ If the training is not completed and there is no improvement in sleep, a negative reward of -3 is given. The reward results are directly fed back to the DQN algorithm.

[0143] Strategy Update: Iterative Optimization of Action Selection Logic. The DQN algorithm optimizes its strategy through experience replay and target network updates: ① Stores the experience data of state-action-reward-next state in the experience pool, and randomly samples batch data to train the Q-network to avoid data correlation affecting training stability; ② Updates the target network parameters regularly (e.g., every 7 days) to make the Q-value calculation closer to the actual reward feedback, and gradually optimizes the action selection strategy (e.g., continuously improving the Q-value of the action to be switched to when the patient has low compliance, ensuring that the action is selected first in similar scenarios in the future). Ultimately, this achieves the goal of adjusting the plan to better fit the patient's actual state, improving treatment compliance and effectiveness, and matching the core requirement of dynamically adjusting and optimizing the treatment plan in the disclosure document.

[0144] The auxiliary diagnosis and treatment system based on a multimodal fusion large model is applicable to auxiliary diagnosis and treatment methods based on a multimodal fusion large model, including:

[0145] The data acquisition layer receives voice, text, physiological signals, and behavioral data.

[0146] The model computation layer includes a full-modal data fusion module, a knowledge-enhanced large model module, and a personalized solution generation module; the model computation layer undertakes the core algorithm calculations and generates auxiliary diagnosis and treatment plans based on the data from the data acquisition layer.

[0147] The application service layer outputs the auxiliary diagnosis and treatment plan generated by the model calculation layer to the user terminal.

[0148] The security layer employs federated learning, differential privacy, and hierarchical access control to enhance data security.

[0149] The model computation layer also includes a lightweight deployment module for compressing models in the full-modal data fusion module, the knowledge-enhanced large model module, and the personalized solution generation module.

[0150] In summary, the full-modal data fusion module is the core of the entire system's data preprocessing, and its output directly supports the core functions of the knowledge enhancement big model module and the personalized solution generation module.

[0151] The module provides aligned multimodal semantic data for the knowledge enhancement model module: This module addresses the temporal offset issue of speech, physiological, and behavioral data through spatiotemporal alignment (DTW algorithm), and then uses Q-Former technology to map non-textual modalities to a unified textual semantic space, constructing a three-dimensional feature matrix of language-physiology-behavior (768 semantic dimensions). This matrix is ​​directly input into the knowledge enhancement model module as the core query source for RAG technology to retrieve knowledge graphs—for example, extracting structured features from the matrix such as adolescents + 65% negative vocabulary + 45ms heart rate variability, providing accurate data support for the large model to match mindfulness training + low-dose SSRIs treatment rules.

[0152] Provide target feature data for the personalized solution generation module: The fused three-dimensional feature matrix will further extract multimodal targets (such as sleep duration shortened by 40 minutes if CBT practice is not completed for 3 consecutive days). These target data are the key input conditions for personalized solution generation. For example, a target of sleep duration shortened by ≥30 minutes will trigger the solution generation module to call the sleep intervention rules in the knowledge graph and add a bedtime relaxation audio module.

[0153] The knowledge-enhancing big model module is the rule hub for solution generation. It uses RAG technology to retrieve the results from the knowledge graph and directly determines the core content of the personalized solution.

[0154] To provide matching treatment rules for solution generation: When the personalized solution generation module receives the patient's baseline data (such as adolescents, no history of drug allergies) + multimodal targets, it will trigger the RAG retrieval process of the knowledge enhancement big model module. First, it extracts search keywords from the three-dimensional feature matrix, and then matches the IF-THEN treatment rules in the knowledge graph (such as the R001 rule: adolescents + negative words over 60% → mindfulness training + low-dose SSRIs) through the vector database + keyword index. It also returns the drug dosage, intervention duration and authoritative evidence corresponding to the rule to ensure that the solution complies with DSM-5, Chinese guidelines for the prevention and treatment of depression and other standards.

[0155] To provide a basis for rule verification for dynamic adjustments: During the dynamic adjustment phase of the personalized plan (such as when a patient fails to complete CBT for 3 consecutive days), the knowledge enhancement big model module will search the knowledge graph again to verify the rationality of switching to lightweight mindfulness training—for example, searching for the adaptation rules for low compliance scenarios in adolescents, confirming that the adjustment conforms to the clinical logic of reducing task difficulty to improve compliance, and avoiding deviation of dynamic adjustments from the treatment guidelines.

[0156] The personalized solution generation module is the core of the system's decision-making and execution, taking the output results from the first two modules to complete the closed loop of solution generation and dynamic optimization.

[0157] The initial plan is generated based on the results of the preceding modules: the target features of the full-modal data fusion module and the matching rules of the knowledge enhancement big model module are integrated to generate a comprehensive drug + psychological intervention plan. For example, by combining the target features of adolescents and the R001 rule, an initial plan of sertraline 25mg / day + 15 minutes of CBT practice per day is output, and authoritative evidence is marked.

[0158] Dynamic optimization scheme based on DQN algorithm: During the scheme execution, patient feedback data (such as CBT completion rate and sleep duration) is collected in real time and input into the full modality data fusion module to regenerate dynamic target features (such as CBT target not completed for 3 consecutive days); after confirming the rationality of the adjustment rules by calling the knowledge enhancement big model module, the optimal adjustment action is selected through DQN algorithm (such as switching to 5 minutes of lightweight mindfulness training), and the task completion rate of symptom improvement is used as a reward signal to iteratively optimize the adjustment strategy to ensure that the scheme continuously adapts to the patient's condition.

[0159] The lightweight deployment module is the foundation for the system's large-scale application. Through model compression and multi-scenario adaptation, the core functions of the first three modules can be implemented in grassroots scenarios: compressing the model size of the first three modules lowers the hardware threshold: through knowledge distillation and parameter pruning techniques, the Q-Former model of the full-modal data fusion module, the RAG retrieval model of the knowledge enhancement large model module, and the DQN model of the personalized solution generation module are compressed from 100 billion parameters to 5 billion parameters. Then, through INT8 precision quantization, the model can run on ordinary computers (memory ≤ 8GB), meeting the hardware conditions of grassroots health centers, schools, and other scenarios.

[0160] This embodiment's solution adapts to data flow requirements across multiple scenarios: it develops browser plugins (for connecting to hospital HIS systems), WeChat mini-programs (for school screening), and offline clients (for grassroots scenarios without network access), ensuring smooth data flow from the first three modules in different scenarios. For example, in remote areas without network access, the offline client can locally store the feature matrix after full-modal data fusion. Once the network is restored, it is synchronized to the knowledge-enhanced big model module to complete rule matching, and then the solution is sent back to the local machine, achieving a closed loop of offline collection, online matching, and local execution.

[0161] This embodiment's solution revolves around the core logic of data acquisition → data analysis → result output → optimization and implementation, with each module collaboratively completing the entire process: The data acquisition phase is led by the multimodal data fusion module, which collects voice, physiological, behavioral, and text data through professional recording devices, wearable devices, and consultation systems. After noise reduction and filtering preprocessing, spatiotemporal alignment is achieved using the DTW algorithm, and a three-dimensional feature matrix of language, physiology, and behavior is generated using Q-Former technology. Simultaneously, multimodal targets such as a 65% proportion of negative vocabulary and abnormal heart rate variability are extracted. The data analysis phase is led by the knowledge enhancement big model module, which combines the targets in the three-dimensional feature matrix with the patient's baseline data (age, medical history) to generate a RAG retrieval query. This query matches treatment rules in the knowledge graph with a vector database and keyword index, outputting treatment rules that conform to authoritative guidelines (such as drug type and intervention method), and verifying rule validity by associating similar cases and excluding contraindications. The result output phase is led by the personalized treatment plan generation module. The system is modularly driven, generating an initial medication + psychological intervention plan (e.g., sertraline 25mg / day + 15 minutes of CBT practice) based on matching treatment rules. This plan is pushed to doctors and patients via lightweight browser plugins or WeChat mini-programs. Doctors can fine-tune the plan based on their clinical experience to adapt to individual differences. The optimization and implementation phase is jointly led by the dynamic adjustment module and the lightweight deployment module. In the dynamic optimization phase, patient plan execution data (e.g., CBT completion rate, sleep duration) is collected in real time and re-entered into the full-modal data fusion module to generate dynamic targets. After confirming the adjustment rules by calling the knowledge-enhanced big model module, the optimal adjustment action (e.g., switching to lightweight training, adding relaxation audio) is selected through the DQN algorithm, and the strategy is iterated with reward signals. In the scenario implementation phase, the adjusted plan is implemented in grassroots, school, and other scenarios through lightweight carriers such as offline clients and mini-programs. At the same time, patient data is stored locally to ensure privacy and security, ultimately achieving a closed loop of data collection-analysis-output-optimization-implementation.

[0162] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0163] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An assisted diagnosis and treatment method based on a multimodal fusion large model, characterized by: Includes the following steps: Step 1: Acquire speech data, physiological signal data, behavioral data, and text data; Step 2: Extract the peak values ​​from the speech data as the speech feature sequence and the abnormal points of the physiological signal in the physiological signal data as the physiological feature sequence. Use the dynamic time warping algorithm to dynamically adjust and align the time of the speech feature sequence and the physiological feature sequence. Step 3: The Q-Former model is used to map the features of the three non-text modalities—aligned speech feature sequences, physiological feature sequences, and behavioral data—to a unified text semantic space. At the same time, text semantic vectors are extracted from the text data. The speech semantic vectors and text semantic vectors are fused to form the language modality. A three-dimensional feature matrix of language, physiology, and behavior is constructed according to the three dimensions of time, modality, and semantics. The three-dimensional feature matrix is ​​normalized and the semantic correlation between modalities is verified. A weighted standardized three-dimensional feature matrix is ​​output. The modality reliability assessment model is constructed to dynamically adjust the weight factors of each modality based on real-time data quality and historical data. Step 4: Extract search keywords based on a weighted standardized three-dimensional feature matrix, and match the semantic vectors of the knowledge graph from the knowledge graph through vector retrieval and keyword retrieval to generate corresponding auxiliary diagnosis and treatment plans.

2. The assisted diagnosis and treatment method based on a multimodal fusion large model according to claim 1, characterized in that, Step 2 specifically includes the following sub-steps: Sub-step 2.1: Convert the timestamps of the speech data and the timestamps of the physiological signals to the same time coordinate system, extract the peak values ​​in the speech data as the speech feature sequence, and extract the abnormal points of the physiological signals in the physiological signal data as the physiological feature sequence; Sub-step 2.2: Identify peak points in the speech feature sequence and abnormal points in the physiological feature sequence using the threshold method; Sub-step 2.3: Construct a distance matrix by calculating the Euclidean distance between peak points and outliers using time difference and feature correlation as metrics. Sub-step 2.4: Use the dynamic time warping algorithm to traverse the distance matrix and find the path with the minimum cumulative distance between peak points and outliers as the optimal matching relationship; Sub-step 2.5: Calculate the global offset compensation value based on the optimal matching relationship, and correct the timestamp of the physiological feature sequence according to the global offset compensation value.

3. The assisted diagnosis and treatment method based on a multimodal fusion large model according to claim 2, characterized in that, The aligned speech feature sequence and physiological feature sequence are also verified. If the average time difference between the peak points in the speech feature sequence and the outliers in the physiological feature sequence exceeds the set threshold, the distance weight and path constraint parameters of the dynamic time warping algorithm are adjusted, and sub-steps 2.4 and 2.5 are repeated until the average time difference between the peak points in the speech feature sequence and the outliers in the physiological feature sequence does not exceed the set threshold.

4. The assisted diagnosis and treatment method based on a multimodal fusion large model according to claim 1, characterized in that, In step 3, the semantic features of the three non-textual modalities—aligned speech feature sequences, physiological feature sequences, and behavioral data—are mapped to a unified text semantic space. Specifically: The speech feature sequence is input into the speech branch of Q-Former, and attention matching is performed between the pre-trained query vector group and the speech features to output 100 768-dimensional speech semantic vectors. The physiological signal sequence is input into the physiological branch of Q-Former, and matched with the physiological signal sequence through a dedicated query vector group, outputting 100 physiological semantic vectors of 768 dimensions. Input the behavioral data into the behavioral branch of Q-Former, and output 100 768-dimensional behavioral semantic vectors after querying and matching the behavioral vectors.

5. The assisted diagnosis and treatment method based on a multimodal fusion large model according to claim 4, characterized in that, In step 3, the text semantic vector of the text data is extracted, specifically as follows: The BERT model is used to process text data, and 100 768-dimensional text semantic vectors are output, with a one-to-one correspondence between the text and non-text modal time frames.

6. The assisted diagnosis and treatment method based on a multimodal fusion large model according to claim 4, characterized in that, In step 3, verifying the semantic association between modalities specifically involves: Calculate the cosine similarity of the semantic vectors of the three modalities—speech feature sequence, physiological feature sequence, and behavioral data—within the same time frame. If the similarity within a certain time frame is less than a set threshold, re-execute the semantic feature mapping until the similarity threshold requirement is met.

7. The assisted diagnosis and treatment method based on a multimodal fusion large model according to claim 1, characterized in that, The standardized three-dimensional feature matrix is ​​a weighted standardized three-dimensional feature matrix. This weighted standardized three-dimensional feature matrix is ​​used to construct a modal reliability assessment model, dynamically adjusting the modal weight factors based on historical data. Specifically: Acquire multimodal historical data from different population groups and extract modal reliability features; An evaluation model was constructed, with modal reliability characteristics as input and the actual contribution of the modality to the diagnosis and treatment outcome as the output label. The evaluation model was trained to quantify the reliability of different populations and different modalities. Establish a weight factor mapping rule to map the reliability score output by the evaluation model to the modality weight factor according to the normalization ratio; The real-time reliability index of the modal data is calculated in real time, and the real-time reliability index is input into the evaluation model to obtain the real-time reliability score. The weighting factors of the modality are dynamically adjusted based on the real-time reliability score.

8. The assisted diagnosis and treatment method based on a multimodal fusion large model according to claim 1, characterized in that, In step 4, semantic vectors of the knowledge graph are matched from the knowledge graph through vector retrieval and keyword retrieval, specifically as follows: The search keywords are converted into 768-dimensional semantic vectors. The cosine similarity between the keywords and the semantic vectors of the knowledge graph is calculated in the vector database, and candidate knowledge with similarity greater than a set threshold is selected. Filter knowledge entries containing core terms by keyword search, and exclude irrelevant knowledge; Candidate knowledge is prioritized, sorted, and integrated in a structured manner to generate an association table, clarifying the correspondence between the semantic vector of the knowledge graph and the candidate knowledge.

9. A multimodal fusion large-scale model-based assisted diagnosis and treatment system, applicable to the multimodal fusion large-scale model-based assisted diagnosis and treatment method as described in any one of claims 1-8, characterized in that, include: The data acquisition layer receives voice, text, physiological signals, and behavioral data. The model computation layer includes a full-modal data fusion module, a knowledge-enhanced large model module, and a personalized solution generation module; The model computation layer is responsible for the core algorithm calculations and generates auxiliary diagnosis and treatment plans based on the data from the data acquisition layer. The application service layer outputs the auxiliary diagnosis and treatment plan generated by the model calculation layer to the user terminal. The security layer employs federated learning, differential privacy, and hierarchical access control to enhance data security.

10. The assisted diagnosis and treatment system based on a multimodal fusion large model according to claim 9, characterized in that, The model computation layer also includes a lightweight deployment module for compressing models in the full-modal data fusion module, the knowledge-enhanced large model module, and the personalized solution generation module.

Images