A diagnosis and improvement method and device combined with multi-modal interaction

Abstract

This invention relates to the technical field of multimodal interactive consultation efficiency improvement methods, specifically: a method and device for improving consultation efficiency by combining multimodal interaction, comprising the following steps: Step 1: Multimodal consultation initiation and initial data collection, building an interactive terminal integrating at least three modalities among voice, image, text, gesture / touch, and physiological signals, and collecting multimodal data of patient identity information, basic information, and initial symptoms; Step 2: Multimodal data preprocessing and noise reduction, cleaning, noise reduction, and standardization of the initial data of each modality collected in Step 1, storing and establishing a data index after verification; In this invention, by building a multimodal interactive terminal, it adapts to different groups of people and consultation scenarios, solves the limitations of single-modal interaction, and lowers the consultation participation threshold for various types of patients.

Description

Technical Field

[0001] This invention relates to the technical field of methods for improving the efficiency of medical consultations by combining multimodal interaction, and in particular to a method and apparatus for improving the efficiency of medical consultations by combining multimodal interaction. Background Technology

[0002] In the current medical consultation scenario, with the increasing demand for medical care, the traditional consultation model is no longer able to meet the consultation needs of the entire population and multiple scenarios.

[0003] On the one hand, patients of different ages and with different physical conditions have varying levels of interaction ability, and some groups are unable to smoothly convey symptom information through a single modality, leading to difficulties in initiating consultations and incomplete information transmission. On the other hand, the uneven distribution of medical resources and the prominent problems of patients blindly registering for appointments and traveling between departments hinder the continuous improvement of consultation quality. Summary of the Invention

[0004] In response to the technical problem that traditional consultation models are no longer suitable for the consultation needs of the entire population and multiple scenarios in the current medical consultation scenario as medical needs continue to grow, this invention provides a consultation efficiency improvement method and device that combines multimodal interaction.

[0005] The technical solution adopted in this invention is: a method for improving the efficiency of online consultation by combining multimodal interaction, specifically including the following steps: Step 1: Multimodal consultation initiation and initial data collection. Build an interactive terminal that integrates at least three of the following modalities: voice, image, text, gesture / touch, and physiological signals. Collect multimodal data on patient identity information, basic information, and initial symptoms. Step 2: Multimodal data preprocessing and noise reduction. The initial data of each modality collected in Step 1 are cleaned, denoised, and standardized. After verification, the data is stored and an index is established. Step 3: Multimodal data feature extraction and deep fusion. Extract key features from the preprocessed data of each modality, and use a cross-modal fusion algorithm to achieve deep fusion of features from each modality, generating a unified multimodal fusion feature vector; Step 4: Multimodal consultation intent recognition and intelligent triage. Based on the multimodal fusion feature vector generated in Step 3, the trained intent recognition model identifies the patient's consultation intent and confidence level, quantifies the urgency of the patient's symptoms, and completes intelligent triage. Step 5: Deepen multimodal interactive consultation with doctors, build a dedicated multimodal interactive terminal for doctors, realize multimodal collaborative consultation between doctors and patients, generate consultation assistance prompts and record the entire consultation process; Step 6: Multimodal assisted diagnosis and decision support, construct a multimodal medical knowledge base and assisted diagnosis model, input multimodal fusion features and consultation-related annotations, output assisted diagnosis-related information and verify the diagnosis results; Step 7: Multimodal output and confirmation of consultation results. Generate various types of multimodal consultation results, adapt to the patient's needs, guide the patient to complete the result confirmation, and complete the result storage and push. Step 8: Multimodal follow-up and data iterative optimization. Develop and execute a personalized multimodal follow-up plan, collect and preprocess follow-up multimodal data, and combine quantitative formulas to iteratively optimize the consultation model and process, forming a complete consultation closed loop. In one embodiment, the interactive terminal in step 1 supports three carriers: mobile phones, tablets, and smart consultation devices, adapting to three scenarios: home, community, and hospital outpatient clinics; the multimodal data collection of initial symptoms includes: voice modality collection of patients' oral symptoms and tone and speed characteristics, image modality collection of high-definition images and shooting information of symptom-related parts, physiological signal modality collection of patients' body temperature, heart rate, and blood pressure physiological parameters, and text / gesture modality collection of symptom details; the data tag format is "patient ID-modality type-collection time-integrity tag", and it is encrypted and stored using the AES-256 encryption algorithm.

[0006] In one embodiment, the preprocessing of each modality data in step 2 specifically includes: denoising speech data using an adaptive wavelet denoising algorithm and extracting MFCC feature vectors; denoising image data using a Gaussian filtering algorithm, followed by size standardization, grayscale conversion, and enhancement processing; segmenting text data using the jieba word segmentation algorithm, removing stop words, performing entity recognition, and converting it into structured text; smoothing gesture / touch data using a moving average algorithm, followed by coordinate standardization and deduplication; denoising physiological signal data using a sliding window filtering algorithm, removing outliers, performing standardization, and calculating statistical features; and verifying the data model, which verifies data integrity, accuracy, and standardization, automatically guiding the patient to re-collect data or automatically correcting it if the verification fails.

[0007] In one embodiment, step 3, the extraction of each modality feature specifically includes: extracting semantic and emotional features for speech features using an LSTM model; extracting visual features for image features using a ResNet-50 model combined with an SE-Net attention mechanism; extracting semantic features for text features using a simplified BERT model; extracting gesture / touch features using a CNN+fully connected layer structure; and extracting physiological signal features using an MLP model. The cross-modal attention fusion algorithm assigns weights to each modality feature through sample training and combines the attention mechanism to achieve synergistic complementarity of each modality feature, generating a 320-dimensional multimodal fusion feature vector. The fusion feature verification model verifies the feature completeness, relevance, and stability. If the verification fails, the weights are automatically adjusted and the feature is re-fused.

[0008] In one embodiment, step 4 is as follows: The intent recognition model is a multimodal consultation intent recognition model based on the Transformer model. The input is a multimodal fusion feature vector, and the output is a consultation intent label and confidence score. The intent labels are divided into three categories: routine consultation, diagnostic assistance, and emergency medical treatment. The model training uses the cross-entropy loss function; the quantization formula in step 4 is... E represents the urgency of the symptoms. These are the weights of voice emotion features, physiological signal features, and multimodal fusion features, respectively, and their sum is 1. These are the quantized values ​​of the corresponding features. An urgency level correction coefficient is applied; the intelligent triage model matches the corresponding department based on the patient's consultation intent, allocates the priority of admission based on the urgency level, and matches doctors based on the patient's location, the doctor's workload, and their specialty.

[0009] In one embodiment, step 5 involves a multimodal interactive terminal for doctors supporting both computers and tablets, with functions for rapid retrieval of multimodal data and initiation of multimodal inquiries. Multimodal collaborative consultations support doctors initiating inquiries via voice, text, and gestures, with patients responding in real-time using five modalities. The terminal supports real-time synchronization of the interaction process. A consultation assistance prompting model, based on multimodal fusion features and consultation intent, generates prompts for missing key symptoms, required supplementary multimodal data, and key questions related to suspected symptoms, pushing these prompts via pop-ups and voice reminders. The entire consultation process is recorded, including the doctor's questions, the patient's responses, and the doctor's operation records, stored encrypted and synchronized in real-time to the electronic medical record system, allowing doctors to manually highlight key points during the consultation.

[0010] In one embodiment, the personalized multimodal follow-up plan in step 8 is formulated based on the patient's diagnosis, treatment plan, and disease severity, including follow-up time, follow-up content, and follow-up method; the follow-up data collection and preprocessing adopts the preprocessing method of step 2, and is stored in association with the patient's initial consultation data and consultation results; the quantification formula in step 8 is... ,in These are the parameters of the medical history model after and before the iteration, respectively. The learning rate is k, and the number of follow-up data samples is k. Let i be the weight of the i-th group of follow-up data. These are the actual feature values ​​and model-predicted feature values ​​of the i-th group of follow-up data, respectively. The iterative optimization cycle is once a month. After optimization, the effect is verified by four indicators. If the indicators are not met, the parameters are readjusted for optimization.

[0011] In one embodiment, the multimodal consultation results in step 7 include a structured text diagnostic report, a speech report synthesized from natural speech, and a visual image report. The text report annotates professional terms, the speech report repeats key information, and the image report includes a medication schedule and symptom comparison chart. Personalized output is determined based on the patient's age, education level, and interaction habits, and supports the patient manually switching output modes. Patient confirmation is conducted using a multimodal approach. If the patient is unclear about the confirmation, the consultation results are automatically interpreted and confirmation is prompted again. Patients can initiate follow-up questions through a multimodal approach. The consultation results are stored in encrypted form and pushed to the patient's terminal and electronic medical record system, and patients can share them with their families in encrypted form. In step 6, the multimodal medical knowledge base adopts distributed storage, supports real-time updates and multimodal retrieval, the auxiliary diagnostic model training adopts the contrastive learning loss function, and the diagnostic result verification model judges the rationality of the diagnostic result through the matching degree and forces abnormal diagnostic results to be re-verified.

[0012] In one embodiment, a diagnostic consultation efficiency improvement device incorporating multimodal interaction includes: The multimodal initial data acquisition module is used to build an interactive terminal that integrates at least three modalities of voice, image, text, gesture / touch and physiological signals, to collect multimodal data of patient identity information, basic information and initial symptoms, and to temporarily store the collected data; The multimodal data preprocessing module is used to clean, reduce noise, and standardize the initial data of each modality collected by the multimodal initial data acquisition module, verify the preprocessed data, store the verified data, and establish a data index. The multimodal feature fusion module is used to extract key features of each modal preprocessed data output by the multimodal data preprocessing module, and to use a cross-modal fusion algorithm to achieve deep fusion of the features of each modality, generating and outputting a unified multimodal fusion feature vector. The consultation intent recognition and intelligent triage module is used to receive the multimodal fusion feature vector output by the multimodal feature fusion module, identify the patient's consultation intent and confidence level through the trained intent recognition model, quantify the urgency of the patient's symptoms through a quantification formula, and complete intelligent triage by combining consultation intent, urgency level and doctor information. The multimodal collaborative consultation module for doctors and patients is used to build a dedicated multimodal interactive terminal for doctors, enabling multimodal collaborative consultations between doctors and patients, generating consultation assistance prompts, automatically recording the entire consultation process and synchronizing it to the electronic medical record system; The multimodal auxiliary diagnosis module is used to build a multimodal medical knowledge base and auxiliary diagnosis model. It receives the multimodal fusion feature vector and consultation-related annotations output by the multimodal feature fusion module, outputs auxiliary diagnosis-related information, and verifies the diagnosis results given by the doctor. The consultation result output and confirmation module is used to generate various types of multimodal consultation results, personalize the output method according to the patient's needs, guide the patient to complete the consultation result confirmation in a multimodal way, and store and push the confirmed consultation results; The follow-up and iterative optimization module is used to develop and execute personalized multimodal follow-up plans based on patient diagnosis results, collect follow-up multimodal data and preprocess it, and achieve iterative optimization of the overall process of consultation-related models and devices through quantitative formulas to form a complete consultation closed loop. The beneficial effects of this invention are as follows: Compared with existing technologies, this invention, by building a multimodal interactive terminal, adapts to different groups of people and consultation scenarios, overcomes the limitations of single-modal interaction, lowers the consultation participation threshold for various patients, ensures that patients with different physical conditions and different expressive abilities can smoothly transmit symptom information, and guarantees the comprehensiveness of initial information collection. The preprocessing and deep fusion of multimodal data solves the current situation of fragmented information across different modalities, integrates various symptom-related data, provides high-quality support for subsequent consultation processes, and improves information utilization efficiency. Standardized consultation intent recognition and intelligent triage achieve accurate matching of patients with corresponding departments and doctors, avoiding patients blindly registering and traveling between departments, and shortening the consultation waiting period. Attached Figure Description

[0013] Figure 1 This is a flowchart of the present invention; Figure 2 This is a flowchart of the multimodal data preprocessing and noise reduction process in this invention; Figure 3 This is a flowchart of the multimodal data feature extraction and deep fusion process in this invention; Figure 4 This is a flowchart of the multimodal consultation intent recognition and intelligent triage process in this invention; Figure 5 This is a flowchart of the multimodal follow-up and data iterative optimization process in this invention. Detailed Implementation

[0014] In the description of this invention, it should be noted that the terms "front", "up", "down", "left", "right", "vertical", "horizontal", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0015] To address the problems existing in the background technology, this application proposes the following technical solution: a method and apparatus for improving the efficiency of online consultation by combining multimodal interaction, specifically including the following steps: Step 1: Initiating multimodal consultation and initial data collection; This step enables the initiation of a consultation, overcoming the limitations of existing single-modal interactions. It allows patients of different ages and physical conditions to quickly initiate consultations and efficiently collect initial symptom information, laying the foundation for subsequent consultation processes and ensuring that the collected data is comprehensive, accurate, and relevant to the patient's actual expressive abilities.

[0016] The specific implementation method is as follows: Multimodal Interaction Portal Setup: An interactive terminal integrating five modalities—voice, image, text, gesture / touch, and physiological signals—is built. The terminal supports three platforms: mobile phones, tablets, and smart diagnostic devices, adapting to three scenarios: home, community, and hospital outpatient clinics. Among these, the mobile phone / tablet terminal focuses on lightweight interaction (voice, text, image, touch), while the smart diagnostic device focuses on professional interaction (physiological signal acquisition, gesture recognition, and high-definition image capture). The terminal interface adopts a minimalist design, supports font enlargement and voice broadcasting, and is suitable for elderly patients. It also supports quick access for emergency cases (one-click triggering of voice and image acquisition), suitable for emergency patients.

[0017] Patient Identity and Basic Information Collection: Patients complete identity verification through multimodal methods (facial recognition, voice response, or ID card scanning and text confirmation). After successful verification, the terminal automatically pops up a basic information collection interface, supporting three collection methods: text input (suitable for younger people, allowing them to quickly fill in structured information such as age, gender, and past medical history), voice narration (suitable for the elderly, where the terminal converts narrated content into structured text through voice recognition while retaining the original voice recording), and gesture / touch selection (suitable for patients with physical limitations or those unable to express themselves clearly, allowing them to complete information filling by clicking on preset options on the interface, such as clicking on past medical history tags like "hypertension" or "diabetes"). After collection, the terminal automatically verifies the completeness of the information. If any items are missing, it provides dual reminders via voice and pop-up windows to ensure that no basic information is omitted.

[0018] Initial symptom multimodal collection: Based on the patient's basic information (such as age and past medical history), the terminal intelligently pushes targeted symptom collection guidance, guiding the patient to upload initial symptom information in a multimodal manner: voice modality (describing current symptoms, such as "headache for 3 days, accompanied by nausea and vomiting," the terminal collects voice data, with a sampling rate of 16kHz, WAV format, and records the speech rate and tone characteristics for subsequent assessment of symptom urgency); image modality (capturing symptom-related areas, such as rashes and swollen areas, the terminal supports autofocus and supplemental lighting, the acquired image resolution is ≥1080P, JPG format, and the shooting angle and shooting time are recorded to ensure clear and identifiable images); physiological signal modality (collecting physiological signals such as body temperature, heart rate, and blood pressure through an intelligent diagnostic instrument, with a sampling frequency of 1Hz and a collection duration of 1 minute to ensure the stability of physiological data); text / gesture modality (supplementing symptom details, such as typing "headache location is on the top of the head" or marking the pain location on a human body contour map using gestures).

[0019] Data Acquisition Temporary Storage and Tagging: The acquired multimodal initial data (speech, image, text, gesture coordinates, physiological signals) are classified and temporarily stored using an encrypted storage method (AES-256 encryption algorithm) to ensure patient privacy and security. At the same time, the terminal automatically adds tags to each type of data. The tag information includes: acquisition time, modality type, data integrity tag ("complete", "missing", "fuzzy"), and patient ID, which facilitates subsequent data preprocessing and traceability. The tag format is "patient ID-modality type-acquisition time-integrity tag", such as "P001-speech-202602101000-complete".

[0020] Step 2: Multimodal data preprocessing and noise reduction; The core purpose of this step is to clean, denoise, and standardize the initial multimodal data collected in step 1, remove invalid and interfering data, unify data format and dimensions, improve data quality, avoid invalid data affecting the accuracy and efficiency of subsequent consultation processes, and ensure that the preprocessed multimodal data can achieve deep fusion.

[0021] The specific implementation method is as follows: Preprocessing of each modal data separately: Preprocessing of speech data: The adaptive wavelet denoising algorithm is adopted to remove the environmental noise (such as home background noise and external noise) and device noise (such as microphone interference noise) in the speech data, and retain the effective speech signal; then endpoint detection is performed on the denoised speech data to remove the silent segments in the speech and extract the effective speech segments; finally, normalization processing is carried out to normalize the amplitude of the speech signal to the interval [-1,1], unify the sampling rate to 16kHz, and convert it into the Mel-frequency cepstral coefficients (MFCC) feature vector for subsequent speech feature extraction. The dimension of the MFCC feature vector is set to 13 dimensions to ensure the efficiency and accuracy of feature extraction.

[0022] Preprocessing of image data: The Gaussian filtering algorithm (with a filter kernel size of 3×3) is adopted to remove the Gaussian noise and salt-and-pepper noise in the image and improve the image clarity; then size normalization processing is carried out to uniformly scale all images to 640×480 pixels, and the bilinear interpolation algorithm is used to ensure that the scaled image is distortion-free; then grayscale processing (for color images) is carried out to convert the color image into a grayscale image and reduce the data volume; finally, image enhancement processing (using the histogram equalization algorithm) is carried out to improve the contrast of the image and highlight the lesion features (such as rash color and swelling contour) for subsequent image feature extraction and doctor viewing.

[0023] Preprocessing of text data: Natural language processing (NLP) technology is adopted to perform word segmentation on the text data (using the jieba word segmentation algorithm) and remove the stop words (such as meaningless words like "de", "le", "shi", etc.); then entity recognition is carried out to extract the key information in the text (such as symptom names, symptom duration, pain degree, etc.) and convert the unstructured text into structured text; finally, normalization processing is carried out to organize the key information in the format of "symptom name - duration - pain degree" and unify the expression of the pain degree (such as "mild", "moderate", "severe", corresponding to the quantization values 1, 2, 3) to ensure the standardization of the text data.

[0024] Preprocessing of gesture / touch data: The collected gesture coordinate data is smoothed (using the moving average algorithm with a window size of 5) to remove the jitter noise in the gesture operation and ensure the stability of the coordinate data; then coordinate normalization processing is carried out to uniformly map the gesture coordinates to the interval [0,1] and unify the coordinate system (with the upper left corner of the human contour map as the origin and the lower right corner as (1,1)); duplicate removal processing is carried out on the touch data to remove the duplicate data generated by accidental touches and retain the effective touch operation records (such as clicking on symptom labels and marking pain locations), and convert the touch operations into structured label data (such as "touch annotation - headache - top of the head").

[0025] Physiological signal data preprocessing: A sliding window filtering algorithm (window size of 10s) is used to remove outliers (such as abnormal data such as body temperature exceeding 42℃ and heart rate below 40 beats / minute) and fluctuation noise from the physiological signals, retaining the effective physiological signals; then, standardization processing is performed to normalize physiological signals such as body temperature, heart rate, and blood pressure according to medical standards (e.g., body temperature is normalized to [35,42]℃, and heart rate is normalized to [40,180] beats / minute), unifying the data units and dimensions; finally, the statistical characteristics of the physiological signals (such as mean and standard deviation) are calculated for subsequent physiological state judgment, such as the standard deviation of heart rate is used to determine whether the patient has abnormal heart rate fluctuations.

[0026] Preprocessed data verification and correction: A data verification model is built to verify the preprocessed modal data. Verification includes: data integrity (whether there are missing items), data accuracy (e.g., whether physiological signals are within the normal range, whether images are clear and identifiable, whether text is ambiguous), and data standardization (whether the format is consistent and whether the dimensions are consistent). If the verification fails, the terminal automatically returns to the acquisition interface of the corresponding modality, guiding the patient to re-acquire the data (e.g., if the image is blurry, prompt "Please retake the picture to ensure the image is clear"), or automatically corrects the data (e.g., if the text is ambiguous, ask the patient for confirmation via voice). After successful verification, the preprocessed data of each modality is stored in the database, and a data index is established (index keywords are patient ID, acquisition time, and modality type) for easy retrieval later.

[0027] Step 3: Multimodal data feature extraction and deep fusion; This step involves extracting key features from the preprocessed data of each modality, and using a cross-modal fusion algorithm to achieve deep fusion of the features of each modality, generating a unified multimodal fusion feature vector. This solves the problem of fragmented information between different modalities in existing technologies, enabling doctors to quickly obtain integrated patient symptom information, improving consultation efficiency, and providing support for subsequent intent recognition and assisted diagnosis.

[0028] The specific implementation method is as follows: Modal feature extraction: Speech feature extraction: Based on the speech MFCC feature vectors preprocessed in step 2, a Long Short-Term Memory (LSTM) network model is used to extract semantic and emotional features of the speech. Semantic features are used to identify the symptoms described by the patient (such as "headache" and "nausea"), while emotional features are used to judge the urgency of the patient's symptoms (such as rapid speech and anxious tone corresponding to high urgency, and slow speech and calm tone corresponding to low urgency). The extracted speech feature vectors are set to 64 dimensions, including 32 dimensions of semantic features and 32 dimensions of emotional features, to ensure the comprehensiveness of the features.

[0029] Image feature extraction: Based on the image data preprocessed in step 2, a convolutional neural network (CNN) model (using the ResNet-50 model to simplify the network structure and improve extraction efficiency) is used to extract visual features of the image, including the shape, color, and texture of lesions (such as the shape and color intensity of rashes, and the outline of swollen areas). During the extraction process, an attention mechanism (SE-Net attention mechanism) is used to highlight the features of the lesion area and suppress invalid features of the background area. The extracted image feature vector is set to 128 dimensions to ensure accurate representation of lesion information in the image.

[0030] Text feature extraction: Based on the structured text data preprocessed in step 2, the BERT model (simplified version, reducing the number of network layers to improve efficiency) is used to extract the semantic features of the text, including the feature representation of key information such as symptom name, duration, and pain level; the extracted text feature vector is set to 64 dimensions to ensure that it can accurately map the key information in the text.

[0031] Gesture / touch feature extraction: Based on the gesture coordinate data and touch label data preprocessed in step 2, a convolutional neural network (CNN) and a fully connected layer structure are used to extract gesture / touch feature vectors. Gesture features mainly represent information such as the pain location and symptom range marked by the patient, while touch features mainly represent the symptom label information selected by the patient. The extracted gesture / touch feature vectors are set to 32 dimensions to ensure the simplicity and effectiveness of the features.

[0032] Physiological signal feature extraction: Based on the physiological signal data and its statistical characteristics after preprocessing in step 2, a multilayer perceptron (MLP) model is used to extract the feature vector of the physiological signal to represent the patient's physiological state (such as whether body temperature and heart rate are normal, and whether there are abnormal fluctuations). The extracted physiological signal feature vector is set to 32 dimensions to ensure that it can accurately reflect the patient's physiological condition.

[0033] Multimodal feature deep fusion: A cross-modal attention fusion algorithm (CM-Attention) is adopted to achieve deep fusion of five modal features, rather than the simple splicing of existing technologies. The core logic of this algorithm is as follows: First, the weights of each modal feature vector are calculated (weights are assigned according to the reliability and relevance of the modal data, such as higher weights for image modality when representing skin symptoms and higher weights for physiological signal modality when representing internal medicine symptoms). The weights are obtained through sample training (the training samples are 10,000+ sets of multimodal consultation data, and the gradient descent algorithm is used to optimize the weight values). Then, through the attention mechanism, the feature vectors of each modality interact and complement each other, highlighting the features of key modalities and suppressing the interference of invalid modalities. Finally, the weighted feature vectors of each modality are fused to generate a unified multimodal fused feature vector. The dimension of the fused feature vector is set to 320 dimensions (64+128+64+32+32) to ensure that the fused features can comprehensively and accurately represent the patient's symptoms and physiological state.

[0034] Fusion Feature Validation and Optimization: A fusion feature validation model is built to validate the generated multimodal fusion feature vectors. Validation includes: feature completeness (whether it contains the key features of the five modalities), feature relevance (whether the correlation between the fusion features and the patient's symptoms meets the standard), and feature stability (whether the fusion features of the same patient collected at different times are consistent). If the validation fails, the weights of each modality feature are automatically adjusted, and fusion is performed again. After successful validation, the fusion feature vector is stored in the database and associated with the patient's basic information and original multimodal data for easy retrieval in subsequent steps.

[0035] Step 4: Multimodal consultation intent recognition and intelligent triage; The core objective of this step is to identify the patient's consultation intent (such as "consulting about cold symptoms", "diagnosing a rash", or "urgent medical assistance") based on the multimodal fusion feature vector generated in step 3. Then, based on the consultation intent, the urgency of the patient's symptoms, and the doctor's specialty, intelligent triage is achieved, accurately matching the patient to the corresponding department and doctor. This avoids patients blindly registering and traveling between departments, shortens consultation waiting time, and reduces the workload of doctors. Compared with existing technologies such as manual triage and single-text triage, this approach is more effective.

[0036] The specific implementation method is as follows: Construction and Training of Consultation Intent Recognition Model: A multimodal consultation intent recognition model based on the Transformer model is constructed. The input of the model is the multimodal fusion feature vector generated in step 3, and the output is the patient's consultation intent label and intent confidence. The intent labels are divided into three categories: routine consultation (such as consultation on common disease symptoms and medication advice), diagnostic assistance (such as needing a doctor to diagnose specific diseases), and emergency medical treatment (such as assistance for acute and critical illnesses). Each category is further subdivided into specific labels (such as "cold consultation" and "hypertension consultation" under routine consultation). The model is trained using 10,000+ sets of labeled multimodal fusion feature samples (each sample corresponds to a clear consultation intent label), using the cross-entropy loss function for training. The number of training iterations is set to 100, and the learning rate is set to 0.001 to ensure that the recognition accuracy of the model is ≥95%.

[0037] Intent recognition and confidence assessment: The multimodal fusion feature vector generated in step 3 is input into the trained intent recognition model. The model outputs the patient's intent label and intent confidence (confidence range is [0,1]). If the confidence is ≥0.9, the recognition is considered accurate, and the intent is directly determined. If 0.7≤confidence<0.9, the recognition is considered questionable, and the terminal asks the patient for confirmation through multimodal interaction (voice and pop-up window) (e.g., "Is your intent to consult about cold symptoms? Please answer 'yes' or 'no' by voice, or click the corresponding option"). If the confidence is <0.7, the recognition fails, and the terminal automatically switches to manual assistance recognition mode. Back-end medical staff assist in determining the intent by reviewing the patient's multimodal data, ensuring that the intent recognition is accurate.

[0038] Symptom urgency quantification: Based on the voice emotion features collected in step 1, the physiological signal features after preprocessing in step 2, and the fusion features in step 3, the urgency of the patient's symptoms is quantified (the urgency value ranges from [1,5], where 1 is the mildest and 5 is the most urgent); the calculation of urgency is combined with the core formula in step 4 to clarify the operation process; at the same time, urgency thresholds are set in accordance with medical standards: urgency ≥ 4 is judged as an emergency, 2 ≤ urgency < 4 is judged as a normal situation, and urgency = 1 is judged as a mild situation.

[0039] Intelligent triage is implemented by building an intelligent triage model. Input parameters include the patient's consultation intent tag, symptom urgency, basic information (such as past medical history), and location. The core logic of the triage model is as follows: First, it matches the corresponding department based on the consultation intent tag (e.g., "rash diagnosis" matches dermatology, "headache diagnosis" matches neurology). Then, it allocates priority based on the urgency of the symptoms (emergency cases are triaged first, with a waiting time ≤ 5 minutes; routine cases are triaged normally, with a waiting time ≤ 30 minutes; minor cases can be scheduled for triage, with a waiting time ≤ 60 minutes). Finally, based on the patient's location, the doctor's workload (current number of patients seen, remaining appointment slots), and the doctor's specialty (e.g., dermatologists specializing in "allergic skin diseases" or "infectious skin diseases"), the optimal doctor is matched. After triage, the terminal pushes the triage results to the patient via multimodal methods (voice, text, and pop-ups), including the attending department, doctor's name, consultation time, and consultation method (online consultation or offline outpatient visit). At the same time, it pushes the patient's multimodal fusion data and basic information to the corresponding doctor, making it easier for the doctor to understand the patient's condition in advance and shorten the consultation time.

[0040] Step 4 Formula: ; The formula's function is to quantify the urgency of a patient's symptoms, providing core quantitative basis for intelligent triage in step 4, clarifying the operational process of "multimodal features, urgency, and triage priority," solving the problems of ambiguous urgency judgment and lack of quantitative standards in existing technologies, ensuring the accuracy and rationality of triage, and simultaneously achieving a closed loop in the first 4 steps (collection-preprocessing-fusion-triage), improving the traceability and optimizability of consultation efficiency.

[0041] Letter meaning explanation: E: The severity of the patient's symptoms, with a value range of [1,5], where 1 is the mildest and 5 is the most urgent. It is calculated by a formula and used to prioritize patients. The speech emotion feature weights, with values ​​ranging from [0.2, 0.4], are obtained through sample training. The more urgent and anxious the speech emotion features, the better. The larger the value, the higher the urgency level. The physiological signal feature weights, with a value range of [0.3, 0.5], are obtained through sample training. The more abnormal the physiological signal (such as excessively high heart rate or body temperature), the more significant the difference. The larger the value, the higher the urgency level. The multimodal fusion feature weights, with values ​​ranging from [0.2, 0.4], are obtained through sample training. The more prominent the lesion and symptom features in the fusion features, the better. The larger the value, the higher the urgency; and it satisfies To ensure reasonable weight allocation and avoid interference from a single modality feature in the judgment of urgency; The quantitative value of speech emotion features, with a range of [1,5], is obtained by converting the speech emotion features extracted in step 3. High values ​​(4-5) correspond to rapid speech and anxious tone, while low values ​​(1-2) correspond to slow speech and steady tone. The quantification value of physiological signal features, with a range of [1,5], is obtained by converting the physiological signal features extracted in step 3. The normal physiological signal corresponds to the low value (1-2), the slightly abnormal physiological signal corresponds to the medium value (3), and the severely abnormal physiological signal corresponds to the high value (4-5). The multimodal fusion feature quantization value, with a value range of [1,5], is obtained by converting the multimodal fusion feature vector generated in step 3. The more obvious the symptoms and the clearer the lesions in the fusion features, the higher the value. The urgency correction coefficient, with a value range of [0, 0.5], is used to correct errors in extreme cases (such as when a patient's physiological signals are normal, but their voice and emotions are extremely anxious and the fusion features show that the symptoms are urgent). It is set by the back-end medical staff based on clinical experience and can be dynamically adjusted according to the actual application scenario.

[0042] Step 5: Deepening multimodal interactive consultation with doctors; The core objective of this step is to enable multimodal collaborative consultation between doctors and patients, addressing the traditional model of "doctor asking questions and patient answering questions" in existing technologies. Doctors can quickly retrieve patients' symptom data and initiate inquiries through multimodal methods, while patients can quickly respond and supplement symptom information through multimodal methods. At the same time, the system provides multimodal auxiliary prompts to help doctors accurately capture key information, avoid missing or incorrect questions, improve consultation accuracy, and shorten consultation time.

[0043] The specific implementation method is as follows: Multimodal Interactive Terminal Setup for Doctors: A dedicated multimodal interactive terminal is built for doctors, integrating five interactive modes: voice, image, text, gesture / touch, and physiological signals. The terminal supports both computers and tablets. It features rapid multimodal data retrieval; doctors can quickly retrieve basic patient information, initial multimodal data (voice, image, physiological signals, etc.), preprocessed data, and fused feature vectors by entering the patient ID or clicking on patient information in the triage list, without repeatedly switching interfaces. Simultaneously, the terminal supports multimodal inquiry initiation, allowing doctors to initiate inquiries with patients via voice (e.g., "Please describe your headache again"), text (e.g., sending a question via a pop-up window), and gestures (e.g., clicking on preset question tags on the interface), adapting to different doctor operating habits.

[0044] Multimodal consultation interaction is implemented as follows: After the doctor initiates a multimodal inquiry through the terminal, the patient receives the inquiry (voice broadcast and pop-up display) through their own interactive terminal and responds in a multimodal manner: voice response (the patient answers the question orally, the terminal automatically recognizes and converts it into text, and pushes it to the doctor's terminal simultaneously), image response (the patient takes supplementary symptom images, such as taking close-up images of rashes when the doctor requests them, and the patient takes and uploads them in real time), text response (the patient answers the question by typing text), gesture / touch response (the patient adds supplementary symptom details by marking gestures, such as marking the area of ​​pain on a human body contour map when the doctor asks about it), and physiological signal response (physiological signals are recollected through the intelligent consultation device, such as taking and uploading the patient's temperature when the doctor requests it to be checked). During the interaction, the terminal supports real-time synchronization, and the doctor can view the patient's response data in real time without waiting for the patient to complete all responses, thus improving the efficiency of the interaction.

[0045] Consultation assistance prompt generation: Based on the multimodal fusion features from step 3 and the consultation intent from step 4, a consultation assistance prompt model is built to provide doctors with real-time prompts, helping them accurately capture key information and avoid missing or incorrect questions. The prompts include: key symptoms that were not asked (e.g., if the patient has a history of hypertension, prompt the doctor to ask "how well has your blood pressure been controlled recently"), multimodal data that needs to be supplemented (e.g., if the patient describes a rash but has not uploaded an image, prompt the doctor to ask the patient to take an image of the rash), and key questions related to suspected symptoms (e.g., if the patient has headaches or nausea, prompt the doctor to ask "whether there are accompanying symptoms such as dizziness or blurred vision"). The prompts are delivered in a multimodal manner (terminal pop-ups and voice reminders), allowing doctors to flexibly adjust the focus of the consultation based on the prompts. The prompts can also be manually turned off to avoid interfering with the consultation process.

[0046] Consultation process recording and synchronization: The terminal automatically records the entire multimodal consultation process between doctors and patients, including the doctor's questions (voice, text, gestures), the patient's responses (data from each modality), and the doctor's operation records (such as retrieving data and initiating prompts). The records are stored in an encrypted manner and are linked to the patient's consultation data, facilitating subsequent diagnostic review and tracing of medical disputes. At the same time, the recorded content is synchronized to the hospital's electronic medical record system in real time, eliminating the need for manual data entry by doctors, reducing their paperwork burden, and improving consultation efficiency. In addition, the terminal supports doctors in manually marking key points of the consultation (such as marking "patient has had a headache for 3 days, accompanied by nausea and vomiting" with gestures), and the marked content is automatically synchronized to the subsequent diagnostic report.

[0047] Step 6: Multimodal assisted diagnosis and decision support; The core objective of this step is to provide doctors with multimodal auxiliary diagnostic suggestions based on multimodal data (initial data, interaction data) and multimodal fusion features from the entire consultation process, combined with a medical knowledge base. This will help doctors quickly clarify the diagnostic direction, improve diagnostic accuracy, and reduce the diagnostic burden on doctors.

[0048] The specific implementation method is as follows: Medical Knowledge Base Construction: A multimodal medical knowledge base is constructed, integrating massive amounts of medical data, including: disease data (symptoms, diagnostic criteria, and treatment plans for common and complex diseases, with multimodal feature descriptions for each disease, such as image features, voice symptom descriptions, and physiological signal features for rashes), drug data (indications, dosage, and contraindications of drugs, with corresponding multimodal medication tips), and clinical case data (clinical cases from past multimodal consultations, including patient multimodal data, diagnostic results, and treatment effects). The knowledge base adopts a distributed storage approach, supporting real-time updates (disease data, drug data, and clinical case data are updated regularly by professional medical staff), and also supports multimodal retrieval (e.g., a doctor entering "rashes, redness, and itching" can retrieve multimodal information and clinical cases related to the corresponding diseases).

[0049] Multimodal Assisted Diagnostic Model Construction and Training: A deep learning-based multimodal assisted diagnostic model was constructed. The model's input consisted of multimodal fusion feature vectors from the entire consultation process (the initial fusion features generated in step 3 and the fusion features of the multimodal data supplemented during the consultation interaction in step 5), and the doctor's key annotations during the consultation. The model's output consisted of a list of suspected symptoms (sorted by similarity), diagnostic criteria for each suspected symptom (corresponding to multimodal data), and auxiliary diagnostic suggestions (such as items requiring further examination and preliminary treatment plans). The model was trained using 5000+ sets of clinical multimodal consultation cases (each case containing complete multimodal data and diagnostic results). A contrastive learning loss function was used for training, with 150 training iterations and a learning rate of 0.001, ensuring that the accuracy of the top 3 correct symptoms in the suspected symptom list output by the model was ≥92%.

[0050] Multimodal assisted diagnosis is implemented by inputting multimodal fusion feature vectors from the entire consultation process and the doctor's key points of consultation into a trained assisted diagnostic model. The model outputs a list of suspected symptoms, diagnostic criteria, and assisted diagnostic suggestions. Doctors can review the model's output based on their clinical experience and simultaneously access relevant data (such as multimodal features of suspected symptoms and clinical cases) from a medical knowledge base via a multimodal terminal to compare with the patient's multimodal data and further clarify the diagnostic direction. If the similarity of the suspected symptoms output by the model is ≥0.9, the doctor can quickly confirm the diagnosis by referring to the assisted diagnostic suggestions. If the similarity is 0.7≤s<0.9, the doctor can supplement the patient with key information through multimodal means based on the diagnostic criteria to further verify the diagnostic direction. If the similarity is <0.7, the doctor can initiate a multidisciplinary consultation request through the multimodal terminal to retrieve the opinions of doctors from relevant departments to ensure diagnostic accuracy.

[0051] Preliminary Verification of Diagnostic Results: A diagnostic result verification model is built. The inputs are the doctor's preliminary diagnostic results, the patient's multimodal fusion features, and disease diagnostic criteria from the medical knowledge base. The core logic of the verification model is to compare the matching degree between the disease features corresponding to the diagnostic results and the patient's multimodal fusion features. If the matching degree is ≥0.85, the diagnostic result is considered reasonable, and the doctor is allowed to confirm the diagnosis. If 0.7≤matching degree<0.85, the diagnostic result is considered questionable, and the doctor is prompted to re-verify (e.g., "The matching degree between the diagnostic result and the patient's multimodal features is low; please check if any key symptoms have been missed"). If the matching degree is <0.7, the diagnostic result is considered abnormal, and the doctor is forced to re-diagnose or initiate a consultation to avoid misdiagnosis or missed diagnosis.

[0052] Step 7: Multimodal output and confirmation of consultation results; The core purpose of this step is to deliver the doctor's diagnosis, treatment plan, and other consultation results to the patient in a multimodal manner, adapting to the information receiving habits of different patients, ensuring that patients can clearly understand the consultation results, and allowing patients to confirm the consultation results through multimodal interaction, avoiding information transmission errors, improving patient acceptance and compliance, compared with the single text output of existing technologies.

[0053] The specific implementation method is as follows: Multimodal generation of consultation results: After the doctor confirms the diagnosis, the terminal automatically generates multimodal consultation results, including three types: text type (a structured diagnostic report containing diagnosis results, symptom analysis, treatment plan, medication advice, precautions, etc., with concise and easy-to-understand language, avoiding excessive technical terms, and annotating technical terms when necessary); voice type (converting the text diagnostic report into voice using natural speech synthesis technology, with a moderate speaking speed and gentle tone, suitable for elderly patients or patients who cannot read text, with the voice duration controlled within 5 minutes, and key information (such as medication dosage and follow-up time) can be repeated); and image type (converting the diagnosis results and treatment plan into visual images, such as medication schedules and precaution diagrams, suitable for patients with strong visual perception, and can also compare the patient's symptom images with normal images, allowing the patient to clearly understand their condition).

[0054] Personalized Multimodal Output Adaptation: The terminal automatically adapts to the optimal multimodal output method based on the patient's basic information (age, education level, interaction habits): For elderly patients, voice and image output are prioritized, while text font size is enlarged; for younger patients, text and image output are prioritized, supporting quick browsing; for patients with physical disabilities or who cannot read or hear, visual output with gesture / touch interaction is prioritized (such as viewing diagnostic results details by clicking on the interface); at the same time, patients can manually switch output methods (such as switching from voice output to text output) to meet their own needs; during the output process, the terminal supports functions such as pause, playback (voice, image), and zoom (text, image) to facilitate careful viewing and listening by patients.

[0055] Patient Multimodal Confirmation: After the consultation results are output, the terminal guides the patient to confirm the results through multimodal methods, including: voice confirmation (the patient answers "clear" or "unclear" in voice, and the terminal automatically recognizes and records it); text confirmation (the patient enters "clear" or clicks the "confirm" button); gesture / touch confirmation (the patient clicks the "confirm" label or checks the confirmation box using gestures). If the patient confirms "clear," the terminal automatically records the confirmation result, completing the consultation result confirmation process. If the patient confirms "unclear," the terminal automatically jumps to the consultation result interpretation interface, interpreting key information in the consultation results (such as medication recommendations and precautions) for the patient through multimodal methods (voice interpretation, image illustration, and text annotation). After interpretation, the terminal guides the patient to confirm again until the patient confirms "clear." If the patient has questions about the consultation results, the terminal supports the patient to ask the doctor follow-up questions through multimodal methods (voice, text, and images), and the doctor responds in real time until the patient has no questions and confirms.

[0056] Consultation result storage and push: After the patient confirms the consultation results, the terminal automatically encrypts and stores the multimodal consultation results (text, voice, and images), and pushes them to the patient's interactive terminal (such as a mobile APP or tablet) and the hospital's electronic medical record system for easy access and follow-up visits. The push method uses multimodal reminders (voice, pop-up windows, and SMS) to ensure that the patient can receive the consultation results in a timely manner. In addition, the terminal supports patients to share consultation results with family members (through encrypted links to ensure privacy and security), making it easier for family members to understand the patient's condition and assist the patient in implementing the treatment plan.

[0057] Step 8: Multimodal follow-up and iterative data optimization; The core objective of this step is to achieve multimodal follow-up based on the consultation results, to understand the patient's recovery status in a timely manner, and to collect multimodal data during the follow-up process for iterative optimization of the entire consultation efficiency improvement method (including data preprocessing algorithms, feature fusion algorithms, intent recognition models, auxiliary diagnostic models, etc.), forming a closed loop of "consultation-diagnosis-follow-up-optimization" to continuously improve consultation efficiency and diagnostic accuracy.

[0058] The specific implementation method is as follows: Personalized multimodal follow-up plan development: Based on the patient's diagnosis, treatment plan, and disease severity, a follow-up plan development model is built to create a personalized multimodal follow-up plan for each patient. The follow-up plan includes: follow-up time (e.g., daily follow-up for acute patients, weekly follow-up for routine patients, and monthly follow-up for mild patients), follow-up content (e.g., inquiry about recovery status, symptom re-examination, collection of physiological signals, and confirmation of medication), and follow-up method (selecting an appropriate multimodal follow-up method based on the patient's interaction habits and condition, such as voice follow-up for elderly patients, text and image follow-up for younger patients, and physiological signal and image follow-up for postoperative patients). After the follow-up plan is developed, it is pushed to the patient and doctor's terminals via multimodal methods, reminding both parties to complete the follow-up on time.

[0059] Multimodal follow-up execution: According to the follow-up plan, the terminal automatically initiates follow-up reminders through multimodal methods (voice, pop-up windows, and SMS). After receiving the reminder, the patient completes the follow-up content through multimodal interaction: voice follow-up (verbal description of recovery status, such as "headache symptoms have been relieved, no nausea or vomiting"), image follow-up (photographing symptom sites after recovery, such as the fading of rashes), physiological signal follow-up (collecting physiological signals such as body temperature and heart rate to confirm the recovery of physiological status), and text / gesture follow-up (supplementing recovery details, such as typing "no adverse reactions after medication" or marking the degree of pain relief through gestures). Doctors can view the patient's follow-up multimodal data in real time through the terminal to assess the patient's recovery. If the patient's recovery is good, the follow-up frequency can be adjusted according to the plan. If the patient's recovery is poor or abnormalities occur, the treatment plan will be adjusted in a timely manner, and the patient will be notified through multimodal methods (such as a voice reminder "Please come back for a follow-up visit in time to adjust the medication plan").

[0060] Follow-up data collection and preprocessing: The terminal automatically collects multimodal follow-up data during the follow-up process. Following the preprocessing method in step 2, the follow-up data is cleaned, denoised, and standardized to remove invalid data and unify the data format and dimensions. Then, the preprocessed follow-up data is associated with the patient's initial consultation data and consultation results and stored in the database to establish a complete patient consultation-follow-up data archive. At the same time, the follow-up data is labeled with information including: follow-up time, follow-up modality, and rehabilitation assessment results ("good", "average", "poor") to facilitate subsequent data iteration and optimization.

[0061] Data Iterative Optimization Implementation: Based on a large amount of collected follow-up multimodal data, a data iterative optimization model is built to iteratively optimize the entire consultation efficiency improvement method. The optimization includes: multimodal data preprocessing algorithm (adjusting noise reduction parameters and standardization methods according to the characteristics of follow-up data to improve data preprocessing quality), multimodal feature fusion algorithm (adjusting the weights of each modality feature to improve the accuracy of fused features), consultation intent recognition model (supplementing new intent samples in follow-up data to improve model recognition accuracy), auxiliary diagnostic model (supplementing clinical cases in follow-up data to optimize the model's diagnostic suggestion output), and intelligent triage model (adjusting triage logic according to the triage effect in follow-up data to improve triage accuracy). The iterative optimization process combines the core formula in step 8, clarifies the operation process, and ensures that the optimized method can continuously improve consultation efficiency and diagnostic accuracy. The iterative optimization cycle is set to once a month. After each optimization, a small-scale test (selecting 100 consultation samples) is conducted to verify the optimization effect and ensure that the optimized method is free of anomalies.

[0062] Step 8 Formula: ; The formula's purpose is to quantify the iterative optimization process of the consultation model (including intent recognition models, auxiliary diagnostic models, etc.), clarify the operational procedures of "follow-up multimodal data, deviation calculation, and model parameter update," and achieve a closed-loop process for the last four steps (consultation deepening - auxiliary diagnosis - result output - follow-up optimization). This solves the problem of existing technologies lacking quantitative optimization standards and models being unable to continuously improve, ensuring that the entire consultation efficiency improvement method can achieve continuous optimization through data feedback, thereby improving consultation quality and efficiency.

[0063] Letter meaning explanation: The parameters of the consultation model after the (n+1)th iteration (including feature weights, model network parameters, etc.) are the core output after iterative optimization, which are used to update the model of the entire consultation system and improve the accuracy and efficiency of subsequent consultations. The parameters of the diagnostic model after the nth iteration are the basis for iterative optimization, and the initial values ​​are the initial parameters after the model training is completed. The iterative learning rate, with a value range of [0.001, 0.01], is used to control the update magnitude of model parameters. This avoids excessively large update magnitudes that could lead to model oscillations and unstable convergence, or excessively small update magnitudes that could result in low iteration efficiency. The learning rate can be dynamically adjusted based on the effect of iterative optimization. For example, when the model error continues to decrease during iteration, the current learning rate is maintained; when the model error tends to stabilize or increase, the learning rate is automatically reduced (e.g., reduced to 0.5 times the current value) to ensure that the model can converge to the optimal parameters quickly and stably.

[0064] The summation operator, where k is the number of follow-up data samples used in a single iteration, with a value range of [500, 1000] groups; the summation operation covers the deviation calculation results of k groups of follow-up data, ensuring that the iterative optimization can be based on a sufficient sample size, avoiding interference from outliers of a single follow-up data sample on the update of model parameters, and improving the stability and reliability of model optimization.

[0065] The weight of the i-th follow-up data group ranges from [0.001, 0.002] and satisfies the following condition: The weights are allocated based on the reliability and relevance of the follow-up data. For example, follow-up data with a rehabilitation assessment result of "poor" (corresponding to significant changes in the patient's condition and high data reference value) have a higher weight, while follow-up data with a rehabilitation assessment result of "good" and no significant fluctuations have a lower weight, ensuring that high-value follow-up data contributes more to model optimization.

[0066] The actual feature value obtained after preprocessing and feature extraction of the i-th group of follow-up multimodal data is the core reference for model iterative optimization. This value fully reflects the actual symptoms and physiological state of the patient during follow-up and is consistent with the format of the fusion feature vector generated in step 3 (320 dimensions), ensuring that it can be matched and calculated with the model parameters.

[0067] The model-predicted feature value corresponding to the i-th group of follow-up data is determined by the consultation model in the current iteration stage. This value is predicted based on the patient's initial medical history and previous follow-up data; this value is related to... The dimensions and format are completely consistent, and are used to match the actual feature values. Calculate the deviation to clarify the gap between the model prediction and the actual situation.

[0068] The deviation between the actual feature value and the model-predicted feature value of the i-th group of follow-up data reflects the current model's prediction error for this group of follow-up data. When the deviation is positive, it means that the model-predicted feature value is lower than the actual value (e.g., the model predicts the patient's heart rate to be normal, but the actual follow-up heart rate is higher). When the deviation is negative, it means that the model-predicted feature value is higher than the actual value. The larger the absolute value of the deviation, the greater the gap between the model prediction and the actual situation, and the model parameters need to be adjusted accordingly.

[0069] Step 8's formula calculates the deviation between the actual follow-up data and the model's predicted values. Combined with the learning rate and sample weights, it progressively updates the model parameters, ensuring that each iteration optimizes model performance based on real follow-up data. This allows the model to continuously adapt to changes in the patient's condition and the actual needs of the consultation scenario, forming a complete closed loop of "consultation-diagnosis-follow-up-optimization." This is one of the biggest differences between this method and existing technologies that employ "loopless iteration." After iterative optimization, the new model parameters ( This will be applied to subsequent consultation processes to continuously improve consultation efficiency and diagnostic accuracy.

[0070] Iterative optimization effect verification: After each iteration of optimization, 100 sets of follow-up multimodal data that did not participate in the iterative training were used as test samples to verify the optimized model. The performance of the algorithm was evaluated using four metrics: multimodal feature fusion accuracy, patient intent recognition accuracy, assisted diagnosis accuracy, and intelligent triage accuracy. A fusion accuracy ≥96%, intent recognition accuracy ≥96%, assisted diagnosis accuracy ≥93%, and triage accuracy ≥98% were considered valid for effective iterative optimization. If these metrics were not met, the follow-up data sample size would be increased (k value increased to 1200 groups), and the learning rate would be adjusted. and sample weights Then, iterative optimization is carried out again until the verification standard is met.

[0071] A diagnostic consultation efficiency improvement device combining multimodal interaction includes: The multimodal initial data acquisition module is used to build an interactive terminal that integrates at least three modalities of voice, image, text, gesture / touch and physiological signals, to collect multimodal data of patient identity information, basic information and initial symptoms, and to temporarily store the collected data; The multimodal data preprocessing module is used to clean, reduce noise, and standardize the initial data of each modality collected by the multimodal initial data acquisition module, verify the preprocessed data, store the verified data, and establish a data index. The multimodal feature fusion module is used to extract key features of each modal preprocessed data output by the multimodal data preprocessing module, and to use a cross-modal fusion algorithm to achieve deep fusion of the features of each modality, generating and outputting a unified multimodal fusion feature vector. The consultation intent recognition and intelligent triage module is used to receive the multimodal fusion feature vector output by the multimodal feature fusion module, identify the patient's consultation intent and confidence level through the trained intent recognition model, quantify the urgency of the patient's symptoms through a quantification formula, and complete intelligent triage by combining consultation intent, urgency level and doctor information. The multimodal collaborative consultation module for doctors and patients is used to build a dedicated multimodal interactive terminal for doctors, enabling multimodal collaborative consultations between doctors and patients, generating consultation assistance prompts, automatically recording the entire consultation process and synchronizing it to the electronic medical record system; The multimodal auxiliary diagnosis module is used to build a multimodal medical knowledge base and auxiliary diagnosis model. It receives the multimodal fusion feature vector and consultation-related annotations output by the multimodal feature fusion module, outputs auxiliary diagnosis-related information, and verifies the diagnosis results given by the doctor. The consultation result output and confirmation module is used to generate various types of multimodal consultation results, personalize the output method according to the patient's needs, guide the patient to complete the consultation result confirmation in a multimodal way, and store and push the confirmed consultation results; The follow-up and iterative optimization module is used to develop and execute personalized multimodal follow-up plans based on patient diagnosis results, collect follow-up multimodal data and preprocess it, and achieve iterative optimization of the overall process of consultation-related models and devices through quantitative formulas, forming a complete closed loop of consultation.

[0072] In summary, this embodiment addresses the limitations of single-modal interaction by building a multimodal interactive terminal that adapts to different populations and consultation scenarios. This lowers the barrier to entry for various patients, ensuring that patients with different physical conditions and communication abilities can smoothly transmit symptom information and guaranteeing the comprehensiveness of initial information collection. Preprocessing and deep fusion of multimodal data resolve the fragmentation of information across different modalities, integrating various symptom-related data to provide high-quality support for subsequent consultation processes and improve information utilization efficiency. Standardized consultation intent recognition and intelligent triage enable precise matching of patients with corresponding departments and doctors, avoiding unnecessary appointments and cross-departmental travel, and shortening consultation waiting periods. Multimodal collaborative consultation and auxiliary prompting functions help doctors accurately capture key symptom information, avoiding missed or incorrect questions, while reducing doctors' paperwork burden, allowing them to focus on core diagnoses and improving consultation efficiency and accuracy. A multimodal auxiliary diagnosis and result verification mechanism, combined with a medical knowledge base, provides doctors with reliable diagnostic support, effectively reducing the risk of misdiagnosis and missed diagnosis. Personalized, multimodal consultation result output and confirmation processes adapt to different patients' information receiving habits, ensuring patients clearly understand diagnostic results and treatment requirements, thereby improving patient acceptance and treatment adherence. Multimodal follow-up and closed-loop iterative optimization enable continuous improvement of the consultation process and model parameters, adapting to changes in clinical scenarios and actual patient needs.

[0073] Although embodiments of the invention have been shown and described, the scope of the invention will be defined by the appended claims and their equivalents by those skilled in the art.

Claims

1. A method for improving the efficiency of medical consultation by combining multimodal interaction, characterized in that, Specifically, the following steps are included: Step 1: Multimodal consultation initiation and initial data collection. Build an interactive terminal that integrates at least three of the following modalities: voice, image, text, gesture / touch, and physiological signals. Collect multimodal data on patient identity information, basic information, and initial symptoms. Step 2: Multimodal data preprocessing and noise reduction. The initial data of each modality collected in Step 1 are cleaned, denoised, and standardized. After verification, the data is stored and an index is established. Step 3: Multimodal data feature extraction and deep fusion. Extract key features from the preprocessed data of each modality, and use a cross-modal fusion algorithm to achieve deep fusion of features from each modality, generating a unified multimodal fusion feature vector; Step 4: Multimodal consultation intent recognition and intelligent triage. Based on the multimodal fusion feature vector generated in Step 3, the trained intent recognition model identifies the patient's consultation intent and confidence level, quantifies the urgency of the patient's symptoms, and completes intelligent triage. Step 5: Deepen multimodal interactive consultation with doctors, build a dedicated multimodal interactive terminal for doctors, realize multimodal collaborative consultation between doctors and patients, generate consultation assistance prompts and record the entire consultation process; Step 6: Multimodal assisted diagnosis and decision support, construct a multimodal medical knowledge base and assisted diagnosis model, input multimodal fusion features and consultation-related annotations, output assisted diagnosis-related information and verify the diagnosis results; Step 7: Multimodal output and confirmation of consultation results. Generate various types of multimodal consultation results, adapt to the patient's needs, guide the patient to complete the result confirmation, and complete the result storage and push. Step 8: Multimodal follow-up and data iterative optimization. Develop and execute a personalized multimodal follow-up plan, collect and preprocess follow-up multimodal data, and combine quantitative formulas to iteratively optimize the consultation model and process, forming a complete consultation closed loop.

2. The method for improving consultation efficiency by combining multimodal interaction according to claim 1, characterized in that, In step 1, the interactive terminal supports three carriers: mobile phones, tablets, and smart consultation devices, and is suitable for three scenarios: home, community, and hospital outpatient clinic. The multimodal data collection of initial symptoms includes: voice modality collection of patients' oral symptoms and tone and speed characteristics; image modality collection of high-definition images and shooting information of symptom-related parts; physiological signal modality collection of patients' body temperature, heart rate, and blood pressure physiological parameters; and text / gesture modality collection of symptom details. The data tag format is "patient ID-modality type-collection time-integrity tag", and it is encrypted and stored using the AES-256 encryption algorithm.

3. The method for improving consultation efficiency by combining multimodal interaction according to claim 2, characterized in that, Step 2's preprocessing of each modality's data specifically includes: Voice data is denoised using an adaptive wavelet denoising algorithm to extract MFCC feature vectors; image data is denoised using a Gaussian filtering algorithm, and then subjected to size standardization, grayscale conversion, and enhancement; text data is segmented using the jieba word segmentation algorithm, stop words are removed, entity recognition is performed, and the data is converted into structured text; gesture / touch data is smoothed using a moving average algorithm, and then subjected to coordinate standardization and deduplication; physiological signal data is denoised using a sliding window filtering algorithm, outliers are removed, and the data is standardized, and statistical features are calculated; the data validation model validates the data's integrity, accuracy, and standardization, and automatically guides the patient to re-collect data or automatically corrects data if validation fails.

4. The method for improving consultation efficiency by combining multimodal interaction according to claim 3, characterized in that, Step 3, specifically the extraction of features for each modality, includes: extracting semantic and emotional features for speech features using an LSTM model; extracting visual features for images using a ResNet-50 model combined with an SE-Net attention mechanism; extracting semantic features for text features using a simplified BERT model; extracting gesture / touch features using a CNN+fully connected layer structure; and extracting physiological signal features using an MLP model. The cross-modal attention fusion algorithm assigns weights to each modality feature through sample training and combines the attention mechanism to achieve synergistic complementarity among the features, generating a 320-dimensional multimodal fusion feature vector. The fusion feature verification model verifies the feature completeness, relevance, and stability. If the verification fails, the weights are automatically adjusted and the features are re-fused.

5. The method for improving consultation efficiency by combining multimodal interaction according to claim 4, characterized in that, Step 4 is detailed as follows: The intent recognition model is a multimodal consultation intent recognition model based on the Transformer model. The input is a multimodal fusion feature vector, and the output is the consultation intent label and confidence score. The intent labels are divided into three categories: routine consultation, diagnostic assistance, and emergency medical treatment. The model training uses the cross-entropy loss function; the quantization formula in step 4 is... E represents the urgency of the symptoms. These are the weights of voice emotion features, physiological signal features, and multimodal fusion features, respectively, and their sum is 1. These are the quantized values ​​of the corresponding features. An adjustment factor for urgency level; The intelligent triage model matches the corresponding department based on the patient's consultation intent, allocates the priority of treatment according to the urgency level, and matches doctors based on the patient's location, the doctor's workload, and their specialty.

6. The method for improving consultation efficiency by combining multimodal interaction according to claim 5, characterized in that, In step 5, the doctor's multimodal interaction terminal supports both computers and tablets, and has the functions of quickly retrieving multimodal data and initiating multimodal inquiries. The multimodal collaborative consultation supports doctors to initiate inquiries through voice, text, and gestures, and patients can respond in real time through five modal methods. The terminal supports real-time synchronization of the interaction process. The consultation assistance prompt model generates prompts based on multimodal fusion features and consultation intent, such as prompts for key symptoms not asked, the need for supplementary multimodal data, and key questions related to suspected diseases, and pushes them through pop-up windows and voice reminders. The entire consultation process is recorded, including the doctor's questions, the patient's responses, and the doctor's operation records. It is stored in encrypted form and synchronized to the electronic medical record system in real time, and supports doctors to manually mark the key points of the consultation.

7. The method for improving consultation efficiency by combining multimodal interaction according to claim 6, characterized in that, Step 8 involves developing a personalized multimodal follow-up plan based on the patient's diagnosis, treatment plan, and disease severity, including follow-up time, content, and method. Follow-up data collection and preprocessing utilize the preprocessing method from Step 2, and are stored in conjunction with the patient's initial consultation data and results. The quantification formula in Step 8 is... ,in These are the parameters of the medical history model after and before the iteration, respectively. The learning rate is k, and the number of follow-up data samples is k. Let i be the weight of the i-th group of follow-up data. These are the actual feature values ​​and model-predicted feature values ​​of the i-th group of follow-up data, respectively. The iterative optimization cycle is once a month. After optimization, the effect is verified by four indicators. If the indicators are not met, the parameters are readjusted for optimization.

8. The method for improving consultation efficiency by combining multimodal interaction according to claim 7, characterized in that, Step 7's multimodal consultation results include a structured text diagnostic report, a natural speech synthesis voice report, and a visual image report. The text report annotates professional terms, the voice report repeats key information, and the image report includes a medication schedule and symptom comparison charts. Personalized output is determined based on the patient's age, education level, and interaction habits, and patients can manually switch output modes. Patient confirmation uses a multimodal approach; if the patient is unclear, the consultation results are automatically interpreted and confirmation is prompted again, and patients can initiate follow-up questions through the multimodal approach. The consultation results are stored in encrypted form and pushed to the patient's terminal and electronic medical record system, and patients can share them with family members in encrypted form. In step 6, the multimodal medical knowledge base adopts distributed storage, supports real-time updates and multimodal retrieval, the auxiliary diagnostic model training adopts the contrastive learning loss function, and the diagnostic result verification model judges the rationality of the diagnostic result through the matching degree and forces abnormal diagnostic results to be re-verified.

9. A diagnostic consultation efficiency improvement device combining multimodal interaction, characterized in that, include: The multimodal initial data acquisition module is used to build an interactive terminal that integrates at least three modalities of voice, image, text, gesture / touch and physiological signals, to collect multimodal data of patient identity information, basic information and initial symptoms, and to temporarily store the collected data; The multimodal data preprocessing module is used to clean, reduce noise, and standardize the initial data of each modality collected by the multimodal initial data acquisition module, verify the preprocessed data, store the verified data, and establish a data index. The multimodal feature fusion module is used to extract key features of each modal preprocessed data output by the multimodal data preprocessing module, and to use a cross-modal fusion algorithm to achieve deep fusion of the features of each modality, generating and outputting a unified multimodal fusion feature vector. The consultation intent recognition and intelligent triage module is used to receive the multimodal fusion feature vector output by the multimodal feature fusion module, identify the patient's consultation intent and confidence level through the trained intent recognition model, quantify the urgency of the patient's symptoms through a quantification formula, and complete intelligent triage by combining consultation intent, urgency level and doctor information. The multimodal collaborative consultation module for doctors and patients is used to build a dedicated multimodal interactive terminal for doctors, enabling multimodal collaborative consultations between doctors and patients, generating consultation assistance prompts, automatically recording the entire consultation process and synchronizing it to the electronic medical record system; The multimodal auxiliary diagnosis module is used to build a multimodal medical knowledge base and auxiliary diagnosis model. It receives the multimodal fusion feature vector and consultation-related annotations output by the multimodal feature fusion module, outputs auxiliary diagnosis-related information, and verifies the diagnosis results given by the doctor. The consultation result output and confirmation module is used to generate various types of multimodal consultation results, personalize the output method according to the patient's needs, guide the patient to complete the consultation result confirmation in a multimodal way, and store and push the confirmed consultation results; The follow-up and iterative optimization module is used to develop and execute personalized multimodal follow-up plans based on patient diagnosis results, collect follow-up multimodal data and preprocess it, and achieve iterative optimization of the overall process of consultation-related models and devices through quantitative formulas, forming a complete closed loop of consultation.

Images