Digital human auxiliary diagnosis method and device, electronic equipment and readable storage medium

Abstract

The invention provides a digital human auxiliary diagnosis method and device, electronic equipment and a readable storage medium, and is applied to the technical field of medical data processing. The method comprises the following steps: processing question and answer audio information of a patient to generate question semantic information and sound feature information of the patient; processing the real-time facial video information and the physiological status information of the patient to generate facial micro-expression feature information and physiological index feature information; processing the sound feature information, the facial micro-expression feature information and the physiological index feature information based on a multi-modal fusion network to generate real-time emotional state information of the patient; processing the real-time emotional state information of the patient, the question semantic information of the patient and the preference information of the patient to generate target reply style information of the digital person; and processing the real-time operation information of the hospital department of the target area and the historical case record information of the patient based on the question and answer planning network, generating resource allocation result information, and generating digital human reply information.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and particularly relates to a digital human assisted diagnosis method, device, electronic device and readable storage medium. Background Art

[0002] With the development of large models and digital human interaction technologies, patients can have voice conversations, ask and answer questions, and have emotional exchanges with Agent intelligent agents through a visual digital human emotional interaction platform, thereby greatly reducing the psychological pressure of patients and increasing psychological comfort. Especially in the medical and health industry, facing the threat to their own health, patients have problems such as lack of professional disease and health knowledge, unfamiliarity with the medical treatment process, confusion about how to utilize medical resources (such as department diagnosis and treatment scope / doctor expertise), and inability to accurately describe their own conditions. Therefore, a visual assisted diagnosis platform based on digital humans is particularly important, especially for special diseases or how to integrate with the current hospital resources and environment. Therefore, how to integrate multi-modal data analysis and improve the accuracy of intelligent question answering on the visual emotional interaction digital human interaction platform is an urgent problem for those skilled in the art to solve.

[0003] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present disclosure, and thus includes information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0004] The purpose of the present application is to provide a digital human assisted diagnosis method, device, electronic device and readable storage medium, which at least to a certain extent overcome the problems existing in the prior art. By obtaining multi-source information of patients, such as question-and-answer audio, facial video, physiological state, preferences and historical cases, etc., as well as real-time operation data of hospital departments. Then, various types of information are processed separately. The question-and-answer audio is converted into question semantics and voice features, and the facial video and physiological state information are converted into corresponding features to judge the emotional state. Then, based on these processing results, a resource allocation result is generated through a question-and-answer planning network, and then combined with the target reply style to generate digital human reply information, providing diagnostic suggestions, department and doctor recommendations for patients. This technology can improve the diagnostic accuracy and patient experience, reduce the waiting time, optimize resource utilization, has significant technical advantages and application value, and brings a new development direction to the field of medical diagnosis.

[0005] Other features and advantages of the present application will become apparent through the following detailed description, or be learned in part through the practice of the present invention.

[0006] According to one aspect of the present application, there is provided a digital human assisted diagnosis method, including: obtaining the Q&A audio information of the patient, real-time facial video information, the physiological state information of the patient, the preference information of the patient, the historical case record information of the patient, and the real-time operation information of the target regional hospital department; processing the Q&A audio information of the patient to generate the question semantic information and voice feature information of the patient; processing the real-time facial video information and the physiological state information of the patient to generate facial micro-expression feature information and physiological index feature information; processing the voice feature information, facial micro-expression feature information and physiological index feature information based on a multi-modal fusion network to generate the real-time emotional state information of the patient; processing the real-time emotional state information of the patient, the question semantic information of the patient and the preference information of the patient to generate the target reply style information of the digital human; processing the real-time operation information of the target regional hospital department and the historical case record information of the patient based on a Q&A planning network to generate resource allocation result information; processing the resource allocation result information based on the target reply style information of the digital human to generate digital human reply information.

[0007] Another aspect of the present application provides a digital human assisted diagnosis device, characterized by including: an acquisition module, configured to obtain the Q&A audio information of the patient, real-time facial video information, the physiological state information of the patient, the preference information of the patient, the historical case record information of the patient, and the real-time operation information of the target regional hospital department; a processing module, configured to process the Q&A audio information of the patient to generate the question semantic information and voice feature information of the patient; process the real-time facial video information and the physiological state information of the patient to generate facial micro-expression feature information and physiological index feature information; process the voice feature information, facial micro-expression feature information and physiological index feature information based on a multi-modal fusion network to generate the real-time emotional state information of the patient; process the real-time emotional state information of the patient, the question semantic information of the patient and the preference information of the patient to generate the target reply style information of the digital human; process the real-time operation information of the target regional hospital department and the historical case record information of the patient based on a Q&A planning network to generate resource allocation result information; process the resource allocation result information based on the target reply style information of the digital human to generate digital human reply information.

[0008] According to yet another aspect of the present application, there is provided a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a second processor, the above-mentioned digital human assisted diagnosis method is implemented.

[0009] A digital human-assisted diagnosis method, device, electronic device, and readable storage medium provided by this application. The server obtains multi-source information of patients, such as Q&A audio, facial video, physiological status, preferences, and historical case information, as well as real-time operation data of hospital departments. Then, various types of information are processed separately. For example, the Q&A audio is converted into question semantics and voice features, and the facial video and physiological status information are converted into corresponding features to judge the emotional state. Then, based on these processing results, a resource allocation result is generated through a Q&A planning network, and combined with the target reply style to generate digital human reply information, providing diagnostic suggestions, department and doctor recommendations for patients. This technology can improve the diagnostic accuracy and patient experience, reduce the waiting time, optimize resource utilization, has significant technical advantages and application value, and brings a new development direction to the medical diagnosis field.

[0010] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 The flowchart showing a digital human-assisted diagnosis method provided by an embodiment of the present application;

[0012] Figure 2 The structural schematic diagram showing a digital human-assisted diagnosis device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0013] The following describes the preferred embodiments of the present invention with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only for the purpose of illustrating and explaining the present invention, and are not used to limit the present invention.

[0014] The following combines Figure 1 to describe the digital human-assisted diagnosis method according to an exemplary embodiment of the present application. In one embodiment, the present application also proposes a digital human-assisted diagnosis method, device, electronic device, and readable storage medium. Figure 1 Schematically shows a flowchart of a digital human-assisted diagnosis method according to an embodiment of the present application. As Figure 1 shown, this method is applied to a server and includes:

[0015] S101, obtaining the Q&A audio information of the patient, real-time facial video information, physiological status information of the patient, preference information of the patient, historical case record information of the patient, and real-time operation information of the target regional hospital department.

[0016] In one implementation, patient Xiao Li came to the hospital for a visit and was about to use the digital human-assisted diagnosis system for preliminary consultation. At this time, the process of the system obtaining relevant information is as follows: Xiao Li said to the microphone on the digital human-assisted diagnosis device, "I've been having headaches lately, feeling extremely tired, and sometimes my chest feels a bit stuffy. What's going on?" The microphone of the system recorded the voice of Xiao Li speaking in real time and obtained the Q&A audio information. After that, the system would process this audio, such as noise reduction, speech-to-text conversion, etc., in order to further analyze the problem semantics and extract voice features. The camera on the device captured Xiao Li's face in real time and obtained real-time facial video information. Through image preprocessing, problems such as light interference and blurring were removed, and then key point detection was carried out to identify the positions and contour changes of Xiao Li's facial features, such as the degree of eyebrow wrinkling, the upward or downward movement of the corners of the mouth, etc. At the same time, expression classification was performed to determine whether Xiao Li's current expression was anxious, calm, or in other emotional states, providing a basis for subsequent analysis of Xiao Li's emotions.

[0017] Xiao Li was wearing a smart bracelet with monitoring functions. The bracelet was connected to the digital human-assisted diagnosis system and transmitted physiological data such as real-time heart rate and blood oxygen saturation to the system. These data were the physiological state information of the patient. The system would perform filtering on these data to remove noise and outliers, obtain stable physiological signal data, and then extract features such as heart rate variability to assist in judging Xiao Li's physical condition. When Xiao Li used this system or visited the hospital before, he had filled out a personal preference questionnaire, indicating that he preferred to choose hospital departments closer to his home and hoped that doctors would use easy-to-understand language when explaining the condition. These information were recorded and stored by the system and were obtained as preference information during this diagnosis process to provide more personalized diagnosis suggestions and services for Xiao Li.

[0018] The system obtained Xiao Li's historical case record information by docking with the hospital's information system. This included Xiao Li's past medical records, such as diseases he had suffered from (such as colds, pneumonia, etc.), diagnosis results, treatment plans, and allergy history (such as being allergic to penicillin). These historical information were crucial for analyzing Xiao Li's current symptoms and potential disease risks, helping the digital human-assisted diagnosis system to understand Xiao Li's health condition more comprehensively, classify the symptoms provided by Xiao Li, judge the most likely problems, and then match the corresponding departments and doctors, as well as the waiting situation for seeing a doctor. The digital human-assisted diagnosis system accessed the hospital's HIS system to obtain real-time operation information of hospital departments in the target area. For example, the registration volume of current internal medicine, cardiovascular department, etc., the consultation efficiency of doctors in each department (such as the time required for an average doctor to consult a patient), and the waiting time of patients in each department. These real-time data allowed the system to fully consider the actual operation situation of the hospital when recommending departments and doctors, providing more reasonable consultation suggestions for Xiao Li.

[0019] S102, process the Q&A audio information of the patient to generate the semantic information of the patient's questions and the voice feature information.

[0020] In one implementation, sample and denoise the Q&A audio information of the patient to generate the preprocessed Q&A audio. Xiao Li came to the hospital and used the digital human-assisted diagnosis device for consultation. He said to the device microphone, "I've been having headaches lately, feeling extremely tired, and sometimes my chest feels a bit stuffy. What's going on?" Due to the noisy hospital environment, the audio collected by the microphone was mixed with noises such as other people's voices, footsteps, and the operating sounds of medical equipment. To ensure the accuracy of subsequent processing, the system uses advanced sampling techniques to collect the audio signal at a certain frequency and applies noise reduction algorithms such as wavelet denoising and adaptive filtering to remove these background noises. Wavelet denoising can separate the noise and the useful signal in different frequency subbands by performing wavelet transform on the audio signal, then remove the signal in the noise subband, and reconstruct the audio signal through inverse wavelet transform; adaptive filtering automatically adjusts the parameters of the filter according to the statistical characteristics of the audio signal to achieve the best noise reduction effect. After these processes, a clearer preprocessed Q&A audio is obtained, providing high-quality audio data for the subsequent steps.

[0021] Perform speech-to-text processing on the preprocessed Q&A audio to generate the initial text information. For the audio after sampling and denoising, the system uses speech-to-text technology for processing. The speech-to-text technology adopts a speech recognition model based on deep learning, such as a model based on convolutional neural network (CNN) and recurrent neural network (RNN). This model is trained with a large amount of speech data and can learn the mapping relationship between speech signals and text. The system inputs the preprocessed audio into the speech recognition model, and the model extracts and analyzes the speech features in the audio, identifies the text corresponding to each speech segment, and finally converts Xiao Li's speech content into the initial text information of "I've been having headaches lately, feeling extremely tired, and sometimes my chest feels a bit stuffy. What's going on?" This step converts the speech information into a text form that can be directly processed by a computer, facilitating subsequent text analysis operations.

[0022] Part-of-speech tagging and syntactic analysis are performed on the initial text information to generate syntactic structure features. For the generated initial text information, the system uses part-of-speech tagging and syntactic analysis tools in natural language processing technology. Part-of-speech tagging classifies the part of speech of each word in the text. For example, using a method that combines rules and statistics, the system identifies "I" as a personal pronoun and the subject of the sentence; "recently" as an adverb of time, used to limit the time range when the action occurs; "have a headache", "be tired", and "be stuffy" as verbs, describing the state of the subject; "this" as a demonstrative pronoun; "what's going on" as an interrogative phrase; and "ah" as a modal particle. In terms of syntactic analysis, the system determines that "I always have a headache recently" is a subject-predicate-object structure, where "I" is the subject, "have a headache" is the predicate, and "recently always" serves as an adverbial modifying the predicate to indicate time frequency; "also feel extremely tired" is a parallel structure for supplementary explanation, sharing the subject "I" with the previous sentence, "feel" is the predicate, and "extremely tired" is the object complement; "sometimes my chest also feels a bit stuffy" is also a subject-predicate-object structure, "sometimes" is the adverbial of time, "chest" is the subject, "feel stuffy" is the predicate, and "also a bit" is an adverb of degree modifying the predicate. Through these part-of-speech tagging and syntactic analysis, the system generates syntactic structure features, clearly showing the components and semantic relationships of the sentence, which helps to deeply understand the meaning of the sentence.

[0023] Speech feature extraction is performed on the preprocessed Q&A audio to generate intonation features, speech rate features, and volume features. In the preprocessed Q&A audio, the system extracts speech features through audio signal processing technology. In terms of intonation features, the system analyzes the fundamental frequency changes in the audio and finds that when Xiao Li mentions symptoms such as "have a headache", "be tired", and "be stuffy", the fundamental frequency has an obvious upward trend, indicating that his intonation rises slightly. This intonation change often reflects the speaker's emphasis and concern about these contents. The calculation of speech rate features is achieved by counting the number of words Xiao Li says per unit time. After calculation, his speech rate is moderate, about 3 - 4 words per second. The acquisition of volume features is through analyzing the amplitude size of the audio signal, and it is determined that Xiao Li's speaking volume is normal, without being too loud or too soft. These intonation, speech rate, and volume features provide more information about Xiao Li's emotions and key points of expression at the speech level, which has important reference value for subsequent diagnosis and interaction.

[0024] Process the syntactic structure features based on the medical knowledge graph and semantic understanding model to generate the semantic information of the patient's question. Based on the previously generated syntactic structure features, the system uses the medical knowledge graph and semantic understanding model to deeply understand the meaning of Xiao Li's question. The medical knowledge graph is a structured knowledge base that contains a large number of medical concepts (such as diseases, symptoms, treatment methods, etc.) and their relationships (such as the association between diseases and symptoms, the correspondence between diseases and treatment methods, etc.). The semantic understanding model uses semantic parsing techniques in natural language processing to perform in-depth semantic understanding of the text. The system first extracts keywords such as "headache", "tiredness", and "chest tightness" from the syntactic structure features, and then searches in the medical knowledge graph for diseases that these symptoms may be associated with. At the same time, the semantic understanding model combines the syntactic structure and context information of the sentence to understand that the intention of Xiao Li's question is to inquire about the diseases that these symptoms may correspond to in himself. For example, the system discovers through the knowledge graph that "headache" may be related to various diseases such as neurological diseases, cardiovascular diseases, and otolaryngological diseases; "tiredness" may be a symptom of physical fatigue, anemia, or certain chronic diseases; "chest tightness" is more closely related to cardiovascular diseases and respiratory diseases. Combining this information, the system determines that the semantic information of Xiao Li's question is to inquire about the diseases that these symptoms may correspond to in himself, providing a clear direction for subsequent diagnosis.

[0025] Normalize and perform feature selection on the intonation features, speech rate features, and volume features to generate voice feature information. The system normalizes the previously extracted intonation features, speech rate features, and volume features. Normalization is to map the values of these features to a unified interval, such as [0,1], which can eliminate the differences in dimension and numerical range between different features, facilitating comparison and analysis. For example, the fundamental frequency change range of intonation, the word count range of speech rate, and the amplitude range of volume are all uniformly mapped to the [0,1] interval. Then perform feature selection. In the current diagnosis scenario, through analysis, it is found that the upward intonation feature when Xiao Li mentions symptoms while speaking is crucial for judging his anxiety, because anxious patients tend to unconsciously raise their intonation when describing symptoms. The speech rate feature is relatively less important in the preliminary diagnosis because the speech rate may be affected by various factors and has a weak direct correlation with diseases. After such normalization and feature selection processing, some features that are less helpful or redundant for diagnosis are removed, and the most representative and relevant features are retained to generate more concise and effective voice feature information for subsequent diagnostic analysis and digital human interaction, helping the digital human to more accurately understand Xiao Li's emotions and needs and provide more appropriate responses.

[0026] S103. Process the real-time facial video information and the patient's physiological state information to generate facial micro-expression feature information and physiological index feature information.

[0027] In one implementation, image preprocessing is performed on real-time facial video information to generate a preprocessed facial video. Xiao Li stands in front of the digital human assisted diagnosis device for consultation, and the camera on the device captures his facial video in real time. Since the ambient light may be uneven, the video screen may be noisy or blurred. The system first performs image preprocessing on the collected real-time facial video information, and uses grayscale transformation, filtering and other technologies to adjust the brightness and contrast of the image and remove noise in the image. For example, a Gaussian filtering algorithm is used to calculate a weighted average value based on the grayscale value of the pixel point and its neighboring pixels to replace the current pixel value, thereby making the image smoother and reducing noise interference. After these processes, a preprocessed facial video is generated to provide clear and stable image data for subsequent analysis. Signal filtering is performed on the patient's physiological state information to generate target physiological signal data. The smart bracelet worn by Xiao Li is connected to the digital human assisted diagnosis system to transmit his heart rate, blood oxygen saturation and other physiological data in real time. However, these data may be affected by factors such as electromagnetic interference during transmission, resulting in some abnormal fluctuations or noise. The system performs signal filtering on these physiological status information, using algorithms such as Kalman filtering, and based on the historical data and current measurement values ​​of the signal, it makes the best estimate of the signal, removes noise and outliers, and obtains stable and reliable target physiological signal data. The physiological data processed in this way can more accurately reflect Xiao Li's real physical condition and provide effective data support for subsequent diagnostic analysis.

[0028] The preprocessed facial video is subjected to key point detection to generate facial key point coordinate features. Based on the preprocessed facial video, the system uses a detection model based on a convolutional neural network (CNN) to detect key points on Xiao Li's face. The detection model mainly includes the following key parts: The input layer is responsible for receiving the preprocessed facial video data. In Xiao Li's case, the facial video images that have been preprocessed, light interference and noise removed, and brightness and contrast adjusted are input into the model. These images are presented in the form of a pixel matrix to provide raw data for subsequent processing. The convolution layer is one of the core components of CNN and consists of multiple convolution kernels. The convolution kernel slides on the image and extracts local features of the image through convolution operations. For example, different convolution kernels can capture features such as edges and textures of the face. When processing Xiao Li's facial image, the convolution layer extracts features such as the outline edge of the eye and the texture direction of the eyebrow, which are crucial for determining the location of facial key points. Multiple convolution layers can be layered to continuously extract more advanced and abstract features.

[0029] The pooling layer mainly downsamples the feature maps output by the convolutional layer, reducing the amount of data, lowering the computational complexity, and at the same time retaining important features. Common pooling methods include max pooling and average pooling. Taking max pooling as an example, it selects the maximum value within a small area as the output of that area. When analyzing Xiao Li's facial image, the pooling layer can simplify the data without losing key information, enabling the model to process data more efficiently. The fully connected layer performs a flatten operation on the feature maps after convolution and pooling, converting them into one-dimensional vectors, and then connects to multiple fully connected layers. The fully connected layer can comprehensively analyze and integrate these features, learning the complex relationships between features. In facial key point detection, the fully connected layer can output prediction values related to the coordinates of facial key points based on the previously extracted facial features. According to the training objective of the model, the coordinate information of the facial key points is output. For Xiao Li's face, the output layer will give the specific coordinates of key points such as the corners of the eyes, the positions of the pupils, the corners of the mouth, and the lip contours. These coordinate information constitute the facial key point coordinate features, providing accurate and crucial basic data for subsequent precise analysis of Xiao Li's facial expression changes, such as judging emotional states like anxiety and calmness.

[0030] Perform expression classification processing on the preprocessed facial video to generate expression category features. Also based on the preprocessed facial video, the system uses an expression classification model to analyze Xiao Li's expressions, which mainly includes the following key parts, and each part works together to achieve accurate expression recognition. The data input layer is responsible for receiving the preprocessed facial video data. In the case of Xiao Li, the input is facial images that have undergone preprocessing operations such as image enhancement and noise reduction, and these images enter the model in a specific format (such as a pixel matrix in RGB format), providing a basis for subsequent feature extraction and analysis. The feature extraction layer consists of multiple convolutional layers and pooling layers. The convolutional layer slides a convolutional kernel over the image to perform a convolution operation on the image, extracting various local features of the face. For example, different convolutional kernels can capture features such as the shape changes of the eyes, the dynamics of the eyebrows, and the displacement of the corners of the mouth. The pooling layer then downsamples the feature maps output by the convolutional layer, reducing the amount of data while retaining key features and lowering the computational complexity of the model. For example, the max pooling layer selects the maximum value in a local area as the output, which can not only highlight key features but also avoid overfitting. Through multiple layers of convolution and pooling operations, the model can extract high-level and abstract expression features from the facial images.

[0031] After feature extraction, the obtained feature map is flattened into a one-dimensional vector and then input into the fully connected layer. The neurons in the fully connected layer are connected to all neurons in the previous layer, enabling a comprehensive analysis of the complex relationships between various expression features extracted previously. During this process, the model learns the associations between different feature combinations and specific expressions. For example, a certain specific combination change in the eyes and eyebrows may represent an anxious emotion. The fully connected layer further extracts information that is significant for expression classification through in-depth integration of these features. Common classifiers such as the Softmax classifier process the feature vector output by the fully connected layer and calculate the probabilities of the facial image belonging to different expression categories. For Xiao Li's facial image, the Softmax classifier will output the probability values of his expression belonging to different categories such as anxious, calm, and painful based on the previously learned feature patterns. For instance, if the model determines that the probability of Xiao Li's expression being anxious is much higher than other categories, it will confirm that Xiao Li's expression is anxious and generate the corresponding expression category features. This feature can intuitively reflect Xiao Li's current emotional state and provide an important basis for subsequent diagnosis and interaction.

[0032] Feature extraction processing is performed on the target physiological signal data to generate heart rate variability features and blood oxygen saturation features. For the target physiological signal data after signal filtering processing, the system further performs feature extraction. For heart rate data, heart rate variability features are calculated by analyzing the changes in the time intervals between adjacent heartbeats to obtain the fluctuations of the heart rate. Under normal circumstances, the heart rate has a certain variability, and certain diseases or emotional states may cause changes in heart rate variability. For blood oxygen saturation data, its value is directly extracted as the blood oxygen saturation feature. These features reflect the functional states of Xiao Li's cardiovascular and respiratory systems and provide important physiological index bases for diagnosis. Quantitative analysis processing is performed on the heart rate variability features and blood oxygen saturation features to generate physiological index quantitative features. For heart rate variability, quantitative indicators such as its standard deviation and low-frequency to high-frequency power ratio are calculated to convert the complex changes in heart rate variability into specific numerical values. For example, the larger the standard deviation, the greater the heart rate variability, indicating that the regulatory ability of the cardiovascular system may be stronger or there may be certain abnormalities. For blood oxygen saturation, its value is compared with the normal range, and quantitative indicators such as the degree of deviation from the normal range are calculated. Through these quantitative analyses, physiological index quantitative features are generated, making the physiological data easier to analyze and compare and providing a more accurate basis for judging Xiao Li's physical condition.

[0033] Quantitatively analyze the facial expression category features and facial key point coordinate features to generate facial micro-expression quantitative features. To analyze Xiao Li's facial expressions more precisely, the system quantitatively analyzes the facial expression category features and facial key point coordinate features. For facial expression categories, according to the feature vectors of different expressions in the database, a quantitative score is assigned to each expression. For example, an anxious expression corresponds to a relatively high score for the degree of tension. For the facial key point coordinate features, calculate the geometric relationships such as the distances and angles between the key points, and perform normalization processing to convert these geometric features into quantitative values. For example, calculate the ratio of the distance between the two eyes to the overall width of the face, the angles of the corners of the mouth rising or falling, etc. Through these quantitative analyses, facial micro-expression quantitative features are generated to more meticulously describe the changes in Xiao Li's facial expressions and emotional states.

[0034] Based on the sentiment analysis model and medical knowledge, process the physiological index quantitative features and facial micro-expression quantitative features to generate sentiment tendency information and the corresponding weight calculation result information. The sentiment analysis model in the digital human assisted diagnosis system is used to comprehensively analyze the physiological indexes and facial micro-expression quantitative features to judge the patient's emotional state. It contains multiple components with different functions but working in cooperation. The input layer is responsible for receiving external data. In this scenario, the input data are the quantitatively processed physiological index features and facial micro-expression features. For Xiao Li's case, the input is the quantitative data of physiological indexes such as reduced heart rate variability and slightly low blood oxygen saturation, as well as the quantitative values corresponding to facial micro-expression features such as frowning eyebrows and anxious looks in the eyes. These data are input into the model in the form of specific vectors or matrices, providing a basis for subsequent analysis.

[0035] The feature processing layer uses data preprocessing techniques to perform operations such as normalization and standardization on the input original features to ensure the comparability of different types of features. For example, map the values of heart rate variability and blood oxygen saturation to a unified interval to eliminate the influence of dimensions and make the model training more stable. Integrate the features from different modalities (physiological indexes and facial micro-expressions) to form a comprehensive feature vector. The concatenation method can be adopted to connect the physiological index vector and the facial micro-expression vector in a certain order, enabling the model to analyze the relationships between these features as a whole. The machine learning algorithm (taking the support vector machine SVM as an example) uses SVM, which will find an optimal classification hyperplane in the high-dimensional space to classify the feature vectors corresponding to different emotional states. In the training stage, SVM learns how to divide different emotional categories according to a large number of sample data with labeled emotional states (including physiological indexes and facial micro-expression features). For Xiao Li's data, SVM will judge which emotional category his comprehensive feature vector is closer to based on the patterns learned previously.

[0036] Deep learning algorithms (taking the convolutional neural network CNN as an example) automatically extract features through convolutional layers. The convolutional kernels in the convolutional layer slide on the input comprehensive feature vector to capture the relationships between local features. For example, it learns the emotional features represented by the combination of heart rate variability and specific facial micro-expressions. The pooling layer then reduces the dimensionality of the output of the convolutional layer, reducing the amount of data while retaining key features. After multiple convolutional and pooling operations, the extracted features are integrated through a fully connected layer, and the prediction results for different emotion categories are output. According to medical knowledge and a large amount of experimental data, weights are assigned to different features. For the case of Xiao Li, the weight of the heart rate variability feature for anxiety judgment is determined to be 0.4, and the weight of the facial micro-expression feature is 0.6. This is because facial micro-expressions may be more intuitive and important in directly reflecting emotions, but physiological indicators can also reflect the relationship between emotions and the body from the side. Through weight assignment, the model can more accurately synthesize different features to judge the emotional tendency. According to the calculation results of the core layer of the model and the weight assignment, the emotional tendency information and the corresponding weight calculation result information are output. For Xiao Li, the output results indicate that he is in an anxious state, and it is given that this judgment is based on the results of the weight calculation of each feature, providing a clear basis for subsequent diagnosis and adjustment of interaction strategies.

[0037] Process the emotional tendency information and the corresponding weight calculation result information to generate comprehensive feature information, where the comprehensive feature information is used to comprehensively represent the patient's emotional and physical states. The system performs the last step of processing on the emotional tendency information and the corresponding weight calculation result information, fusing and integrating this information. According to the weight calculation results, weighted summation or other fusion operations are performed on different emotional tendency information to generate comprehensive feature information. This comprehensive feature information can comprehensively represent Xiao Li's emotional and physical states. For example, the comprehensive feature information may show that Xiao Li is currently in a moderately anxious state and there are certain physiological abnormality risks in the body, and it is necessary to further pay attention to the conditions of the cardiovascular and respiratory systems. This comprehensive feature information provides an important basis for the subsequent diagnosis suggestions and interaction strategies of the digital human.

[0038] S104, Process the voice feature information, facial micro-expression feature information, and physiological index feature information based on a multi-modal fusion network to generate the patient's real-time emotional state information.

[0039] In one implementation, the voice feature information, facial micro-expression feature information, and physiological index feature information are normalized and standardized to generate preprocessed multi-modal feature information. When Xiao Li interacts with the digital human, the system has obtained his voice feature information (such as rising intonation, moderate speech rate, and normal volume), facial micro-expression feature information (changes in the coordinates of facial key points such as frowning eyebrows and nervous eyes, and the expression classification is anxiety), and physiological index feature information (decreased heart rate variability and slightly low blood oxygen saturation). Since the data ranges and dimensions of these feature information are different, for the convenience of subsequent unified analysis, the system normalizes and standardizes them. For the intonation feature in the voice feature information, assuming that the original fundamental frequency change range is [80Hz, 200Hz], the system maps it to the [0, 1] interval through the normalization formula. For the speech rate feature, if the normal speech rate range is [2, 6] words per second, the speech rate of 3 - 4 words per second of Xiao Li is also normalized accordingly. The same applies to the volume feature, mapping the actual volume amplitude range to a unified interval.

[0040] In terms of facial micro-expression feature information, the facial key point coordinate features are normalized according to the overall facial size, making the key point coordinates of different faces comparable. The expression category features are standardized by methods such as one-hot encoding, converting the category of "anxiety" into a numerical vector that can be processed by a computer. In the physiological index feature information, quantization indicators such as the standard deviation of heart rate variability and the blood oxygen saturation value are standardized according to their respective normal ranges. For example, the normal standard deviation range of heart rate variability is [5, 15], and the measured standard deviation of Xiao Li's heart rate variability is 8, which is converted into the corresponding value through the standardization formula; the normal range of blood oxygen saturation is [95%, 100%], and Xiao Li's blood oxygen saturation is 93%, which is also standardized accordingly. After these processes, preprocessed multi-modal feature information is generated, providing a unified data basis for subsequent analysis.

[0041] Based on the emotion classification model library, feature matching and analysis processing are performed on the preprocessed multi-modal feature information to generate emotion category prediction information and emotion intensity prediction information. The system inputs the preprocessed multi-modal feature information into the emotion classification model library. The emotion classification model library contains various models trained with a large amount of data, such as convolutional neural network (CNN) models or recurrent neural network (RNN) models based on deep learning. These models have learned a large amount of multi-modal data (voice, facial expressions, physiological indicators, etc.) with emotion labels and can identify the emotion categories and intensities corresponding to different feature combinations.

[0042] The model analyzes the preprocessed multi-modal feature information of Xiao Li, and perceives from the voice features that the intonation changes when he describes his symptoms reflect a worried mood; the facial micro-expression features show that his eyebrows are tightly frowned and his eyes are tense, which conforms to the characteristics of an anxious expression; the increase in heart rate variability and slightly low blood oxygen saturation in the physiological index features are also somewhat related to the physiological reactions under an anxious state. Combining this information, the model predicts that Xiao Li's emotion category is "anxiety" and predicts his emotion intensity as "moderate" according to the obviousness of the features, thus generating emotion category prediction information and emotion intensity prediction information.

[0043] Based on the emotion category prediction information and emotion intensity prediction information, the key data in the preprocessed multi-modal feature information are located and extracted to generate key emotion feature data. According to the generated emotion category prediction information (anxiety) and emotion intensity prediction information (moderate), the system locates and extracts the key data in the preprocessed multi-modal feature information. In the voice feature information, the feature data corresponding to the most obvious upward intonation when Xiao Li mentions his symptoms are mainly extracted, because this part of the data strongly reflects his concern and worry about his own symptoms. In terms of facial micro-expression feature information, the coordinate change data of facial key points closely related to the anxious expression are extracted, such as the degree of narrowing of the distance between the eyebrows and the degree of widening of the eyes, etc., which are all key facial features representing anxious emotions. In the physiological index feature information, the data points of the increase in heart rate variability and slightly low blood oxygen saturation and their related time series data are located, because these data are related to the physiological changes under an anxious state. By extracting these key data, key emotion feature data are obtained, which more accurately represent Xiao Li's current emotional state.

[0044] Fuse and weight the key emotion feature data to generate the real-time emotion state information of the patient. The real-time emotion state information of the patient is used to characterize the patient's current emotion state and the intensity of the emotion. The system fuses and weights the key emotion feature data. Different features have different importance for judging the emotion state, so weights need to be assigned to them. For example, through a large number of experiments and data analysis, it is determined that the weight of the voice feature in judging Xiaoli's current emotion state is 0.3, the weight of the facial micro-expression feature is 0.4, and the weight of the physiological index feature is 0.3. For the key voice feature data, calculate its contribution value to the emotion state according to its proportion and weight in the whole voice data; the same applies to the key facial micro-expression feature data and the key physiological index feature data. Then, fuse and calculate these contribution values, such as using the weighted sum method. Suppose the contribution value of the key voice feature data is 0.2, the contribution value of the key facial micro-expression feature data is 0.3, and the contribution value of the key physiological index feature data is 0.2. The comprehensive value obtained through weighted sum calculation is 0.2×0.3 + 0.3×0.4 + 0.2×0.3 = 0.24. This comprehensive value represents the quantification result of Xiaoli's current emotion state. Combining the emotion category prediction information (anxiety), finally generate the real-time emotion state information of the patient, indicating that Xiaoli is currently in a moderate anxiety state. This information accurately characterizes Xiaoli's current emotion state and the intensity of the emotion, providing an important basis for the subsequent digital human to select an appropriate response style and provide diagnostic suggestions.

[0045] S105. Process the real-time emotion state information of the patient, the semantic information of the patient's question, and the preference information of the patient to generate the target response style information of the digital human.

[0046] In one implementation, feature extraction and classification processing are performed on the real-time emotion state information of the patient, the semantic information of the patient's question, and the preference information of the patient to generate emotion tendency feature information, question type feature information, and preference pattern feature information. When Xiaoli interacts with the digital human, the system has obtained his real-time emotion state information (being in a moderate anxiety state), the semantic information of the question (asking about the diseases corresponding to headache, fatigue, and chest tightness), and the preference information (tending to choose a hospital department close to home and hoping that the doctor can explain the condition in an easy-to-understand way).

[0047] The system extracts the emotional tendency of anxiety from his real-time emotional state information. Further analysis reveals that his emotions fluctuate greatly when describing symptoms, and this emotional instability is also regarded as part of the emotional tendency characteristics. These characteristics are quantified. For example, a numerical range for the degree of anxiety is set, and a specific value is assigned to his anxiety level according to Xiao Li's performance, representing his emotional tendency characteristic information. Regarding the problem semantic information, the system determines that this belongs to the category of symptom consultation questions. Further breakdown shows that since it involves multiple physical discomfort symptoms, it can be classified as a comprehensive symptom consultation question. By analyzing the keywords and grammatical structures in the question, features related to symptom description and disease inquiry are extracted and transformed into feature vectors recognizable by the computer, forming the problem type feature information. For the preference information, the system extracts two key elements: distance preference (hoping to choose a hospital department close to home) and communication style preference (hoping for easy-to-understand explanations). The distance preference is divided into different levels according to the distance and transformed into numerical features; for the communication style preference, corresponding identifiers or feature vectors are set. For example, using "1" to represent the easy-to-understand communication style preference and "0" to represent other styles, thus generating the preference pattern feature information.

[0048] Based on the preset reply strategy library, the emotional tendency feature information, problem type feature information, and preference pattern feature information are processed to generate the reply style type influence feature and the reply style matching degree evaluation feature. The preset reply strategy library contains various reply strategies for different situations. Regarding Xiao Li's anxiety emotional tendency, the strategy library will retrieve strategies suitable for calming anxious patients, and these strategies will affect the adjustment of the reply style towards a calming direction, forming the reply style type influence feature. For example, in language expression, there may be a greater tendency to use gentle and comforting words and tones. For the problem type, the strategy library will match professional answer strategies for comprehensive symptom consultation questions. The matching degrees of these strategies with Xiao Li's problem type are different, and the system calculates the matching degree evaluation feature of the reply style with the current problem type. For example, some strategies may be more comprehensive and accurate in answering comprehensive symptom questions, and the matching degree is higher. Similarly, for the preference pattern, the strategy library will consider the distance preference and communication style preference, retrieve strategies that can meet Xiao Li's preferences, and calculate the matching degree of these strategies with his preferences, forming the reply style matching degree evaluation feature. If there is a guiding strategy specifically for patients who hope to seek medical treatment in a hospital close to home, and the communication style of this strategy meets Xiao Li's requirements, then its matching degree is high; otherwise, it is low.

[0049] Based on the reply style type influence features and the reply style matching degree evaluation features, a set of target reply strategies is screened out from the preset reply strategy library. According to the reply style type influence features and the reply style matching degree evaluation features, the system screens out a set of target reply strategies from the preset reply strategy library. In this set, there are reply strategies that not only consider Xiao Li's anxiety but also target his comprehensive symptom consultation questions and meet his preferences. For example, the screened strategies may include first soothing Xiao Li's anxiety in plain language, such as "Don't worry. Many people have similar symptoms. Let's analyze them step by step."; then, based on his symptoms and combined with medical knowledge, giving preliminary possible causes, like "Headache, fatigue, and chest tightness may be related to various factors, such as recent fatigue and cardiovascular problems."; and when recommending departments and doctors, giving priority to hospital departments close to home, such as "It is recommended that you can first go to the cardiovascular department of [hospital name] near your home. The doctors there are experienced and can diagnose you better." These strategies form the set of target reply strategies.

[0050] Based on the set of target reply strategies, the system marks and screens the patient's real-time emotional state information, the patient's question semantic information, and the patient's preference information to generate the screening result of key reply data. For the real-time emotional state information, the part related to anxiety and that needs to be soothed in the reply is marked; for the question semantic information, the key symptoms and the core of the question are marked; for the preference information, the key content related to distance preference and communication style preference is marked. Then, the system screens the information according to these marks and extracts the key information. For example, the key time points and manifestations with a higher degree of anxiety are extracted from the emotional state information; the key symptoms such as headache, fatigue, and chest tightness and the core demand of "inquiring about the disease situation" are extracted from the question semantic information; the specific need for a hospital department close to home and the requirement for a plain communication style are extracted from the preference information. These key information form the screening result of key reply data.

[0051] Integrate and weight the screening results of key response data to generate the quantified information on the target response style of the digital human. The quantified information on the target response style of the digital human is used to represent the response style and adaptation degree that the digital human should adopt for the patient's situation. The system integrates and weights the screening results of key response data. Different types of key information have different importance in determining the response style, so weights need to be assigned. Suppose that through a large number of experiments and data analyses, the weight of the emotional tendency feature is determined to be 0.3, the weight of the question type feature is 0.4, and the weight of the preference pattern feature is 0.3. For each piece of information in the screening results of key response data, calculate the contribution value according to its feature type and weight. For example, the contribution value of the key manifestation with a relatively high degree of anxiety in the emotional state is 0.2, the contribution value of the key symptoms and demands in the question semantics is 0.3, and the contribution value of the key needs in the preference information is 0.2. Through weighted summation, that is, (0.2×0.3 + 0.3×0.4 + 0.2×0.3 = 0.24), a comprehensive value is obtained. This comprehensive value combines the characteristics of the strategies in the target response strategy set to form the quantified information on the target response style of the digital human. This information indicates that for Xiao Li's situation, the digital human should adopt a response style that balances calming emotions, professionalism in answering questions, and meeting preferences, and through this quantified value, it can be evaluated that this response style has a relatively high degree of adaptation to Xiao Li and can better meet his needs.

[0052] S106. Process the real-time operation information of the hospital departments in the target area and the historical case record information of the patient based on the question and answer planning network to generate the resource allocation result information.

[0053] In one implementation, perform data extraction processing on the real-time operation information of the hospital departments in the target area and the historical case record information of the patient to generate the department registration volume information, doctor consultation efficiency information, waiting time information, patient's past diagnosis information, disease history information, and allergy history information. Xiao Li comes to the hospital and uses the digital human-assisted diagnosis system. The system accesses the hospital's HIS system to obtain the real-time operation information of the hospital departments in the target area and Xiao Li's historical case record information. For the real-time operation information of the hospital departments, the system extracts that the current registration volume of the internal medicine department is 50 people, and the registration volume of the cardiovascular department is 30 people; on average, each internal medicine doctor needs 15 minutes to consult one patient, and the average consultation time of cardiovascular department doctors is 20 minutes; the average waiting time of the current patients in the internal medicine department is 45 minutes, and the average waiting time in the cardiovascular department is 60 minutes. These respectively form the department registration volume information, doctor consultation efficiency information, and waiting time information. From Xiao Li's historical case record information, the system obtains that he had visited the hospital in 2022 due to a cold, and the diagnosis result was a common cold, and the treatment plan was to take cold medicine; it also learned that he had a history of pneumonia and was allergic to penicillin. These information constitute the patient's past diagnosis information, disease history information, and allergy history information.

[0054] The extracted information mentioned above is input into the Q&A planning network for preliminary feature transformation processing to obtain the transformed department operation feature information and patient historical diagnosis feature information. The system inputs the extracted department registration volume information, doctor consultation efficiency information, waiting time information, patient past diagnosis information, disease history information, and allergy history information into the Q&A planning network for preliminary feature transformation. For example, for the department registration volume information, the network may transform it into a feature value reflecting the busyness of the department. For instance, the registration volume of 50 people in the internal medicine department and 30 people in the cardiovascular department are respectively transformed into busyness values of 0.8 and 0.6 through specific formulas. For the doctor consultation efficiency information, it will be transformed into relevant feature values of the doctor's diagnosis speed. For example, the average consultation time of 15 minutes for internal medicine doctors and 20 minutes for cardiovascular department doctors are transformed into relative consultation speed indicators. The patient past diagnosis information, disease history information, and allergy history information will also be transformed into feature vectors convenient for subsequent analysis. For example, the history of cold and pneumonia is transformed into a vector representation related to the disease, reflecting the disease risk characteristics of Xiao Li. After these processes, the transformed department operation feature information and patient historical diagnosis feature information are obtained.

[0055] Based on the adaptation rule information corresponding to the system version update, the transformed department operation feature information and patient historical diagnosis feature information are subjected to standardization and integration processing, converting various types of information into a unified processable format to generate a standardized medical information set. With the update of the system version, when using the sentiment analysis model and medical knowledge to comprehensively process the quantitative features of physiological indicators and the quantitative features of facial microexpressions, there are various adaptation rule information, and these rules ensure the accuracy and reliability of the analysis results, as follows: Data association rules, including the following: Physiological and expression association: Establish an internal connection between physiological indicators and facial microexpressions. For example, it is stipulated that when the heart rate variability decreases and at the same time there are facial microexpressions such as frowning and anxious eyes, it may imply the existence of emotional problems. This association rule is summarized based on medical research and a large amount of clinical data, providing a basic framework for comprehensive analysis. Feature and emotion association: Define the corresponding relationship between various physiological and facial microexpression features and specific emotional states. For example, it is set that when the heart rate variability decreases, the blood oxygen saturation is slightly low, and the face shows an anxious expression, it is probably corresponding to the anxious emotion; while a stable heart rate and a relaxed facial expression are associated with a calm emotion. These rules are obtained through the analysis and induction of a large amount of experimental data and medical cases.

[0056] Weight allocation rules include the following: Importance assessment rule: Based on medical knowledge and data statistical analysis, determine the importance of different features in judging the emotional state. For example, facial microexpressions can more directly reflect the current emotion, so a higher weight is assigned to the facial microexpression feature; while physiological indicators can also reflect emotions, but relatively indirectly, and the weight is slightly lower. In the case of Xiao Li, the weight of the facial microexpression feature is 0.6, and the weight of the heart rate variability feature is 0.4, which is determined after weighing various factors. Dynamic adjustment rule: The weights are not fixed and will be dynamically adjusted according to different situations. For patients with different diseases or different age groups, the importance of each feature may be different. For example, for patients with cardiovascular diseases, the weights of physiological indicators such as heart rate variability may increase when judging emotions and physical conditions; for pediatric patients, facial microexpressions may be more able to accurately reflect emotions, and their weights will be adjusted accordingly.

[0057] Medical knowledge adaptation rules include the following: Disease-emotion association rule: Combine medical knowledge to link physiological and expression changes with specific diseases. If a patient has a history of cardiovascular disease, then the emotional problems reflected by a decrease in heart rate variability and abnormal blood oxygen saturation may be related to the onset or exacerbation of cardiovascular disease. This requires the system to not only consider the current physiological and expression features during analysis, but also combine the patient's medical history for comprehensive judgment. Symptom interpretation rule: Use medical knowledge to explain the meanings of various physiological and expression changes. For example, frowning and anxious eyes may be manifestations of emotional anxiety, but may also be related to physical discomfort symptoms such as headache. Based on these rules, the system comprehensively judges whether Xiao Li's anxiety is related to physical discomfort, so as to generate more accurate emotional tendency information.

[0058] Model training and update rules include the following: Data update rule: As new medical research results and a large amount of new data accumulate, the data used to train the sentiment analysis model is updated in a timely manner. For example, when new research discovers the association between a new physiological indicator and emotions, the relevant data is incorporated into the training set so that the model can learn more comprehensive and accurate relationships. Model optimization rule: According to new data and feedback in actual applications, the structure and parameters of the sentiment analysis model are optimized. If it is found in actual applications that the model has a low accuracy in judging certain specific emotional states, the structure of the model needs to be adjusted, such as increasing or adjusting the number of layers and nodes in the neural network, or retraining the model to improve its performance. Based on these rules, the system standardizes and integrates the converted department operation feature information and patient historical diagnosis feature information. For example, the new system version may stipulate that all department-related feature values should be uniformly mapped to the interval [0,1] for convenient subsequent unified calculation and analysis. For the previously obtained department busyness values of 0.8 and 0.6, as well as other department operation feature information, standardization processing is carried out according to the new rules so that they can be compared and analyzed on a unified scale. Similarly, the patient historical diagnosis feature information will also be unified in terms of format and numerical range according to the new rules. For example, the disease vector representation is adjusted according to the new standard, and various disease and allergy history information is converted into a unified coding format, finally generating a standardized medical information set.

[0059] Based on the question-and-answer planning network, importance assessment and screening processing are performed on the standardized medical information set to generate the importance weights of each piece of information in resource allocation decisions. The system uses the question-and-answer planning network to analyze the standardized medical information set again, comprehensively considering various factors, and evaluates and screens the importance of various types of information in the set in resource allocation decisions. For example, in the case of Xiao Li, considering that his current symptoms (headache, fatigue, chest tightness) may be related to cardiovascular diseases, and combining his pneumonia history, the importance of cardiovascular department-related information (such as the registration volume of the cardiovascular department, doctor reception efficiency, waiting time) in resource allocation decisions will be increased. Through complex calculations and analyses, the network assigns a weight of 0.2 to the department registration volume information, a weight of 0.3 to the doctor reception efficiency information, a weight of 0.2 to the waiting time information, a weight of 0.15 to the patient's past diagnosis information, a weight of 0.1 to the disease history information, and a weight of 0.05 to the allergy history information, thus generating the importance weights of each piece of information in resource allocation decisions.

[0060] Process the importance weights of each piece of information in the resource allocation decision to generate resource allocation result information, where the resource allocation result information is used to represent the department allocation information and doctor allocation information of the hospital department resource allocation result in the target area. The system further processes the generated importance weights to determine the resource allocation result. By calculating and integrating these weights, such as using weighted summation, etc., and comprehensively considering the resource status of the hospital department and Xiao Li's personal situation, the resource allocation result information is generated. Suppose after calculation, the system determines that Xiao Li is more suitable to visit the cardiovascular department, and recommends Doctor Wang in the cardiovascular department with higher reception efficiency and relatively shorter waiting time (based on the previous weight calculation, comprehensively considering factors such as the department registration volume, doctor reception efficiency, and waiting time). Here, the cardiovascular department is the department allocation information, and Doctor Wang is the doctor allocation information. These pieces of information constitute the resource allocation result information, providing specific medical advice for Xiao Li and reflecting the result of the hospital department resource allocation in the target area.

[0061] S107. Process the resource allocation result information based on the target response style information of the digital human to generate digital human response information.

[0062] In one implementation, comprehensive analysis and processing are performed based on the target response style quantization information of the digital human, the department allocation information and doctor allocation information in the resource allocation result, and the real-time emotional state information of the patient to generate an interactive strategy correlation evaluation value. When Xiao Li interacts with the digital human, the system has obtained various pieces of information about him. The target response style quantization information of the digital human shows that for Xiao Li's situation, the digital human should adopt a response style that achieves a balance among calming emotions, answering professionally, and satisfying preferences, and its quantization value indicates a relatively high degree of adaptation of this style (assuming the quantization value is 0.8). In the resource allocation result information, the department allocation information is the cardiovascular department, and the doctor allocation information is Doctor Wang, which is obtained by comprehensively considering Xiao Li's symptoms, medical history, and the real-time operation situation of the hospital department. The real-time emotional state information of Xiao Li shows that he is in a moderately anxious state.

[0063] The system comprehensively analyzes and processes this information, considering the quantitative value of the target response style, the degree of fit between the department and doctor allocation information and Xiao Li's needs, and the impact of his anxiety on the interaction. For example, the system performs a weighted calculation on the quantitative value of the target response style, department allocation information (assuming the professional score of the cardiovascular department in dealing with similar symptoms is 0.9), doctor allocation information (assuming the ability score of Doctor Wang in dealing with cardiovascular diseases and calming patients is 0.85), and emotional state information (assuming the impact factor corresponding to the anxiety level is 0.7). Assuming the weights are 0.3, 0.3, 0.3, and 0.1 respectively, then the interaction strategy correlation evaluation value = 0.8×0.3 + 0.9×0.3 + 0.85×0.3 + 0.7×0.1 = 0.825. This evaluation value reflects the overall correlation degree between the current resource allocation and response style and Xiao Li's situation, and is used to adjust the response strategy subsequently.

[0064] Based on the interaction strategy correlation evaluation value, an optimization and adjustment process is carried out on the response strategy vector within the preset response content template library to generate a response content adjustment deviation vector. Based on the obtained interaction strategy correlation evaluation value of 0.825, the system optimizes and adjusts the response strategy vector within the preset response content template library. The preset response content template library contains various response strategy vectors for different situations, and each vector represents a possible response method, covering aspects such as diagnostic suggestions, department and doctor recommendation information, and response tone.

[0065] The system will compare the current evaluation value with the ideal interaction strategy evaluation criterion. Assuming the ideal interaction strategy evaluation criterion value is 0.9 (representing a state of perfect matching of the patient's needs), then the gap between the two is the direction and degree of adjustment required. Through calculation, the response content adjustment deviation vector is obtained. For example, in terms of diagnostic suggestions, it may be necessary to increase the detailed explanation of the possibility of cardiovascular diseases, and the adjustment amplitude is assumed to be +0.1; in terms of department and doctor recommendation information, it is necessary to highlight Doctor Wang's experience in dealing with patients with similar symptoms, and the adjustment amplitude is assumed to be +0.05; in terms of the emotional adaptation of the response content, it is necessary to further strengthen the calming tone, and the adjustment amplitude is assumed to be -0.02 (the negative sign indicates that it is necessary to strengthen on the basis of the original calming degree). These adjustment amplitudes form the response content adjustment deviation vector, which is used to guide the generation of a more accurate response subsequently.

[0066] The deviation vector of the reply content is analyzed and converted to generate the digital human reply information, where the digital human reply information is used to represent the diagnostic suggestions given by the digital human to the patient, the department and doctor recommendation information, and the emotional adaptation quantification index of the reply content. The system analyzes and converts the deviation vector of the reply content to generate the final digital human reply information. In the part of diagnostic suggestions, combining Xiao Li's symptoms (headache, fatigue, chest tightness), medical history (pneumonia history), and current medical knowledge, the digital human replies: "Xiao Li, these symptoms of yours, like headache, fatigue, and chest tightness, may be related to some problems in the cardiovascular system. Considering your pneumonia history, the risk of cardiovascular disease may be relatively higher. However, don't worry too much for now. This is just a preliminary judgment." In terms of department and doctor recommendation information, the digital human will say: "It is recommended that you go to the cardiovascular medicine department of our hospital. Doctor Wang has rich experience in this area. He has treated many patients with similar symptoms and can diagnose and treat you more accurately. Currently, the waiting time for Doctor Wang is relatively short, allowing you to receive medical treatment as soon as possible." In terms of the emotional adaptation of the reply content, the digital human will use a more gentle and soothing tone, and the expression will also cooperate, such as showing a concerned look, making the reply more amiable, so that the emotional adaptation quantification index of the reply content meets the requirements. Through these adjustments, the entire reply content more accurately meets Xiao Li's needs and also reflects the personalization and intelligence of the digital human-assisted diagnosis system.

[0067] The server obtains multi-source information of the patient, such as question-and-answer audio, facial video, physiological state, preferences, and historical case information, as well as the real-time operation data of hospital departments. Then, each type of information is processed separately. For example, the question-and-answer audio is converted into question semantics and voice features, and the facial video and physiological state information are converted into corresponding features to judge the emotional state. Then, based on these processing results, a resource allocation result is generated through a question-and-answer planning network, and combined with the target reply style to generate the digital human reply information, providing diagnostic suggestions, department and doctor recommendations for the patient. This technology can improve the diagnostic accuracy and patient experience, reduce the waiting time, optimize resource utilization, has significant technical advantages and application value, and brings a new development direction to the medical diagnosis field.

[0068] In one implementation, as Figure 2 shown, the present application also provides a digital human-assisted diagnosis device, including:

[0069] An acquisition module 201, configured to acquire the question-and-answer audio information of the patient, the real-time facial video information, the physiological state information of the patient, the preference information of the patient, the historical case record information of the patient, and the real-time operation information of the hospital departments in the target area;

[0070] The processing module 202 is configured to process the Q&A audio information of the patient to generate the question semantic information and voice feature information of the patient; process the real-time facial video information and the physiological state information of the patient to generate the facial micro-expression feature information and physiological index feature information; process the voice feature information, facial micro-expression feature information and physiological index feature information based on the multi-modal fusion network to generate the real-time emotional state information of the patient; process the real-time emotional state information, question semantic information and preference information of the patient to generate the target response style information of the digital human; process the real-time operation information of the target regional hospital department and the historical case record information of the patient based on the Q&A planning network to generate the resource allocation result information; and process the resource allocation result information based on the target response style information of the digital human to generate the digital human response information.

[0071] Each embodiment in the present application is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the digital human-assisted diagnosis method, electronic device, electronic equipment, and readable storage medium, since they are basically similar to the digital human-assisted diagnosis method embodiment described above, the description is relatively simple, and the relevant parts can refer to the partial description of the digital human-assisted diagnosis method embodiment described above.

Claims

1. A digital human-assisted diagnosis method, characterized in that: include: Obtain the patient's audio information of questions and answers, real-time facial video information, the patient's physiological status information, the patient's preference information, the patient's historical case record information, and the real-time operation information of the hospital departments in the target area; Processing the patient's question and answer audio information to generate the patient's question semantic information and sound feature information; Processing real-time facial video information and the patient's physiological status information to generate facial micro-expression feature information and physiological index feature information; Based on the multimodal fusion network, the sound feature information, facial micro-expression feature information and physiological index feature information are processed to generate the patient's real-time emotional state information; Process the patient's real-time emotional state information, the patient's question semantic information, and the patient's preference information to generate the digital person's target response style information; Based on the question-answering planning network, the real-time operation information of the hospital departments in the target area and the historical case record information of the patients are processed to generate resource allocation result information; The resource allocation result information is processed based on the target response style information of the digital human to generate the digital human response information.

2. The method according to claim 1, characterized in that Process the patient's question and answer audio information to generate the patient's question semantic information and sound feature information, including: Perform sampling and noise reduction processing on the patient's question and answer audio information to generate pre-processed question and answer audio; Perform speech-to-text processing on the pre-processed question-answer audio to generate initial text information; Perform part-of-speech tagging and syntactic analysis on the initial text information to generate grammatical structure features; Perform speech feature extraction on the pre-processed question and answer audio to generate intonation features, speech speed features, and volume features; Based on the medical knowledge graph and semantic understanding model, the grammatical structure features are processed to generate the patient's problem semantic information; The intonation features, speech speed features and volume features are normalized and feature selected to generate sound feature information.

3. The method according to claim 1, characterized in that Process the real-time facial video information and the patient's physiological status information to generate facial micro-expression feature information and physiological indicator feature information, including: Perform image preprocessing on real-time facial video information to generate preprocessed facial video; Perform signal filtering on the patient's physiological state information to generate target physiological signal data; Perform key point detection on the pre-processed facial video to generate facial key point coordinate features; Perform expression classification on the preprocessed facial video to generate expression category features; Perform feature extraction processing on target physiological signal data to generate heart rate variability features and blood oxygen saturation features; Quantitatively analyze and process the characteristics of heart rate variability and blood oxygen saturation to generate quantitative characteristics of physiological indicators; The expression category features and facial key point coordinate features are quantitatively analyzed and processed to generate quantitative features of facial micro-expressions.

4. The method according to claim 1, characterized in that Based on the multimodal fusion network, the sound feature information, facial micro-expression feature information and physiological index feature information are processed to generate the patient's real-time emotional state information, including: Normalize and standardize the sound feature information, facial micro-expression feature information and physiological index feature information to generate pre-processed multi-modal feature information; Based on the sentiment classification model library, feature matching and analysis are performed on the preprocessed multimodal feature information to generate sentiment category prediction information and sentiment intensity prediction information; Based on the emotion category prediction information and the emotion intensity prediction information, key data in the preprocessed multimodal feature information is located and extracted to generate key emotion feature data; The key emotional feature data are fused and weighted to generate the patient's real-time emotional state information, wherein the patient's real-time emotional state information is used to characterize the patient's current emotional state and emotional intensity.

5. The method according to claim 1, characterized in that The patient's real-time emotional state information, the patient's question semantic information, and the patient's preference information are processed to generate the digital human's target response style information, including: Extract and classify the patient's real-time emotional state information, the patient's question semantic information, and the patient's preference information to generate emotional tendency feature information, question type feature information, and preference pattern feature information; Based on the preset answer strategy library, the emotional tendency feature information, question type feature information, and preference pattern feature information are processed to generate answer style type influence features and answer style matching evaluation features; Based on the response style type influence characteristics and response style matching evaluation characteristics, a target response strategy set is selected from the preset response strategy library; Based on the target response strategy set, the patient's real-time emotional state information, the patient's question semantic information and the patient's preference information are marked and screened to generate key response data screening results; The key response data screening results are integrated and weighted to generate quantitative information on the digital human's target response style, where the digital human's target response style quantitative information is used to characterize the response style and degree of adaptation that the digital human should adopt according to the patient's condition.

6. The method according to claim 1, characterized in that Based on the question-answer planning network, the real-time operation information of the hospital departments in the target area and the historical case record information of the patients are processed to generate resource allocation result information, including: Data is extracted and processed from the real-time operation information of hospital departments in the target area and the historical case records of patients to generate information on department registration volume, doctor reception efficiency, waiting time, patients' past diagnosis information, disease history information, and allergy history information; The extracted information is input into the question-answer planning network for preliminary feature conversion processing to obtain converted department operation feature information and patient historical diagnosis feature information; Based on the adaptation rule information corresponding to the system version update, the converted department operation characteristic information and patient historical diagnosis characteristic information are standardized and integrated, and various types of information are converted into a unified processable format to generate a standardized medical information set; Based on the question-answering planning network, the importance of standardized medical information sets is evaluated and screened to generate the importance weight of each information in resource allocation decisions; The importance weights of each information in the resource allocation decision are processed to generate resource allocation result information, wherein the resource allocation result information is used to characterize the department allocation information and doctor allocation information of the resource allocation results of the target regional hospital departments.

7. The method according to claim 6, characterized in that The resource allocation result information is processed based on the target response style information of the digital human to generate the digital human response information, including: Based on the quantitative information of the digital human's target response style, the department allocation information and doctor allocation information in the resource allocation results, and the patient's real-time emotional state information, a comprehensive analysis is performed to generate an interaction strategy association evaluation value; Based on the interaction strategy association evaluation value, the reply strategy vector in the preset reply content template library is optimized and adjusted to generate a reply content adjustment deviation vector; The response content adjustment deviation vector is parsed and converted to generate digital human response information, where the digital human response information is used to characterize the diagnostic suggestions given by the digital human to the patient, the department and doctor recommendation information, and the emotional adaptation quantitative indicators of the response content.

8. A digital human-assisted diagnosis device, characterized in that: The device comprises: An acquisition module is used to acquire the patient's question and answer audio information, real-time facial video information, the patient's physiological state information, the patient's preference information, the patient's historical case record information, and the real-time operation information of the hospital departments in the target area; The processing module is used to process the patient's question and answer audio information to generate the patient's question semantic information and sound feature information; process the real-time facial video information and the patient's physiological state information to generate facial micro-expression feature information and physiological indicator feature information; process the sound feature information, facial micro-expression feature information and physiological indicator feature information based on the multimodal fusion network to generate the patient's real-time emotional state information; process the patient's real-time emotional state information, the patient's question semantic information and the patient's preference information to generate the digital human's target response style information; process the real-time operation information of the hospital department in the target area and the patient's historical case record information based on the question and answer planning network to generate resource allocation result information; process the resource allocation result information based on the digital human's target response style information to generate the digital human's response information.

9. An electronic device, characterized in that: include: a first processor; and a memory for storing executable instructions of the first processor; Wherein, the first processor is configured to execute the digital human assisted diagnosis method described in any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the second processor, the digital human assisted diagnosis method described in any one of claims 1 to 7 is implemented.

Images