A service robot for identifying traditional Chinese medicine constitution based on personification interaction and an identification method thereof

Abstract

The application discloses a traditional Chinese medicine constitution identification service robot based on personification interaction and an identification method thereof, and the robot comprises the following modules: a digital human interaction module which provides a virtual traditional Chinese medicine image and realizes voice interaction; a multi-modal data acquisition module which acquires multi-modal data; a multi-modal feature extraction submodule which extracts discriminative feature vectors of each mode; a multi-round cross-modal communication fusion submodule which performs cross-modal information interaction on the feature vectors of each mode, outputs a probability distribution of a traditional Chinese medicine constitution type, and selects a candidate constitution set; a dynamic inquiry decision submodule which generates inquiry questions according to the candidate constitution set; a constitution reasoning submodule which integrates the candidate constitution set, the inquiry questions and voice answer information into structured multi-modal evidence texts, and outputs a final constitution determination result and confidence instructions; and a report generation module which generates a structured health report based on the constitution determination result and a preset report template. The application realizes global cross-modal comprehensive analysis, and can improve the accuracy of constitution identification.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, specifically to a TCM constitution identification service robot based on anthropomorphic interaction and its identification method. Background Technology

[0002] Traditional Chinese medicine diagnosis relies on the four diagnostic methods of observation, auscultation and olfaction, inquiry, and palpation, requiring experienced TCM doctors to make comprehensive judgments. This approach suffers from problems such as uneven distribution of medical resources and time-consuming and labor-intensive treatment. With the development of artificial intelligence technology, intelligent TCM diagnostic equipment has emerged, simulating the TCM diagnostic process through image recognition, signal processing, and other technologies.

[0003] However, existing intelligent TCM diagnostic systems have the following shortcomings:

[0004] First, the interaction methods are simple and rigid, mostly using graphical interfaces or simple voice prompts, lacking a human-like interactive experience, and the user experience is not natural enough.

[0005] Secondly, the diagnostic process is fragmented, with each step—face consultation, tongue diagnosis, pulse diagnosis, and questioning—being relatively independent, lacking a unified interactive entity to guide users through the entire process.

[0006] Third, it lacks real-time capability; traditional text-based dialogues or pre-recorded voice recordings cannot achieve smooth and natural real-time interaction. Summary of the Invention

[0007] In order to overcome the defects and shortcomings of existing technologies, this invention provides a TCM constitution identification service robot based on anthropomorphic interaction and its identification method. This invention uses a digital human interaction module as an interaction medium and integrates speech recognition, large-scale language model, speech synthesis and face animation generation technologies to achieve a natural and smooth TCM four diagnostic methods interactive experience.

[0008] To achieve the above objectives, the present invention adopts the following technical solution:

[0009] This invention provides a TCM constitution identification service robot based on anthropomorphic interaction, comprising: a digital human interaction module, a multimodal data acquisition module, a constitution identification module, and a report generation module;

[0010] The digital human interaction module is used to provide a virtual TCM avatar and enable voice interaction;

[0011] The multimodal data acquisition module is used to acquire multimodal data, including facial images, tongue images, pulse signals, and voice response information;

[0012] The constitution identification module is used for constitution analysis and reasoning based on multimodal data, including: a multimodal feature extraction submodule, a multi-round cross-modal communication fusion submodule, a dynamic consultation decision submodule, and a constitution reasoning submodule;

[0013] The multimodal feature extraction submodule is used to extract discriminative feature vectors for each modality from facial images, tongue images, and pulse signals;

[0014] The multi-round cross-modal communication fusion submodule is used to exchange feature vectors from three modalities—face, tongue, and pulse—through stacked multi-round SRMC modules to output probability distributions of various TCM constitution types and select a candidate constitution set.

[0015] The dynamic consultation decision submodule is used to generate consultation questions based on the candidate constitution set;

[0016] The constitution reasoning submodule is used to integrate the candidate constitution set, consultation questions and voice response information into structured multimodal evidence text, and output the final constitution judgment result and confidence statement based on the large language model;

[0017] The report generation module is used to generate structured health reports based on the physical fitness assessment results and preset report templates.

[0018] As a preferred technical solution, the digital human interaction module includes: a speech recognition submodule, a dialogue generation submodule, a speech synthesis submodule, and a face animation generation submodule;

[0019] The speech recognition submodule is used to collect user speech and convert it into text data;

[0020] The dialogue generation submodule is used to generate response text based on user input text;

[0021] The speech synthesis submodule is used to convert response text into speech signals;

[0022] The face animation generation submodule is used to generate digital facial animation videos that are synchronized with the speech based on the speech signal.

[0023] As a preferred technical solution, the face animation generation submodule includes: an audio feature extractor, a lip shape prediction network, an expression generation network, and a video rendering engine;

[0024] Audio feature extractors are used to extract acoustic features from speech signals;

[0025] The lip shape prediction network uses a Transformer architecture to predict lip shape parameter sequences based on acoustic features;

[0026] Facial expression generation networks are used to generate facial expression parameters that match the emotions conveyed in speech.

[0027] The video rendering engine is used to render face video frames based on pre-trained digital human models, according to lip shape parameter sequences and facial expression parameters.

[0028] As a preferred technical solution, the multimodal data acquisition module includes an image acquisition unit, a pulse acquisition unit, and a voice acquisition unit;

[0029] The image acquisition unit acquires facial and tongue images;

[0030] The pulse acquisition unit is equipped with a piezoelectric pulse sensor array for acquiring pulse waveform timing data;

[0031] The voice acquisition unit is equipped with a microphone array for acquiring voice response information.

[0032] As a preferred technical solution, the image acquisition unit acquires multiple frames of video images, performs detection and key point localization on each frame of video images, performs affine alignment correction based on the line connecting the eyes, expands the bounding box according to a set ratio and then crops to obtain a facial image, and selects the frame with the highest clarity score as the final facial image.

[0033] The image acquisition unit performs pixel-level semantic segmentation on the acquired image based on the tongue segmentation network, extracts the tongue region, performs LAB color space conversion on the tongue image and CLAHE adaptive histogram equalization on the L brightness channel to obtain the final tongue image.

[0034] After acquiring pulse waveform time-series data, the pulse acquisition unit performs baseline drift correction, bandpass filtering, and peak normalization preprocessing on the original pulse waveform to obtain the final pulse waveform time-series data.

[0035] As a preferred technical solution, the multimodal feature extraction submodule includes a facial feature encoder, a tongue feature encoder, and a pulse feature encoder;

[0036] The facial feature encoder and the tongue feature encoder each use VGG-19 convolutional neural network as the backbone network. Each retains the original five groups of VGG-19 convolutional pooling blocks. After the feature map is output from the fifth group of convolutional pooling blocks, a global average pooling layer and a fully connected mapping layer are connected to compress and map each feature into a multi-dimensional semantic feature vector.

[0037] The pulse feature encoder uses a one-dimensional residual convolutional neural network. It takes the preprocessed standardized pulse waveform sequence as input and extracts multi-scale temporal features through four sets of one-dimensional residual convolutional blocks. Each set of one-dimensional residual convolutional blocks contains two layers of one-dimensional convolution, batch normalization and residual skip connections. After the output of the last set of residual blocks, a global average pooling layer and a fully connected mapping layer are connected to compress and map the temporal features of the pulse waveform into a multi-dimensional semantic feature vector.

[0038] As a preferred technical solution, the multi-round cross-modal communication fusion submodule is used to exchange feature vectors from three modalities—face, tongue, and pulse—through stacked multi-round SRMC modules for cross-modal information interaction, specifically including:

[0039] For each modality's feature vector, feature decoupling is performed, mapping it to modality-specific features and modality-shared features respectively;

[0040] Minimize the squared Frobenius norm of the inner product matrix of mode-specific features and mode-shared features within each mode;

[0041] A cross-modal similarity alignment loss is constructed and optimized by averaging the cosine dissimilarity of shared features among all modal pairs;

[0042] For each modality, unique features from the other two modalities are aggregated as complementary information;

[0043] The complementary information is concatenated with the original features of the corresponding modality and then input into the residual fusion network. The updated features are obtained through residual connections.

[0044] After multiple rounds of cross-modal information interaction, the final unique features and shared features of the three modalities are aggregated into a multimodal fusion representation.

[0045] The multimodal fusion representation is processed by a multilayer perceptron classifier and a softmax activation function to output the probability distribution of various TCM constitution types. The top-k constitutions are selected as the candidate constitution set.

[0046] As a preferred technical solution, the unique features from the other two modes are aggregated for each mode as complementary information, represented as follows:

[0047] ;

[0048] in, Represents complementary information vectors, This indicates a splicing operation. This represents the mode-specific feature corresponding to mode i. Represents facial modality. Indicates tongue modality, This indicates the pulse mode.

[0049] As a preferred technical solution, the dynamic consultation decision submodule generates consultation questions based on the candidate constitution set, specifically including:

[0050] The dynamic consultation decision-making submodule constructs structured prompts containing key points for identifying candidate constitutions based on the candidate constitution set and the probability values ​​of each constitution. The large language model uses information gain as the screening criterion to select a subset of consultation questions.

[0051] This invention also provides a method for identifying TCM constitution based on anthropomorphic interaction using a TCM constitution identification service robot, comprising the following steps:

[0052] Based on the digital human interaction module, a virtual TCM avatar is provided and voice interaction is enabled to guide users into the testing process;

[0053] Multimodal data is collected using a multimodal data acquisition module, including facial images, tongue images, pulse signals, and voice response information.

[0054] The multimodal feature extraction submodule extracts discriminative feature vectors for each modality from facial images, tongue images, and pulse signals.

[0055] Based on the multi-round cross-modal communication fusion submodule, the feature vectors of the three modalities of face, tongue and pulse are used to perform cross-modal information interaction through stacked multi-round SRMC modules, outputting the probability distribution of various TCM constitution types and selecting a candidate constitution set;

[0056] Based on the dynamic consultation decision submodule, consultation questions are generated according to the candidate constitution set;

[0057] The constitution reasoning submodule integrates candidate constitution sets, consultation questions, and voice response information into structured multimodal evidence text, and outputs the final constitution determination result and confidence statement based on a large language model;

[0058] Based on the report generation module, a structured health report is generated according to the physical fitness assessment results and the preset report template.

[0059] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0060] (1) This invention uses a digital human interaction module as an interaction medium to provide a humanized, friendly and natural TCM diagnosis interaction experience, breaking through the limitations of the rigid interaction of traditional intelligent diagnosis systems. It guides users to complete the four diagnostic processes of face, tongue, pulse and questioning throughout the process. The diagnosis process is coherent and unified, avoiding the problem of process fragmentation.

[0061] This invention employs streaming speech-driven face video generation technology to achieve real-time synchronization of speech and facial expressions;

[0062] This invention employs multi-round cross-modal communication fusion technology to decouple the features of facial diagnosis, tongue diagnosis, and pulse diagnosis into modality-specific features and modality-shared features. Effective decoupling is achieved through orthogonal constraints and cross-modal similarity alignment loss. Complementary discriminative information from other modalities is injected round by round through multi-round residual fusion, progressively enhancing the representational capabilities of each modality. Combined with a large-scale language model, dynamic consultation decision-making and constitution reasoning are performed to achieve global cross-modal comprehensive analysis, significantly improving the accuracy of constitution identification.

[0063] This invention can be deployed in smart terminals, health management platforms, and telemedicine scenarios, and has strong scalability. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the overall architecture of the TCM constitution identification service robot based on anthropomorphic interaction according to the present invention.

[0065] Figure 2 This is a schematic diagram of the overall architecture of the digital human interaction module of the present invention;

[0066] Figure 3 This is a flowchart illustrating the identification method of the TCM constitution identification service robot based on anthropomorphic interaction according to the present invention. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0068] Example 1

[0069] like Figure 1 As shown, this embodiment provides a TCM constitution identification service robot based on anthropomorphic interaction, including: a digital human interaction module, a multimodal data acquisition module, a constitution identification module, and a report generation module;

[0070] like Figure 2 As shown, the digital human interaction module is used to provide a virtual TCM avatar and realize voice interaction, including: a speech recognition submodule, a dialogue generation submodule, a speech synthesis submodule, and a face animation generation submodule;

[0071] The speech recognition submodule uses the Whisper end-to-end speech recognition model to collect user speech in real time and convert it into text data. This submodule is noise robust and can accurately recognize user speech in everyday environments.

[0072] The dialogue generation submodule is equipped with a large language model (such as GPT-4, Claude, or open-source large model) to generate response text based on user input text. Specifically, it can generate natural language responses that conform to the TCM diagnosis scenario based on preset TCM diagnosis prompt word templates and user input. This submodule supports multi-turn dialogue management, can maintain dialogue context, and achieve coherent consultation interaction.

[0073] The speech synthesis submodule uses the CosyVoice neural network speech synthesis model to convert response text into speech signals. It supports configuration with a gentle and professional speech style with TCM characteristics and supports speech rate and pitch adjustment.

[0074] The face animation generation submodule is used to generate digital human face animation videos synchronized with speech based on the speech signal in a streaming manner. This submodule receives speech features as input, first generates a sparse representation sequence of face feature points through a facial landmark prediction network, and then combines preset image and video materials to generate and output a synthetic video through a rendering engine.

[0075] Specifically, the face animation generation submodule adopts a streaming processing architecture, including: an audio feature extractor, a lip shape prediction network, an expression generation network, and a video rendering engine;

[0076] Audio feature extractors extract acoustic features such as Mel spectrum from speech signals;

[0077] The lip shape prediction network uses a Transformer architecture to predict lip shape parameter sequences based on acoustic features;

[0078] Facial expression parameters generated by an expression generation network and matched with speech emotion;

[0079] The video rendering engine is based on a pre-trained digital human model and renders facial video frames in real time according to lip shape and expression parameters, achieving streaming output while generating and playing, with a latency of less than 2 seconds.

[0080] In this embodiment, the multimodal data acquisition module is used to acquire images of the user's face, tongue, and pulse signals, and includes: an image acquisition unit, a pulse acquisition unit, and a voice acquisition unit;

[0081] The image acquisition unit is equipped with a high-definition camera to acquire facial and tongue images of the user under the guidance of the digital human's voice. Specifically, when the digital human prompts the user to look directly at the camera, the image acquisition unit continuously acquires multiple frames of video images. A face detection algorithm based on MTCNN (Multi-task Cascaded Convolutional Neural Network) is used to detect and locate key points in each frame. Affine alignment correction is performed based on the line connecting the eyes. After expanding the bounding box by a fixed ratio, the image is cropped into a standard 224×224 pixel facial image, and the frame with the highest clarity score is selected as the final facial image input. When the digital human prompts the user to open their mouth and stick out their tongue, a tongue segmentation network based on the U-Net structure is used to perform pixel-level semantic segmentation on the acquired image, accurately extract the tongue region and remove oral background interference. Then, the tongue image is converted to LAB color space and CLAHE adaptive histogram equalization is applied to the L brightness channel to eliminate the interference of ambient light changes on tongue color feature extraction and ensure the consistency and reliability of tongue image color.

[0082] The pulse acquisition unit is equipped with a pressure sensor array to collect the user's pulse waveform time-series data. Specifically, the pulse acquisition unit uses a piezoelectric pulse sensor. When the digital human prompts the user to place their wrist in the sensor groove and keep it relaxed, the unit synchronously collects the radial artery pulse waveform time-series data of each channel at a sampling frequency of not less than 500Hz for a duration of not less than 20 seconds. After the acquisition is completed, the original pulse waveform is preprocessed with baseline drift correction, bandpass filtering (0.5Hz~40Hz), and peak normalization to eliminate respiratory motion and high-frequency noise interference, and outputs a high-quality standardized pulse waveform sequence for use by the subsequent multimodal feature extraction submodule.

[0083] The voice acquisition unit is equipped with a microphone array for acquiring user voice responses.

[0084] In this embodiment, the constitution identification module is used for constitution analysis and reasoning based on multimodal data, including: a multimodal feature extraction submodule, a multi-round cross-modal communication fusion submodule, a dynamic consultation decision submodule, and a constitution reasoning submodule;

[0085] The multimodal feature extraction submodule is used to extract discriminative feature vectors for each modality from facial images, tongue images, and pulse waveforms, respectively.

[0086] Specifically, the facial feature encoder and the tongue feature encoder each use a VGG-19 convolutional neural network as their backbone network. The parameters of the two encoders are independent and their weights are not shared. Each encoder retains the original five sets of VGG-19 convolutional pooling blocks. After the output feature map (14×14×512) from the fifth set of convolutional pooling blocks, a global average pooling layer (GAP) and a fully connected mapping layer are applied to compress and map the features from each path into d-dimensional semantic feature vectors. and ( =256);

[0087] The pulse feature encoder employs a one-dimensional residual convolutional neural network (1D-ResNet). It takes a preprocessed, standardized pulse waveform sequence as input and sequentially passes it through four sets of one-dimensional residual convolutional blocks to extract multi-scale temporal features. Each set of one-dimensional residual convolutional blocks contains two layers of one-dimensional convolution, batch normalization, and residual skip connections. The kernel sizes are 7, 5, 5, and 3, respectively, and the number of channels are 64, 128, 256, and 512, respectively. After the output of the last set of residual blocks, a global average pooling layer and a fully connected mapping layer are applied to compress and map the temporal features of the pulse waveform. 3D semantic feature vector ;

[0088] This embodiment employs an independent weighted three-way structure instead of a weight-sharing structure because facial images, tongue images, and pulse waveforms differ significantly in data morphology, feature distribution, and modes of interest. Independent encoders avoid mutual interference in feature learning between modalities, allowing each network to fully optimize the discriminative feature representation of its corresponding modality. 3D eigenvectors are denoted as The input is sent to the multi-round cross-modal communication fusion submodule for cross-modal information interaction;

[0089] The multi-round cross-modal communication fusion submodule is used for multi-round cross-modal information interaction of feature vectors from three modalities: facial diagnosis, tongue diagnosis, and pulse diagnosis. This allows each modality to absorb complementary discriminative information from other modalities in each round, achieving a progressive enhancement of representational ability and ultimately outputting a candidate constitution set. The basic unit of this submodule is the Single-Round Modality Communication (SRMC) module, which is stacked through K rounds of SRMC modules (…). Preferred To achieve multi-round interaction, in each round of the SRMC module, perform the following operations:

[0090] First, for the feature vector of each mode m... Feature decoupling is performed by using two independent multilayer perceptron (MLP) networks. Mapped to modality-specific features respectively Features shared with modality Both are dimensional vector, where the modality-specific feature extraction network The modality-sharing feature extraction network is responsible for extracting discriminative information unique to this modality. Responsible for extracting common semantic information across modalities, and applying orthogonal constraint loss to ensure effective decoupling between unique features and shared features. By minimizing the intramodal and The squared Frobenius norm of the inner product matrix forces the unique features and shared features of the same modality to be orthogonal in the feature space, i.e.:

[0091] ;

[0092] This avoids information redundancy between the two types of features; at the same time, to promote semantic alignment of cross-modal common information, a cross-modal similarity alignment loss is applied. The optimization is achieved by averaging the cosine dissimilarity of shared features among all modal pairs, i.e.:

[0093] ;

[0094] in For all modal pairs, the shared features of the three modalities tend to be consistent in the semantic space, forming a cross-modal consensus representation.

[0095] After completing feature decoupling, for each mode The unique features from the other two modalities are aggregated as complementary information, that is:

[0096] ;

[0097] The discriminative cues unique to each of the other modalities are concatenated to form the complementary information vector of that modality. ; then complementary information With the original features of this mode After splicing, the data is fed into the residual fusion network. The refined features after this round of updates are obtained through residual connections, namely:

[0098] ;

[0099] Thus, while retaining the original discriminative information of each modality, complementary physical cues provided by other modalities are selectively absorbed; this process constitutes one round of SRMC operation. In multi-round cross-modal communication, the first... wheel( The input to ) is the output of the previous round. (in (as initial features), the first The update formula for the wheel is:

[0100] ;

[0101] in For the first The complementary information obtained from the modality-specific features of other modules is aggregated in each round; each round of the SRMC module has independent learnable parameters, including its own modality-specific feature extraction network. Modality sharing feature extraction network and residual fusion network After K rounds of communication, each modal feature undergoes multiple rounds of complementary information injection and progressive refinement, resulting in a progressively enhanced representational capability. After the final round of communication, the refined three modal features... , , Perform the specific-shared feature decoupling again to obtain the final modality-specific features. Features shared with modality The final unique features of the three modalities are aggregated with the shared features to form a multimodal fusion representation. Specifically, the shared features are taken as the average of the shared features of the three modalities.

[0102] ;

[0103] To capture global cross-modal consensus information, the final fused representation is as follows:

[0104] ;

[0105] Dimensions ,Will Input to a three-layer fully connected layer ( → → →9-dimensional) and Dropout regularization layer (dropout rate) The multilayer perceptron classifier is used to output the probability distribution of 9 TCM constitution types through the Softmax activation function; finally, the top 3 constitutions with the highest probabilities (Top-3) are selected as the candidate constitution set and input into the dynamic consultation decision submodule for further identification.

[0106] The dynamic consultation decision-making submodule is equipped with a large language model to dynamically generate personalized, targeted consultation questions based on the candidate constitution set. Specifically, the dynamic consultation decision-making submodule receives the Top-3 candidate constitution set and the probability value of each constitution from the multi-round cross-modal communication fusion submodule as input, constructs structured prompt words containing key points for candidate constitution identification, and drives the large language model to select 5 consultation question subsets from a preset library of 60 standard consultation questions, using information gain as the selection criterion. The information gain criterion requires that the selected question set has the highest discriminative power among the candidate constitutions, that is, the different answers from the user to the question can most effectively reduce the uncertainty of the candidate constitution set. After the selected question subsets are dynamically sorted, the digital human asks the user questions one by one through voice interaction.

[0107] The constitution inference submodule is equipped with a large-scale language model for comprehensive constitution determination based on data from the four diagnostic methods of face diagnosis, tongue diagnosis, pulse diagnosis, and questioning. Specifically, the constitution inference submodule integrates the Top-3 candidate constitutions and their probability values, along with 5 sets of questioning questions and user voice response recognition text, from the multi-round cross-modal communication fusion submodule into structured multimodal evidence text. This text is then input into the large-scale language model using a one-shot inference approach. Based on its built-in TCM four diagnostic methods integration logic, the large-scale language model performs weighted fusion and logical reasoning on the evidence from each dimension of observation (facial complexion, tongue appearance), pulse diagnosis, and questioning, simulating the diagnostic thinking process of a TCM expert using the four diagnostic methods, and outputs the final constitution determination result and confidence statement.

[0108] In this embodiment, the report generation module is used to generate a constitution diagnosis report and conditioning plan. Specifically, the report generation module adopts a large language model, based on the constitution assessment results and preset report templates, and generates a structured health report containing constitution performance, emotional regulation, dietary conditioning, daily life regulation, exercise health care and acupoint health care through a single prompt process, and the report is broadcast by a digital human.

[0109] Specifically, after the user begins the physical examination, the digital human guides the user through the testing process via voice. Then, the user enters the facial, tongue, and pulse diagnosis data collection phase, sequentially collecting facial images, tongue images, and pulse waveforms. A multimodal feature extraction submodule extracts feature vectors for each modality, followed by multiple rounds of cross-modal communication fusion submodules for decoupling unique and shared features, aggregation of complementary information, and residual fusion. This enhances the multimodal representation capability round by round, outputting the Top 3 candidate constitutions. Next, the user enters the dynamic consultation phase, where the LLM (Linguistic Model) selects 5 consultation questions based on the Top 3 constitutions. The digital human asks the questions and collects the user's voice responses. Finally, the large language model integrates the facial, tongue, pulse, and consultation data to determine the user's constitution, generating a report and ending the physical examination.

[0110] The constitution identification module analyzes and processes the collected multimodal data. Specifically, the multimodal feature extraction submodule uses a VGG-19 convolutional neural network to extract d-dimensional feature vectors from facial and tongue images, and a one-dimensional residual convolutional neural network to extract d-dimensional temporal feature vectors from pulse waveforms.

[0111] The multi-round cross-modal communication fusion submodule performs multi-round information exchange on the feature vectors of facial diagnosis, tongue diagnosis, and pulse diagnosis. In each round, the features of each modality are decoupled into modality-specific features and modality-shared features. Orthogonal constraints are used to ensure that the two types of feature information are not redundant. Similarity loss is used to align the shared features of each modality to form cross-modal consensus. The unique features of other modalities are aggregated as complementary information and injected into each modality through residual fusion. After multiple rounds of communication, the final unique features and shared features of each modality are aggregated and input into the Softmax classifier to output the probability distribution of nine constitutions. Finally, the three constitutions with the highest probabilities are selected as the candidate constitution set.

[0112] The dynamic consultation decision-making submodule receives a set of candidate constitutions and the probability value of each constitution as input, constructs structured prompts, and calls a large language model to select the five most distinguishable candidate constitutions from a pre-set database of 60 consultation questions. These questions are asked to the user one by one through digital human voice interaction, and the user's voice answers are recognized and recorded.

[0113] The constitution reasoning submodule integrates the candidate constitution set and probability values ​​of each constitution, the consultation questions, and user answers into structured input, and calls a large-scale language model for comprehensive reasoning. The large-scale language model can globally examine the evidence from inspection, tongue diagnosis, pulse diagnosis, and consultation, simulating the diagnostic thinking of TCM experts who combine the four diagnostic methods, and outputs the final constitution determination result.

[0114] The report generation module, based on the constitution assessment results, uses One-Shot Prompt Engineering to call upon a large language model to generate a structured health report. The report template includes: constitution characteristics (general features, common manifestations, psychological characteristics, disease tendency, environmental adaptability), emotional regulation (psychological adjustment suggestions, music therapy suggestions), dietary regulation (recommended porridge, dishes, and teas), daily life regulation (suggestions for living environment and lifestyle habits), exercise and health care (suggestions for exercise type and timing), and acupoint health care (selection of acupoints, efficacy, location, operation methods, and moxibustion suggestions).

[0115] The generated report is summarized and narrated by a digital human, while the full report is displayed on the interface in a graphic format. The system supports generating QR codes, which users can scan with their mobile devices to view the full report, enabling cross-device access.

[0116] The system of this invention can be deployed on various terminal platforms: as a standalone application on smart screens, all-in-one machines and other devices, suitable for community health service centers, health centers and other scenarios; as an embedded module integrated into health management terminals and other devices; and as a cloud service providing remote TCM constitution identification services via Web or mobile App.

[0117] Example 2

[0118] like Figure 3 As shown, this embodiment provides a method for identifying TCM constitution based on anthropomorphic interaction using a TCM constitution identification service robot. The method includes the following steps:

[0119] S1: The digital human interaction module provides a virtual TCM avatar and enables voice interaction to guide users through the physical fitness testing process;

[0120] In this embodiment, the digital human interaction module adopts streaming speech-driven face video generation technology to achieve real-time synchronous output of speech synthesis and face animation with a delay of less than 2 seconds, and supports a streaming interaction mode that generates and plays simultaneously.

[0121] S2: Under the voice guidance of the digital human interaction module, multimodal data is collected based on the multimodal data acquisition module, including facial images, tongue images, pulse signals and voice response information;

[0122] Specifically, the multimodal data acquisition module includes an image acquisition unit, a pulse acquisition unit, and a voice acquisition unit. The pulse acquisition unit is equipped with a piezoelectric pulse sensor array. The pulse sensor acquires radial artery pulse waveform time-series data at a sampling frequency of not less than 500Hz. After baseline drift correction, bandpass filtering, and peak normalization preprocessing, a standardized pulse waveform sequence is output.

[0123] S3: Based on the multimodal feature extraction submodule, discriminative feature vectors of each modality are extracted from facial images, tongue images and pulse signals;

[0124] S4: Based on the multi-round cross-modal communication fusion submodule, the feature vectors of the face, tongue and pulse modes are used to perform cross-modal information interaction through stacked multi-round SRMC modules, including multi-round feature decoupling, complementary information aggregation and residual fusion operations, outputting the probability distribution of various TCM constitution types and selecting candidate constitution sets;

[0125] S5: A large language model based on the dynamic consultation decision submodule generates consultation questions based on the candidate constitution set, asks questions to users in voice form, collects users' voice answers and converts them into text;

[0126] In this embodiment, user questions and answers are conducted in a multi-turn dialogue format. Each round of dialogue includes: digital human voice output, synchronized playback of facial animation, user voice input, and a complete interactive cycle of speech recognition to text conversion.

[0127] S6: The constitution reasoning submodule integrates the candidate constitution set, consultation questions and voice answer information into structured multimodal evidence text, performs constitution reasoning based on a large language model, and outputs the final constitution judgment result and confidence statement;

[0128] S7: Based on the report generation module, a structured health report is generated according to the physical fitness assessment results and the preset report template. The report is then broadcast and displayed via voice using the digital human interaction module.

[0129] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A service robot for identifying traditional Chinese medicine constitution based on anthropomorphic interaction, characterized in that, include: Digital human interaction module, multimodal data acquisition module, physical fitness identification module, and report generation module; The digital human interaction module is used to provide a virtual TCM avatar and enable voice interaction; The multimodal data acquisition module is used to acquire multimodal data, including facial images, tongue images, pulse signals, and voice response information; The constitution identification module is used for constitution analysis and reasoning based on multimodal data, including: multimodal feature extraction submodule, multi-round cross-modal communication fusion submodule, dynamic consultation decision submodule, and constitution reasoning submodule; The multimodal feature extraction submodule is used to extract discriminative feature vectors for each modality from facial images, tongue images, and pulse signals; The multi-round cross-modal communication fusion submodule is used to exchange feature vectors from three modalities—face, tongue, and pulse—through stacked multi-round SRMC modules to output probability distributions of various TCM constitution types and select a candidate constitution set. The dynamic consultation decision submodule is used to generate consultation questions based on the candidate constitution set; The constitution reasoning submodule is used to integrate the candidate constitution set, consultation questions and voice response information into structured multimodal evidence text, and output the final constitution judgment result and confidence statement based on the large language model; The report generation module is used to generate structured health reports based on the physical fitness assessment results and preset report templates.

2. The TCM constitution identification service robot based on anthropomorphic interaction according to claim 1, characterized in that, The digital human interaction module includes: a speech recognition submodule, a dialogue generation submodule, a speech synthesis submodule, and a face animation generation submodule; The speech recognition submodule is used to collect user speech and convert it into text data; The dialogue generation submodule is used to generate response text based on user input text; The speech synthesis submodule is used to convert response text into speech signals; The face animation generation submodule is used to generate digital facial animation videos that are synchronized with the speech based on the speech signal.

3. The TCM constitution identification service robot based on anthropomorphic interaction according to claim 2, characterized in that, The face animation generation submodule includes: an audio feature extractor, a lip shape prediction network, an expression generation network, and a video rendering engine; Audio feature extractors are used to extract acoustic features from speech signals; The lip shape prediction network uses a Transformer architecture to predict lip shape parameter sequences based on acoustic features; Facial expression generation networks are used to generate facial expression parameters that match the emotions conveyed in speech. The video rendering engine is used to render face video frames based on pre-trained digital human models, according to lip shape parameter sequences and facial expression parameters.

4. The TCM constitution identification service robot based on anthropomorphic interaction according to claim 1, characterized in that, The multimodal data acquisition module includes an image acquisition unit, a pulse acquisition unit, and a voice acquisition unit; The image acquisition unit acquires facial and tongue images; The pulse acquisition unit is equipped with a piezoelectric pulse sensor array for acquiring pulse waveform timing data; The voice acquisition unit is equipped with a microphone array for acquiring voice response information.

5. The TCM constitution identification service robot based on anthropomorphic interaction according to claim 4, characterized in that, The image acquisition unit acquires multiple frames of video images, performs detection and key point localization on each frame of video images, performs affine alignment correction based on the line connecting the eyes, expands the bounding box according to a set ratio and then crops to obtain the facial image, and selects the frame with the highest clarity score as the final facial image. The image acquisition unit performs pixel-level semantic segmentation on the acquired image based on the tongue segmentation network, extracts the tongue region, performs LAB color space conversion on the tongue image and CLAHE adaptive histogram equalization on the L brightness channel to obtain the final tongue image. After acquiring pulse waveform time-series data, the pulse acquisition unit performs baseline drift correction, bandpass filtering, and peak normalization preprocessing on the original pulse waveform to obtain the final pulse waveform time-series data.

6. The TCM constitution identification service robot based on anthropomorphic interaction according to claim 1, characterized in that, The multimodal feature extraction submodule includes a facial feature encoder, a tongue feature encoder, and a pulse feature encoder; The facial feature encoder and the tongue feature encoder each use VGG-19 convolutional neural network as the backbone network. Each retains the original five groups of VGG-19 convolutional pooling blocks. After the feature map is output from the fifth group of convolutional pooling blocks, a global average pooling layer and a fully connected mapping layer are connected to compress and map each feature into a multi-dimensional semantic feature vector. The pulse feature encoder uses a one-dimensional residual convolutional neural network. It takes the preprocessed standardized pulse waveform sequence as input and extracts multi-scale temporal features through four sets of one-dimensional residual convolutional blocks. Each set of one-dimensional residual convolutional blocks contains two layers of one-dimensional convolution, batch normalization and residual skip connections. After the output of the last set of residual blocks, a global average pooling layer and a fully connected mapping layer are connected to compress and map the temporal features of the pulse waveform into a multi-dimensional semantic feature vector.

7. The TCM constitution identification service robot based on anthropomorphic interaction according to claim 1, characterized in that, The multi-round cross-modal communication fusion submodule is used to exchange feature vectors from three modalities—face, tongue, and pulse—through stacked multi-round SRMC modules for cross-modal information exchange. Specifically, it includes: For each modality's feature vector, feature decoupling is performed, mapping it to modality-specific features and modality-shared features respectively; Minimize the squared Frobenius norm of the inner product matrix of mode-specific features and mode-shared features within each mode; A cross-modal similarity alignment loss is constructed and optimized by averaging the cosine dissimilarity of shared features among all modal pairs; For each modality, unique features from the other two modalities are aggregated as complementary information; The complementary information is concatenated with the original features of the corresponding modality and then input into the residual fusion network. The updated features are obtained through residual connections. After multiple rounds of cross-modal information interaction, the final unique features and shared features of the three modalities are aggregated into a multimodal fusion representation. The multimodal fusion representation is processed by a multilayer perceptron classifier and a softmax activation function to output the probability distribution of various TCM constitution types. The top-k constitutions are selected as the candidate constitution set.

8. The TCM constitution identification service robot based on anthropomorphic interaction according to claim 7, characterized in that, For each modality, the unique features from the other two modalities are aggregated as complementary information, represented as: ； in, Represents complementary information vectors, This indicates a splicing operation. This represents the mode-specific feature corresponding to mode i. Represents facial modality. Indicates tongue modality, This indicates the pulse mode.

9. The TCM constitution identification service robot based on anthropomorphic interaction according to claim 1, characterized in that, The dynamic consultation decision-making submodule generates consultation questions based on the candidate constitution set, specifically including: The dynamic consultation decision-making submodule constructs structured prompts containing key points for identifying candidate constitutions based on the candidate constitution set and the probability values ​​of each constitution. The large language model uses information gain as the screening criterion to select a subset of consultation questions.

10. The identification method of the TCM constitution identification service robot based on anthropomorphic interaction according to any one of claims 1-9, characterized in that, Includes the following steps: Based on the digital human interaction module, a virtual TCM avatar is provided and voice interaction is enabled to guide users into the testing process; Multimodal data is collected based on the multimodal data acquisition module, including facial images, tongue images, pulse signals, and voice response information; The multimodal feature extraction submodule extracts discriminative feature vectors for each modality from facial images, tongue images, and pulse signals. Based on the multi-round cross-modal communication fusion submodule, the feature vectors of the three modalities of face, tongue and pulse are used to perform cross-modal information interaction through stacked multi-round SRMC modules, outputting the probability distribution of various TCM constitution types and selecting a candidate constitution set; Based on the dynamic consultation decision submodule, consultation questions are generated according to the candidate constitution set; The constitution reasoning submodule integrates candidate constitution sets, consultation questions, and voice response information into structured multimodal evidence text, and outputs the final constitution determination result and confidence statement based on a large language model; Based on the report generation module, a structured health report is generated according to the physical fitness assessment results and the preset report template.

Images