Depression diagnosis and treatment integrated humanoid robot based on multi-source information perception

Abstract

The invention discloses a depression diagnosis and treatment integrated humanoid robot based on multi-source information perception, and belongs to the field of medical instruments. The depression diagnosis and treatment integrated humanoid robot comprises a head module, a trunk module and a limb module which are worn outside a human body; a high-resolution camera, a small integrated microphone array, a gas sensor, a photoelectric sensor, a pressure sensor, an infrared sensor and a numerical control chip are arranged in the head module; the high-resolution camera is used for collecting facial recognition and eyeball movement features of a user; the small-sized integrated microphone array is used for dynamically collecting human body sound. According to the method, early warning, accurate diagnosis, personalized treatment and effective accompanying of depression can be realized, the accuracy and objectivity of depression diagnosis are improved, the illness state of a patient is comprehensively evaluated through multi-source information perception, all-weather emotion accompanying is provided, the psychological state of the patient is improved, the treatment compliance is improved, the problem of uneven distribution of medical resources is relieved, and the treatment effect is improved. And patients in remote areas can obtain high-quality diagnosis and treatment services.

Description

Technical Field

[0001] The present application relates to the field of medical instrument technology, and in particular to an integrated humanoid robot for depression diagnosis and treatment based on multi-source information perception. Background Art

[0002] Depression is a common and serious psychological disorder, and its incidence rate is increasing year by year, which has brought a heavy burden to society and individuals. Traditional methods of diagnosis and treatment of depression mainly rely on subjective questionnaires, doctors' experience judgments and drug treatments, which have many limitations. First, subjective questionnaires and doctors' experience judgments are often affected by patients' subjective feelings, doctors' personal experience and prejudice, resulting in inaccurate diagnostic results. Secondly, although drug treatment can relieve symptoms to a certain extent, it lacks personalization and cannot provide precise treatment for the specific conditions of different patients, and there may be problems such as side effects and drug resistance.

[0003] In addition, patients are often reluctant to seek medical treatment due to "stigma" and privacy concerns, which further exacerbates the problem of delayed treatment. At the same time, existing depression diagnosis and treatment technologies are still single-minded and non-quantitative in terms of perception and diagnostic methods, and cannot fully utilize multi-source information to comprehensively assess the patient's condition.

[0004] With the rapid development of technologies such as artificial intelligence, big data, and the Internet of Things, new ideas and methods have been provided for the diagnosis and treatment of depression. How to apply these advanced technologies to the diagnosis and treatment of depression, improve diagnostic accuracy, achieve personalized treatment, and provide comprehensive patient support has become an urgent problem to be solved. Summary of the invention

[0005] In response to the shortcomings of the existing technology, this application proposes an integrated humanoid robot for depression diagnosis and treatment based on multi-source information perception, which aims to achieve early warning, accurate diagnosis, personalized treatment and effective companionship of depression by integrating multi-source information and applying advanced algorithms and models, thereby improving diagnosis and treatment efficiency, improving patient prognosis and reducing medical costs.

[0006] To achieve the above objectives, this application is implemented through the following technical solutions: A depression diagnosis and treatment integrated humanoid robot based on multi-source information perception, comprising: a head module, a trunk module and limb modules worn on the outside of the human body, wherein the head module is internally configured with a high-resolution camera, a small integrated microphone array, a gas sensor, a photoelectric sensor, a pressure sensor, an infrared sensor and a CNC chip; The high-resolution camera is used to collect the user's facial recognition and eye movement features; the small integrated microphone array is used to collect the human body sound dynamics, and the gas sensor is used to collect the human body odor; the photoelectric sensor, pressure sensor and infrared sensor are used to collect the user's pulse wave; the CNC chip is used to store all software algorithms; The working method of the humanoid robot is: High-resolution cameras are used to collect user facial recognition and eye movement features, small integrated microphone arrays to collect human voice dynamics, gas sensors to collect human odor, photoelectric sensors, pressure sensors and infrared sensors to collect user pulse waves; Depression outcomes are judged through the multimodal heterogeneous body surface information collected from facial expressions, eye movements, human body odor, and human pulse waves; Build a patient corpus, use a small integrated microphone array to collect real data from patient conversation records, online support group discussion content, and psychological counseling texts, annotate the emotional category, psychological state level, and conversation intention of the corpus, and use data enhancement technology to expand the corpus size; A large-scale corpus is used for pre-training to obtain an initial powerful large language model. Psychological knowledge is introduced in the pre-training stage of the large language model, and the model generation content is optimized through preset labels. The focus is on introducing the ability to process proprietary vocabulary in the field of mental health. The feature extraction algorithm is adjusted according to the voice characteristics of patients with depression. Combined with the specific needs of patients with depression, a variety of companionship modes including listening mode, knowledge sharing mode and guidance mode are designed to adapt to the different psychological states of patients.

[0007] Preferably, the high-resolution camera uses a 3-CCD camera of Sony DXC390P as a reference camera, and the resolution of the high-resolution camera is 2560×1440.

[0008] Preferably, the software algorithm includes a human-computer interaction and companionship module and a depression recognition module.

[0009] Preferably, the human-computer interaction and companionship module includes the following operating steps: S1: Construction of corpus of patients with depression; S2: Setting up automatic speech recognition module; S3: training large language model; S4: Set the speech synthesis module.

[0010] Preferably, the construction of the corpus of depression patients includes data collection, data annotation and corpus expansion; The data collection includes real data of patient conversation records, online support group discussion content and psychological counseling texts. The data annotation includes emotion categories, psychological state levels and conversation intentions. The corpus expansion uses data enhancement technology to expand the corpus size.

[0011] Preferably, the automatic speech recognition module is used to convert the speech signal Convert to text sequence , for the sequence alignment problem, the automatic speech recognition module uses the connectionist temporal classification loss optimization: ; Among them, the conditional probability The probability of all possible alignment paths To define: ; in, : the sequence of each alignment path; : A compression function that maps the alignment path to the target sequence; : is the length of the text sequence; is the length of the speech signal; Represents the sequence of each alignment path at time elements.

[0012] Preferably, the training of the large language model includes pre-training the large language model, fine-tuning the large language model and model early warning; The pre-trained large language model is used to obtain an initial powerful large language model. Psychological knowledge is introduced in the pre-training stage of the large language model, and the model generation content is optimized by preset tags. The feature extraction algorithm is adjusted according to the speech characteristics of patients with depression. In the long dialogue scenario, the long short-term memory and attention mechanism are used to retain the context information; The large language model is fine-tuned using labeled data and multi-task learning. A large number of text fragments generated by the large language model after fine-tuning are collected and sorted by human annotators. The sorting results are used as input to train the reward model. Based on the reinforcement learning of reward feedback, the model learns to distinguish between high-quality and low-quality texts, sets personalized content generation, and combines the patient's historical interaction data to generate targeted conversation content. The model warning is used to automatically record the patient's emotional fluctuations and language changes during voice interaction and analyze his or her mental health trends.

[0013] Preferably, the speech synthesis module realizes the natural generation of speech waveform from text in the speech synthesis process; its core is to represent the text sequence Mapping to spectrogram , and then from the spectrogram Generate time domain waveform ; Representing a text sequence Mapping to spectrogram : ; ; in, : Hidden state of the decoder; : Hidden representation generated by the text encoder; : No. Moment The attention weights of the text representations; : attention score function; Indicates the length of the text sequence, that is, the text sequence The number of elements in ; The loss function is to minimize the generated spectrogram and real spectrogram The error: ; in, Spectrogram representing the generated model At the moment slices, Represents the true spectrogram at time slices, express and The square of the Euclidean distance between them is used to quantify the difference between them; from the spectrogram Generate time domain waveform , the spectrogram Convert to time domain waveform , achieved through conditional probability modeling: ; in, :time The waveform sampling value of :exist All waveform sampling values ​​before time; : Spectrogram; Use dilated convolution to extract long sequence dependency features: ; in, Indicates time The predicted value of the waveform sampling value; : convolution kernel size; : convolution weight; : Expansion coefficient, controlling the growth of the receptive field; : Indicates time Before The waveform sampling value of time steps; Use cross-cutting loss optimization: ; in, is the model parameter.

[0014] Preferably, the depression recognition module obtains the facial expression depression judgment result through the convolutional neural network CNN , Eye movement features obtained through long short-term memory network LSTM depression judgment results , Analysis of depression judgment results through odor threshold , the final depression verdict The following is achieved by weighted fusion of the judgment results of three features: facial expression, eye movement and smell analysis: ; in, : The depression judgment result obtained by the deep learning model based on facial expression features, its value is 1 indicates a high probability of depression; : The depression judgment result obtained by the deep learning model based on the eye movement characteristics, and its value is between; : The depression judgment result obtained through odor analysis, its value is between; : The depression judgment result obtained by pulse wave velocity analysis is between; : The weight of each judgment result, satisfying ; Characteristic facial expression judgment results Use convolutional neural network CNN and combine it with emotion classification task for pre-training: ; in, : Facial expression images or video frames collected by a high-resolution camera; : weight and bias parameters; : Classification function, output classification results; among them, the activation function used by the convolutional neural network is: ; ; in, : represents the Sigmoid activation function; : represents weight; : indicates bias; : represents input data; Eye movement characteristics in the judgment of depression Processing dynamic features based on long short-term memory network LSTM: ; in, : The eye trajectory time series data collected by the high-resolution camera, including pupil diameter, eye movement speed, and gaze point coordinate changes; : represent weight and bias parameters respectively; among them, the long short-term memory network uses the group check attention mechanism: ; ; in, :Check matrix, check the space Divided into The attention scores of each group were calculated independently. : No. The query vector of the group; : key matrix, not grouped, globally shared; : Value matrix, not grouped, globally shared; : sequence length, : is the feature dimension; Smell analysis depression judgment formula, depression judgment results based on exhaled breath components The results are calculated by comparing the detected gas concentration with the threshold of the healthy control group: ; in, : The concentration of hydrogen monoxide, hydrogen sulfide and carbon monoxide in the patient's exhaled gas measured by gas sensors; : The average concentration thresholds of hydrogen monoxide, hydrogen sulfide and carbon monoxide in the gas of the healthy control group; : Standard deviation of hydrogen monoxide, hydrogen sulfide, and carbon monoxide in the healthy control group, used for standardization; : Sigmoid function, used to map the difference to the range ,and .

[0015] Preferably, the pulse wave velocity analysis depression judgment formula is determined based on the relationship between the collected pulse wave velocity of the patient and the average value of the healthy control group: ; in, : The patient's pulse wave velocity collected by photoelectric, pressure, and infrared sensors; : Mean pulse wave velocity of healthy control group; : Standard deviation of pulse wave velocity in healthy control group; : Sigmoid function, used to map the standardized difference to the range .

[0016] Compared with the prior art, the depression diagnosis and treatment integrated humanoid robot based on multi-source information perception provided by this application has achieved the following beneficial effects: (1) Improving diagnostic accuracy and objectivity: Through multi-source information perception, including facial expressions and eye movement features collected by high-resolution cameras, sound dynamics collected by small integrated microphone arrays, odor information collected by gas sensors, and pulse wave information collected by photoelectric, pressure, and infrared sensors, the patient's condition can be comprehensively assessed and the accuracy and objectivity of diagnosis can be improved.

[0017] (2) Personalized treatment: Based on the patient's specific conditions, such as genotype, medical history, symptoms, etc., personalized treatment plans are recommended through algorithm models to better meet the patient's needs and improve treatment outcomes and patient satisfaction.

[0018] (3) Providing emotional companionship around the clock: The humanoid robot is equipped with human-computer interaction and companionship modules, which can accompany patients around the clock, provide emotional support, improve the patient's psychological state, and improve treatment compliance. It not only helps alleviate the patient's loneliness and anxiety, but also enhances the patient's confidence in treatment.

[0019] (4) Alleviating the uneven distribution of medical resources: Through the remote diagnosis and treatment services provided by humanoid robots, patients in remote areas can also obtain high-quality diagnosis and treatment services, thereby alleviating the problem of uneven distribution of medical resources, helping to improve the overall medical level and reduce medical costs.

[0020] (5) This invention provides strong support for the innovation and development of medical technology by integrating multi-source information and applying advanced algorithms and models.

[0021] In summary, the integrated humanoid robot for depression diagnosis and treatment based on multi-source information perception provided by the present application has achieved remarkable beneficial effects in improving diagnostic accuracy and objectivity, realizing personalized treatment, providing round-the-clock emotional companionship, alleviating the uneven distribution of medical resources, and promoting medical technology innovation and development. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1It is a schematic diagram of the head module structure of a humanoid robot for integrated diagnosis and treatment of depression based on multi-source information perception according to the present invention; Figure 2 This is a multi-modal depression recognition flow chart of a depression diagnosis and treatment integrated humanoid robot based on multi-source information perception of the present invention; Figure 3 This is a flow chart of a gas-sensitive sensor of a humanoid robot for integrated diagnosis and treatment of depression based on multi-source information perception according to the present invention.

[0024] In the figure: 1. High-resolution camera; 2. Small integrated microphone array; 3. Gas sensor; 4. Photoelectric sensor; 5. Pressure sensor; 6. Infrared sensor; 7. CNC chip. DETAILED DESCRIPTION

[0025] The embodiments of the present application provide a method and device for optimizing the detection and extraction process of tea polyphenols based on near-infrared spectroscopy. The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described here can be implemented in an order other than that illustrated or described here. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] Example: Figure 1 , Figure 2 and Figure 3 As shown, a depression diagnosis and treatment integrated humanoid robot based on multi-source information perception of the present invention adopts a bionic appearance design close to the proportion of a real person, including a head module, a torso module and limb modules worn on the outside of the human body, and the interior of the head module is configured with a high-resolution camera 1, a small integrated microphone array 2, a gas sensor 3, a photoelectric sensor 4, a pressure sensor 5, an infrared sensor 6 and a CNC chip 7; The high-resolution camera 1 is used to collect the user's facial recognition and eye movement features; the small integrated microphone array 2 collects the human body sound dynamics; there are multiple gas sensors 3, which are distributed in an array and are used to collect human body odor; the photoelectric sensor 4, pressure sensor 5 and infrared sensor 6 are used to collect the user's pulse wave; the CNC chip 7 is used to store all software algorithms; The working method of the humanoid robot is: A. Use a high-resolution camera 1 to collect user facial recognition and eye movement features, a small integrated microphone array 2 to collect human voice dynamics, a gas sensor 3 to collect human odor, and a photoelectric sensor 4, a pressure sensor 5 and an infrared sensor 6 to collect user pulse waves; B. Determine depression outcomes through the multimodal heterogeneous body surface information collected, including facial expressions, eye movements, human body odor, and human pulse waves; C. Build a patient corpus, use a small integrated microphone array 2 to collect real data such as patient conversation records, online support group discussion content, psychological counseling texts, etc., annotate the emotional category, psychological state level and conversation intention of the corpus, and use data enhancement technology to expand the corpus size; D. Use large-scale corpus pre-training to obtain an initial powerful large language model. In the pre-training stage of the large language model, psychological knowledge is introduced, and the model generation content is optimized through preset labels. The focus is on introducing the ability to process proprietary vocabulary in the field of mental health. The feature extraction algorithm is adjusted according to the voice characteristics of patients with depression. Combined with the specific needs of patients with depression, a variety of companionship modes (such as listening mode, knowledge sharing mode, and guidance mode) are designed to adapt to the different psychological states of patients.

[0027] The high-resolution camera 1 uses a Sony DXC 390P 3-CCD camera as a reference camera, and the resolution of the high-resolution camera 1 is 2560×1440.

[0028] Among them, the software algorithm includes a human-computer interaction and companionship module and a depression recognition module. The companionship logic is centered on the large language model. Combined with the specific needs of patients with depression, a variety of companionship modes (such as listening mode, knowledge sharing mode, and guidance mode) are designed to adapt to the different psychological states of patients and provide language output to relieve stress. Multimodal interaction constructs companionship logic based on the large language model, combines deep learning speech recognition and synthesis technology to achieve natural human-computer dialogue, and collects and builds a multimodal corpus to support Chinese natural language understanding.

[0029] Among them, the human-computer interaction and companionship module includes the following operating steps: 1. Construction of a corpus of patients with depression; 2. Setting up an automatic speech recognition module; 3. Training a large language model; 4. Setting up a speech synthesis module.

[0030] Among them, the construction of the corpus of depression patients includes data collection, data annotation and corpus expansion; the data collection includes real data such as patient conversation records, online support group discussion content, psychological counseling texts, etc., and the data is de-identified to ensure privacy and security; data annotation includes emotion categories (such as sadness, anxiety, indifference), psychological state levels (mild depression, moderate depression) and conversation intentions, etc., and the expert team guides the annotation to ensure the quality of the corpus; corpus expansion uses data enhancement technology (such as sentence conversion, semantic replacement) to expand the scale of the corpus and enhance the model's ability to understand diverse inputs.

[0031] Among them, based on the automatic speech recognition module, it supports efficient transcription of Mandarin Chinese and dialects. The user's voice input is converted into text by the automatic speech recognition module. The large language model updates the patient's emotional state and psychological characteristics in real time based on the voice and text data, generates a response, and then outputs it through the speech synthesis module. The main task of the automatic speech recognition module is to convert the voice signal Convert to text sequence ; For the sequence alignment problem, the automatic speech recognition module uses connectionist temporal classification loss optimization: ; Among them, the conditional probability The probability of all possible alignment paths To define: ; in, : the sequence of each alignment path; : A compression function that maps the alignment path to the target sequence; : is the length of the text sequence, : is the length of the speech signal; Represents the sequence of each alignment path at time Due to the temporal misalignment between speech signals and text sequences (i.e., one phoneme may correspond to multiple speech frames, or one speech frame may contain information of multiple phonemes), the model needs to be optimized through the connectionist temporal classification (CTC) loss. In this process, Indicates that on the alignment path, the moment The corresponding phoneme or phoneme combination.

[0032] The training of the large language model includes pre-training the large language model, fine-tuning the large language model and model early warning; The pre-trained large language model is used to obtain an initial powerful large language model. Psychological knowledge is introduced in the pre-training stage of the large language model. The model generates content by optimizing preset labels (such as depression and anxiety tendency) to avoid triggering sensitive emotions of patients. The focus is on introducing the processing ability of specialized vocabulary in the field of mental health. The feature extraction algorithm is adjusted according to the speech characteristics of patients with depression (such as low voice volume and slow speech speed). Time series modeling is used to enhance the recognition ability of incoherent sentences. In long dialogue scenarios, long short-term memory and attention mechanism are used to retain contextual information to ensure the coherence and relevance of the dialogue. The large language model is fine-tuned using labeled data and multi-task learning to allow the model to better meet the task requirements of diagnosis and treatment of patients with depression. A large number of text fragments generated by the large language model after fine-tuning are collected, and human annotators are asked to sort them. The sorting results are used as input to train the reward model. Reinforcement learning based on reward feedback allows it to learn to distinguish high-quality and low-quality texts. The model parameters are adjusted according to the output of the reward model to optimize the reward function, and the model is guided to generate text that is more in line with human preferences; personalized content generation is set, and targeted dialogue content is generated in combination with the patient's historical interaction data. A dialogue guidance strategy is set to combine open and closed questions, provide daily psychological support, and guide patients to complete mindfulness training, deep breathing and other relaxation exercises; The model warning is used to automatically record the patient's emotional fluctuations and language changes during voice interaction, analyze their mental health trends, and conduct periodic tracking and adjustment of the patient's status in combination with the doctor's advice. If the patient is identified to have symptoms of depression, the system automatically triggers the warning mechanism.

[0033] The speech synthesis module realizes highly natural speech synthesis, supports emotional speech output, and generates warm and infectious speech responses through parameterized dynamic control of intonation and rhythm, thus improving the patient interaction experience. The speech synthesis module realizes the natural generation of speech waveforms from text in the speech synthesis process. Its core is to represent text sequences Mapping to spectrogram , and then from the spectrogram Generate time domain waveform .

[0034] Representing a text sequence Mapping to spectrogram : ; ; in, : Hidden state of the decoder; : Hidden representation generated by the text encoder; : No. Moment The attention weights of the text representations; : attention score function; Indicates the length of the text sequence, that is, the text sequence The number of elements in ; in, Spectrogram representing the generated model At the moment slices, Represents the true spectrogram at time slices, express and The square of the Euclidean distance between them is used to quantify the difference between the two; the loss function is to minimize the generated spectrogram and real spectrogram The error: ; Loss Function It is used to measure the difference between the spectrogram generated by the model and the real spectrogram. By minimizing this loss function, the model can be optimized so that the spectrogram it generates is closer to the real spectrogram.

[0035] From the spectrogram Generate time domain waveform , the spectrogram Convert to time domain waveform , achieved through conditional probability modeling: ; in, :time The waveform sampling value of :exist All waveform sampling values ​​before time; : Spectrogram; Use dilated convolution to extract long sequence dependency features: ; in, : convolution kernel size; : convolution weight; : Expansion coefficient, controlling the growth of the receptive field; Indicates time The predicted value of the waveform sampling value is obtained by extracting features from past waveform sampling values ​​through dilated convolution. Indicates time Before The waveform sampling value of the time step is Value and Values ​​can capture waveform characteristics within different time ranges.

[0036] Use cross-cutting loss optimization: ; in, are model parameters.

[0037] Among them, by collecting multi-modal heterogeneous body surface information, the results are weighted and judged; the depression recognition module obtains the facial expression depression judgment results through the convolutional neural network CNN , Eye movement features obtained through long short-term memory network LSTM depression judgment results , Analysis of depression judgment results through odor threshold , the final depression verdict The following is achieved by weighted fusion of the judgment results of three features: facial expression, eye movement and smell analysis: ; : The depression judgment result obtained by the deep learning model based on facial expression features ( 1 indicates a high likelihood of depression); :The depression judgment result obtained by the deep learning model based on the eye movement characteristics ( between); : The results of depression judgment obtained through odor analysis ( between); : Depression judgment results obtained through pulse wave velocity analysis ( between); : The weight of each judgment result, satisfying ;The weights are verified by feature importance experiments; Characteristic facial expression judgment results Use convolutional neural network CNN and combine it with emotion classification task for pre-training: ; in, : Facial expression images or video frames collected by a high-resolution camera; : weight and bias parameters; : Classification function, output classification results; Among them, the activation function used by the convolutional neural network is: ; ; in, : represents the Sigmoid activation function; : weight; : bias; : Input data; Eye movement characteristics in the judgment of depression Processing dynamic features based on long short-term memory network LSTM: ; : Eye trajectory time series data (pupil diameter, eye movement speed, gaze point coordinate changes) collected by a high-resolution camera; : weight and bias parameters; Among them, the long short-term memory network uses the group-check attention mechanism: ; ; : Check the matrix, and change the check Divided into There were 3 groups, and attention scores were calculated independently for each group; : No. The query vector of the group; : Key matrix, not grouped, globally shared; : Value matrix, not grouped, globally shared; : sequence length, : Feature dimension; The formula for judging depression based on odor analysis is that the components of nitric oxide, hydrogen sulfide, and carbon monoxide in the exhaled gas of patients with depression will change. The concentrations of nitric oxide and hydrogen sulfide will decrease compared with the healthy control group, while the concentration of carbon monoxide will increase compared with the healthy control group. The judgment result of depression based on the components of exhaled gas is The results are calculated by comparing the detected gas concentration with the threshold of the healthy control group: ; in, : respectively represent the concentrations of hydrogen monoxide, hydrogen sulfide and carbon monoxide in the patient's exhaled gas measured by gas sensors; : represent the average concentration thresholds of hydrogen monoxide, hydrogen sulfide and carbon monoxide gases in the healthy control group; : standard deviation of the healthy control group, used for standardization; : Sigmoid function, used to map the difference to ( ) range, and .

[0038] Among them, the pulse wave velocity analysis depression judgment formula is as follows: The emotional pathways of the brain may affect heart beats through autonomic nervous activity. Pulse wave velocity (PWV) can be used as a clinical biomarker. An increase in pulse wave velocity indicates more severe depression. The pulse wave velocity analysis depression judgment formula is determined based on the relationship between the collected patient pulse wave velocity and the average pulse wave velocity of the healthy control group: ; : The patient's pulse wave velocity collected by photoelectric, pressure, and infrared sensors; : Mean pulse wave velocity of healthy control group; : Standard deviation of pulse wave velocity in healthy control group; : Sigmoid function, used to map the standardized difference to ( ).

[0039] When working, this depression diagnosis and treatment integrated humanoid robot based on multi-source information perception can achieve early warning, accurate diagnosis, personalized treatment and effective companionship of depression through the humanoid robot hardware configuration, human-computer interaction and companionship module and multi-modal depression recognition module, improve the accuracy and objectivity of depression diagnosis, and comprehensively evaluate the patient's condition through multi-source information perception. It provides all-weather emotional companionship, improves the patient's psychological state, and improves treatment compliance. It helps to alleviate the problem of uneven distribution of medical resources and enable patients in remote areas to obtain high-quality diagnosis and treatment services.

[0040] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0041] In addition, the terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previously associated objects are in an "or" relationship.

[0042] It should be understood that in the embodiments of the present application, "B corresponding to A" means that B is associated with A, and B can be determined based on A, but it should also be understood that determining B based on A does not mean determining B only based on A, and B can also be determined based on A and / or other information.

[0043] A person of ordinary skill in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0044] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0045] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. There may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or units, or may be electrical, mechanical or other forms of connection.

[0046] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units, and some or all of the units may be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0047] In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.

[0048] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by hardware, firmware, or a combination thereof. When implemented by software, the above functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by a computer. For example, but not limited to: computer-readable media can include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage media or other magnetic storage devices, or can be used to carry or store desired data in the form of instructions or data structures. Any other medium in which program code can be stored and accessed by a computer. In addition, any connection can appropriately become a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, wireless and microwave are included in the fixation of the medium to which it belongs. As used in this application, disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs use lasers to optically reproduce data. The above combinations should also be included in the protection scope of computer-readable media.

[0049] In short, the above is only a preferred embodiment of the technical solution of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A humanoid robot for integrated diagnosis and treatment of depression based on multi-source information perception, characterized in that: include: A head module, a torso module and limb modules worn on the outside of a human body, wherein the head module is internally provided with a high-resolution camera (1), a small integrated microphone array (2), a gas sensor (3), a photoelectric sensor (4), a pressure sensor (5), an infrared sensor (6) and a numerical control chip (7); The high-resolution camera (1) is used to collect facial recognition and eye movement features of the user; the small integrated microphone array (2) is used to collect human voice dynamics, and the gas sensor (3) is used to collect human body odor; the photoelectric sensor (4), pressure sensor (5) and infrared sensor (6) are used to collect the user's pulse wave; the numerical control chip (7) is used to store all software algorithms; The working method of the humanoid robot is: A high-resolution camera (1) is used to collect user facial recognition and eye movement features, a small integrated microphone array (2) is used to collect human voice dynamics, a gas sensor (3) is used to collect human odor, and a photoelectric sensor (4), a pressure sensor (5) and an infrared sensor (6) are used to collect the user's pulse wave; Depression outcomes are judged through the multimodal heterogeneous body surface information collected from facial expressions, eye movements, human body odor, and human pulse waves; Construct a patient corpus and use a small integrated microphone array (2) to collect real data from patient conversation records, online support group discussion content, and psychological counseling texts, annotate the emotional categories, psychological state levels, and conversational intent of the corpus, and use data enhancement technology to expand the corpus size; A large-scale corpus is used for pre-training to obtain an initial powerful large language model. Psychological knowledge is introduced in the pre-training stage of the large language model, and the model generation content is optimized through preset labels. The focus is on introducing the ability to process proprietary vocabulary in the field of mental health. The feature extraction algorithm is adjusted according to the voice characteristics of patients with depression. Combined with the specific needs of patients with depression, a variety of companionship modes including listening mode, knowledge sharing mode and guidance mode are designed to adapt to the different psychological states of patients.

2. The humanoid robot for integrated diagnosis and treatment of depression based on multi-source information perception according to claim 1, characterized in that: The high-resolution camera (1) uses a Sony DXC390P 3-CCD camera as a reference camera. The resolution of the high-resolution camera (1) is 2560×1440.

3. The humanoid robot for integrated diagnosis and treatment of depression based on multi-source information perception according to claim 1, characterized in that: The software algorithm includes a human-computer interaction and companionship module and a depression recognition module.

4. The humanoid robot for integrated diagnosis and treatment of depression based on multi-source information perception according to claim 3, characterized in that: The human-computer interaction and companionship module includes the following operating steps: S1: Construction of corpus of patients with depression; S2: Setting up automatic speech recognition module; S3: training a large language model; S4: Set up the speech synthesis module.

5. The humanoid robot for integrated diagnosis and treatment of depression based on multi-source information perception according to claim 4, characterized in that: The construction of the depression patient corpus includes data collection, data annotation and corpus expansion; The data collection includes real data of patient conversation records, online support group discussion content and psychological counseling texts. The data annotation includes emotion categories, psychological state levels and conversation intentions. The corpus expansion uses data enhancement technology to expand the corpus size.

6. The humanoid robot for integrated diagnosis and treatment of depression based on multi-source information perception according to claim 4, characterized in that: The automatic speech recognition module is used to convert the speech signal Convert to text sequence , for the sequence alignment problem, the automatic speech recognition module uses the connectionist temporal classification loss optimization: ； Among them, the conditional probability The probability of all possible alignment paths To define: ； in, : the sequence of each alignment path; : A compression function that maps the alignment path to the target sequence; : is the length of the text sequence; is the length of the speech signal; The elements of the sequence representing each alignment path at time t.

7. The humanoid robot for integrated diagnosis and treatment of depression based on multi-source information perception according to claim 4, characterized in that: The training of the large language model includes pre-training the large language model, fine-tuning the large language model and model early warning; The pre-trained large language model is used to obtain an initial powerful large language model. Psychological knowledge is introduced in the pre-training stage of the large language model, and the model generation content is optimized by preset tags. The feature extraction algorithm is adjusted according to the speech characteristics of patients with depression. In the long dialogue scenario, the long short-term memory and attention mechanism are used to retain the context information. The large language model is fine-tuned using labeled data and multi-task learning. A large number of text fragments generated by the large language model after fine-tuning are collected and sorted by human annotators. The sorting results are used as input to train the reward model. Based on the reinforcement learning of reward feedback, the model learns to distinguish between high-quality and low-quality texts, sets personalized content generation, and combines the patient's historical interaction data to generate targeted conversation content. The model warning is used to automatically record the patient's emotional fluctuations and language changes during voice interaction and analyze his or her mental health trends.

8. The humanoid robot for integrated diagnosis and treatment of depression based on multi-source information perception according to claim 4, characterized in that: The speech synthesis module realizes the natural generation of speech waveform from text during the speech synthesis process; The core is to represent the text sequence Mapping to spectrogram , and then from the spectrogram Generate time domain waveform ; Representing a text sequence Mapping to spectrogram : ； ； in, : Hidden state of the decoder; : Hidden representation generated by the text encoder; : No. Moment The attention weights of the text representations; : attention score function; Indicates the length of the text sequence, that is, the text sequence The number of elements in ; The loss function is to minimize the generated spectrogram and real spectrogram The error: ； in, Spectrogram representing the generated model At the moment slices, Represents the true spectrogram at time slices, express and The square of the Euclidean distance between them is used to quantify the difference between them; from the spectrogram Generate time domain waveform , the spectrogram Convert to time domain waveform , achieved through conditional probability modeling: ； in, :time The waveform sampling value of :exist All waveform sampling values ​​before time; : Spectrogram; Use dilated convolution to extract long sequence dependency features: ； in, Indicates time The predicted value of the waveform sampling value; : convolution kernel size; : convolution weight; : Expansion coefficient, controlling the growth of the receptive field; : Indicates time Before The waveform sampling value of time steps; Use cross-cutting loss optimization: ； in, is the model parameter.

9. The humanoid robot for integrated diagnosis and treatment of depression based on multi-source information perception according to claim 3, characterized in that: The depression recognition module obtains the facial expression depression judgment result through the convolutional neural network CNN , Eye movement features obtained through long short-term memory network LSTM depression judgment results , Analysis of depression judgment results through odor threshold , the final depression verdict The following is achieved by weighted fusion of the judgment results of three features: facial expression, eye movement and smell analysis: ； in, : The depression judgment result obtained by the deep learning model based on facial expression features, its value is In between, 1 indicates a high possibility of depression; : The depression judgment result obtained by the deep learning model based on the eye movement characteristics, and its value is between; : The result of depression judgment obtained through odor analysis, its value is between; : The depression judgment result obtained by pulse wave velocity analysis is between; : The weight of each judgment result, satisfying ; Characteristic facial expression judgment results Use convolutional neural network CNN and combine it with emotion classification task for pre-training: ； in, : Facial expression images or video frames collected by a high-resolution camera; , : weight and bias parameters; : Classification function, output classification results; among them, the activation function used by the convolutional neural network is: ； ； in, : represents the Sigmoid activation function; : represents weight; : indicates bias; : represents input data; Eye movement characteristics in the judgment of depression Processing dynamic features based on long short-term memory network LSTM: ； in, : The eye trajectory time series data collected by the high-resolution camera, including pupil diameter, eye movement speed, and gaze point coordinate changes; : represent weight and bias parameters respectively; among them, the long short-term memory network uses the group check attention mechanism: ； ； in, :Check matrix, check the space Divided into The attention scores of each group were calculated independently. : No. The query vector of the group; : key matrix, not grouped, globally shared; : Value matrix, not grouped, globally shared; : sequence length, : is the feature dimension; Smell analysis depression judgment formula, depression judgment results based on exhaled breath components The results are calculated by comparing the detected gas concentration with the threshold of the healthy control group: ; in, : The concentration of hydrogen monoxide, hydrogen sulfide and carbon monoxide in the patient's exhaled gas measured by gas sensors; : The average concentration thresholds of hydrogen monoxide, hydrogen sulfide and carbon monoxide in the gas of the healthy control group; : Standard deviation of hydrogen monoxide, hydrogen sulfide, and carbon monoxide in the healthy control group, used for standardization; : Sigmoid function, used to map the difference to the range ,and .

10. The humanoid robot for integrated diagnosis and treatment of depression based on multi-source information perception according to claim 9, characterized in that: The pulse wave velocity analysis depression judgment formula is determined based on the relationship between the collected pulse wave velocity of the patient and the average value of the healthy control group: ； in, : The patient's pulse wave velocity collected by photoelectric, pressure, and infrared sensors; : Mean pulse wave velocity of healthy control group; : Standard deviation of pulse wave velocity in healthy control group; : Sigmoid function, used to map the standardized difference to the range .

Images