Digital auxiliary system suitable for mental diseases and electronic equipment thereof

Abstract

The invention provides a digital auxiliary system suitable for mental diseases and an electronic device thereof, and the method comprises the steps: obtaining first inertial measurement information which represents the inertial measurement information of a user body; according to the first inertial measurement information, posture action information and speech information are extracted; and according to the posture action information and the speech information, performing analysis to obtain a recognition result about mental diseases. According to the invention, synchronous and long-time continuous monitoring of postures, actions and speaking of the user in the natural life and working state is realized through the inertial measurement unit information of the body of the user, and the user can conveniently and rapidly carry out real-time monitoring. The digital behavior characteristic change of the mental disease patient is jointly described from the two important dimensions of the social function, namely, the posture action and the speaking behavior, digital auxiliary diagnosis and illness state evaluation are achieved, and the serious privacy leakage risk brought by conventional video and microphone recording is avoided.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent sensor health monitoring, and in particular, relates to a digital assistance system suitable for mental illness and an electronic device thereof.

[0002] In particular, a digital auxiliary diagnosis and disease monitoring system suitable for mental illness can analyze and obtain auxiliary diagnosis and / or disease assessment identification results of mental illness, including auxiliary diagnosis identification results of depression and / or disease assessment identification results of depression. Background Art

[0003] The biological pathogenesis of mental illnesses including depression, bipolar disorder, and anxiety disorders is unclear. In clinical practice at home and abroad, the diagnosis of the disease and the assessment of its severity still rely entirely on interviews between psychiatrists and patients, just as they did decades ago. For example, the Mini-International Neuropsychiatric Interview (MINI) is used to understand the patient's mental and behavioral clinical symptoms in detail to make a diagnosis of the disease, and the Hamilton Depression Rating Scale (HAMD) is used to assess the severity of the disease. This is highly subjective, and patients suffer from the "observer effect" and recall bias. It is time-consuming and inefficient, and is a huge waste and occupation of the already scarce resources of psychiatrists.

[0004] For a long time, molecular biology and imaging technology have been used to study indicators of structural and functional changes in various systems, organs, tissues, and subcellular structures, such as inflammatory factors, neurotransmitters, neuroplasticity, and changes in brain structure and nuclear magnetic resonance imaging, in an attempt to find reliable and objective biomarkers of mental illness, but no biomarkers with high sensitivity and high specificity have been found. In recent years, with the rapid development of microelectronics technology, smart phones, and artificial intelligence, digital biomarkers such as voice, gait, sleep, heart rate variability, electroencephalogram, expression, eye movement, social network platform habits, and other digital information have been valued and developed vigorously in the field of depression research.

[0005] Patent document CN117898727A discloses a depression detection method based on gait and facial expression change analysis, including the following technical features: obtaining gait data and facial expression change video of the user to be tested when walking; performing feature image encoding processing on the gait data to obtain a gait feature image; performing feature image encoding processing on the facial expression change video to obtain a facial expression feature image; based on a trained CNN-LSTM model, processing the gait feature image and facial expression feature image to obtain a prediction result.

[0006] This existing technology needs to rely on video cameras to obtain video images, but its shortcomings are: first, the camera can only obtain short-term fragmentary information, and cannot obtain long-term continuous posture and behavior information of the patient in the natural life and work state of the "real world"; second, the camera contains sensitive privacy information.

[0007] Patent document CN118522312A discloses a depression detection and depression degree assessment system based on speech and text, including: a speech acquisition module for obtaining pre-processed speech data and transcribed text; a local feature acquisition module for obtaining speech local features and text local features; a feature enhancement module for obtaining time-series speech features and time-series text features; a pre-training module for obtaining pre-trained speech and text unimodal models and features; an automatic detection and assessment module for realizing automatic detection and degree assessment of depression. In summary, the present invention designs a multimodal model of a series-parallel hybrid structure based on physiological signals of two modes, speech and text. The two modes use the pre-training model in parallel to obtain feature coding and perform feature enhancement; two highly adaptive fusion modules are designed to realize automatic recognition and severity assessment of depression.

[0008] This existing technology relies on a microphone to obtain speech audio signals. However, the disadvantage is that the microphone can only obtain short-term fragmentary information and cannot obtain long-term continuous posture and behavior information of the patient in his or her natural life and work state. Otherwise, long-term continuous recording will bring serious privacy leakage risks to the patient and the person he or she is communicating with.

[0009] As the example above, gestures and speech are the two most critical dimensions for people to implement social functions. The onset and serious development of mental illnesses, which lead to the impairment of social functions, are basically accompanied by abnormalities in the corresponding characteristics of the patient's gestures and speech. However, so far, there is no technical solution that can be simple and convenient for patients to use (wear) for a long time, and can continuously calculate the two key dimensions of patient gestures and speech for a long time in the real world without interference in natural life and work, and thereby realize digital assistance functions for mental illness, such as digital auxiliary diagnosis and / or disease assessment functions. Summary of the invention

[0010] In view of the defects in the prior art, the object of the present invention is to provide a digital assistance system suitable for mental illness and an electronic device thereof.

[0011] A digital assistance system suitable for mental illness provided by the present invention comprises:

[0012] A first inertial measurement information acquisition module: acquires first inertial measurement information, wherein the first inertial measurement information represents inertial measurement information of the user's body;

[0013] A posture and motion information acquisition module: extracting posture and motion information according to the first inertial measurement information, wherein the posture and motion information represents the posture information and motion information of the user's body trunk;

[0014] A speech information acquisition module: extracting speech information according to the first inertial measurement information, wherein the speech information represents vibration information of the user's body caused by speaking;

[0015] The first processing module: according to the posture action information and the speech information, analyze and obtain the recognition result about the mental illness.

[0016] Preferably, the first inertial measurement information includes three-axis acceleration information and / or three-axis angular velocity information; among the three-axis acceleration and three-axis angular velocity information, one axis acceleration or one axis angular velocity is sampled at a higher frequency, and the remaining axis accelerations and axis angular velocities are sampled at a lower frequency, preferably the acceleration of the axis perpendicular to the body contact surface is sampled at a higher frequency;

[0017] The first inertial measurement information is collected from one or more parts of the user's body; the one part is the sternum part of the chest area.

[0018] Preferably, the posture action information includes posture action categories, wherein the posture action categories include: standing, sitting, lying, standing to sitting, sitting to standing, walking, running, going upstairs, going downstairs, and step frequency;

[0019] The gesture action information includes time information, wherein the time information includes the time, duration, and cumulative duration of each type of gesture action;

[0020] The posture action information and speech information are respectively extracted from the lower frequency information and the higher frequency information of the first inertial measurement information.

[0021] Preferably, the speech information includes speech features, wherein the speech features indicate: whether to speak or not, speech speed, pauses, amplitude, and fundamental frequency;

[0022] The speech information includes speech time information, wherein the speech time information includes the time when each speech feature occurs and the duration of each speech.

[0023] Preferably, in the first processing module, through the recognition model, according to the input posture and action information, the characteristics and combined characteristics presented by the speech information, the recognition results of auxiliary diagnosis and / or condition assessment of mental illness are output.

[0024] Preferably, the first inertial measurement information is information collected by an inertial measurement information sensor; the first inertial measurement information represents the inertial measurement information of the user's front chest area body surface;

[0025] The first processing module comprises:

[0026] Benchmark acquisition module: obtains the benchmark of gesture information and speech information;

[0027] Comparison module: compares the gesture information with the benchmark, compares the speech information with the benchmark, and obtains two comparison results;

[0028] Fusion module: Based on the two comparison results, the recognition result of mental illness is obtained.

[0029] According to a computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the following steps are implemented:

[0030] A first inertial measurement information acquisition step: acquiring first inertial measurement information, wherein the first inertial measurement information represents inertial measurement information of the user's body;

[0031] Posture and motion information acquisition step: extracting posture and motion information according to the first inertial measurement information, wherein the posture and motion information represents the posture information and motion information of the user's body trunk;

[0032] Speech information acquisition step: extracting speech information according to the first inertial measurement information, wherein the speech information represents vibration information of the user's body caused by speaking;

[0033] The first processing step: analyzing the gesture information and the speech information to obtain recognition results about mental illness.

[0034] An application management system according to the present invention provides an application;

[0035] The application is configured to perform the following steps when executed:

[0036] A first inertial measurement information acquisition step: acquiring first inertial measurement information, wherein the first inertial measurement information represents inertial measurement information of the user's body;

[0037] Posture and motion information acquisition step: extracting posture and motion information according to the first inertial measurement information, wherein the posture and motion information represents the posture information and motion information of the user's body trunk;

[0038] Speech information acquisition step: extracting speech information according to the first inertial measurement information, wherein the speech information represents vibration information of the user's body caused by speaking;

[0039] The first processing step: analyzing and obtaining a recognition result of a mental illness according to the gesture information and the speech information;

[0040] The first display step: displaying the identification results of mental illness.

[0041] An electronic device provided according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the following steps when executed by the processor:

[0042] A first inertial measurement information acquisition step: acquiring first inertial measurement information, wherein the first inertial measurement information represents inertial measurement information of the user's body;

[0043] Posture and motion information acquisition step: extracting posture and motion information according to the first inertial measurement information, wherein the posture and motion information represents the posture information and motion information of the user's body trunk;

[0044] Speech information acquisition step: extracting speech information according to the first inertial measurement information, wherein the speech information represents vibration information of the user's body caused by speaking;

[0045] The first processing step: analyzing and obtaining a recognition result of a mental illness according to the gesture information and the speech information;

[0046] Alternatively, the electronic device comprises a digital assistance system for mental illness according to any one of claims 1 to 6;

[0047] Alternatively, the electronic device comprises a computer-readable storage medium storing a computer program as claimed in claim 7;

[0048] Alternatively, the electronic device includes the application management system as described in claim 8.

[0049] Preferably, it also includes: an inertial measurement information sensor;

[0050] The inertial measurement information sensor can be attached to the skin surface of the user.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] 1. The present invention can analyze and process the user's posture, movement and speech characteristics through a simple inertial measurement information data source, so as to determine whether the current user suffers from a mental illness and the severity of the illness, etc.;

[0053] 2. The present invention is based on a sensor (such as an inertial measurement unit) worn on the user's chest area, which can effectively monitor the user's all-weather inertial measurement information, calculate the user's natural life and work state, the continuous posture and action information of the multiple characteristics of the multiple characteristics of speech information, these characteristics are closely related to the clinical manifestations of mental illness, and the analysis is explainable;

[0054] 3. The present invention uses an inertial measurement unit to collect body inertial measurement information caused by body trunk posture and speech vibration as the information source. It does not require image acquisition equipment or sound pickup equipment such as microphones, and can essentially avoid the risk of privacy leakage of users. On this basis, it can be simple and convenient for patients to use (wear) for a long time. It can continuously calculate the two key dimensions of patient posture and speech for a long time without interference in natural life and work, and thereby realize technical solutions such as digital assisted diagnosis and disease assessment functions of mental illnesses including depression, solving the reliable and objective digital assistance capabilities for mental illnesses that psychiatrists have long pursued. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0056] Figure 1 It is a schematic diagram of the structure of a digital assistance system suitable for mental illness provided according to the present invention.

[0057] Figure 2 It is a schematic diagram of the structure of the first processing module in the digital assistance system suitable for mental illness provided by the present invention.

[0058] Figure 3 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0059] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0060] like Figure 1 , Figure 2As shown, a digital assistance system suitable for mental illness provided by the present invention comprises:

[0061] A first inertial measurement information acquisition module: acquiring first inertial measurement information, wherein the first inertial measurement information represents inertial measurement information of the user's body; the inertial measurement information is collected by monitoring the inertial measurement information changes caused by posture and / or movement changes and / or speech vibrations through a sensor worn on the skin surface of the user's chest area;

[0062] Wherein, the first inertial measurement information represents the inertial measurement information of the user's front chest area. The first inertial measurement information includes acceleration information of multiple axes and / or angular velocity information of multiple axes. For example, the first inertial measurement information is obtained from an inertial measurement unit (IMU), and the first inertial measurement information includes XYZ three-axis acceleration and / or XYZ three-axis angular velocity. In a preferred example, the multiple accelerations and multiple angular velocity information are accelerations and angular velocities synchronized in acquisition time. The acceleration information and angular velocity information represent the acceleration information and angular velocity information of one or more parts of the user's body, and preferably obtain the acceleration information and angular velocity information of the upper and middle parts of the sternum representing the user's chest area. In the three-axis acceleration and three-axis angular velocity information, the sampling frequency of the acceleration of the axis perpendicular to the body contact surface is high-frequency sampling, preferably 1000-2000Hz, and the remaining two-axis accelerations and three-axis angular velocities are all sampled at low frequencies, preferably 50Hz.

[0063] Posture and motion information acquisition module: according to the first inertial measurement information, posture and motion information is extracted, wherein the posture and motion information represents the posture information and motion information of the user's body trunk; the part of the first inertial measurement information corresponding to the posture and motion information of the user's body trunk is collected from the sensor worn on the trunk. In a preferred embodiment, the posture and motion information of the body trunk includes any one or more of the following: standing, sitting, lying, standing to sitting, sitting to standing, walking, running, going upstairs, going downstairs, step frequency, and more posture and motion information, such as switching between various lying positions and various lying positions, walking back and forth, etc.

[0064] Speech information acquisition module: Extract speech information based on the first inertial measurement information, wherein the speech information represents the vibration information of the user's body caused by speaking; the vibration information of the user's body caused by speaking includes any one or more of the vibration information of the user's torso caused by speaking, the vibration information of the user's head caused by speaking, and the vibration information of the user's shoulder and neck caused by speaking; accordingly, the part of the first inertial measurement information corresponding to the vibration information of the user's body caused by speaking is collected from any one or more of the sensors worn on the torso, head, shoulder and neck. The speech information includes any one or more of the following: whether to speak, speech speed, pauses, amplitude, fundamental frequency, and more time domain, frequency domain, power spectrum, inverse spectrum and other features.

[0065] The first processing module analyzes the posture and action information and the speech information to obtain a recognition result about the mental illness.

[0066] More specifically, the first processing module: analyzes the recognition results of mental illnesses such as depression based on the posture and action information and the speech information. Through the recognition model, according to the characteristics and combined features presented by the input posture and action information and speech information, the recognition results of auxiliary diagnosis and / or disease assessment of mental illness are output. Specifically, in the first processing module, according to the posture and action information and the speech information, it is analyzed whether it is depression or anxiety, etc.; the degree of change of the collected human posture and action information and the information caused by vibration is analyzed to further identify the severity of the disease. For example, through a pre-trained posture and action-speech-based intelligent recognition model, according to the characteristics and combined features presented by the input posture and action information and speech information, the recognition results of auxiliary diagnosis and disease assessment of mental illnesses including depression are output.

[0067] The first processing module preferably includes:

[0068] Benchmark acquisition module: acquires a benchmark for posture, motion information and speech information; in a preferred example, the benchmark can be configured, for example, as features of the posture, motion information and speech information of a normal population, or as judgment features of the posture, motion information and speech information of a specific mental illness (such as judgment features of depression, anxiety, and bipolar disorder), or as a historical benchmark for the patient's own posture, motion information and speech information (for example, based on features of the patient's posture, motion information and speech information in a healthy state, or based on features of the patient's posture, motion information and speech information in an ill state).

[0069] Comparison module: compares the gesture information with the benchmark, and compares the speech information with the benchmark to obtain two comparison results; wherein the comparison includes comparison between information features;

[0070] Fusion module: Based on the two comparison results, the recognition result of mental illness is obtained.

[0071] The present invention will be described in more detail below.

[0072] If a person's gesture information and speech information are compared with a "gesture-speech" model established based on normal people and / or mental illness patients and / or their own past data as the benchmark:

[0073] In terms of posture and action information dimension:

[0074] If the calculation shows an increase in static postures, especially a large number of semi-recumbent postures during daytime working hours, and a decrease in movements such as walking and running, and a longer transition time from standing to sitting and from sitting to standing, a slower walking frequency, and an increased swaying when walking, then it indicates the presence of psychomotor retardation, a higher possibility of a new onset or relapse of depression, or a more serious condition;

[0075] If the calculation shows that the number of transitions from standing to sitting or from sitting to standing increases (i.e. fidgeting, a typical manifestation of anxiety), the time for posture transitions decreases, and the number of back-and-forth movements increases, then the possibility of anxiety or mania increases;

[0076] If the calculation shows that the lying position changes frequently in the early morning, and / or the sitting and standing positions appear, it indicates early awakening, which is one of the characteristics of depressive insomnia;

[0077] If the calculation shows that the patient lies down for a long time during the day and rarely changes his / her lying position, it indicates that the possibility of daytime sleepiness increases;

[0078] The greater the degree of change in posture and movement (ie, compared with the normal population, and / or a benchmark established by the patient's own data), the more serious the condition; that is, the greater the deviation from the normal benchmark, the more serious the condition.

[0079] In terms of speech information dimension:

[0080] If the calculation shows that the amount of speech decreases, the speed slows down, the pauses increase, the volume decreases, and the volume variability decreases, it indicates that the possibility of depression increases;

[0081] If the calculation shows that the speech volume increases, the speaking speed increases, the pauses decrease, and the volume increases, it indicates different states of anxiety or mania;

[0082] The greater the degree of change in the speech information (i.e., compared with the normal population and / or the benchmark established by the own data), the more serious the disease is; that is, the greater the degree of deviation from the normal benchmark, the more serious the disease is. More specifically, in other preferred examples:

[0083] For a case where the benchmark is configured as the features of the normal population’s posture motion information and speech information, the greater the deviation from the benchmark, the more serious the condition;

[0084] In the case where the benchmark is configured as the judgment features of the gesture information and speech information of a specific mental illness, the smaller the deviation from the benchmark, the more serious the illness;

[0085] For a benchmark configured as a historical benchmark of the patient's own posture, movement, and speech information when the patient is in an illness state, the smaller the deviation from the benchmark, the more serious the illness.

[0086] The present invention is further described in detail below.

[0087] Single-dimensional information of gesture and movement information or single-dimensional information of speech information is likely to cause defects of low specificity or low auxiliary diagnosis efficiency in assisting diagnosis of mental illness with unknown etiology;

[0088] For example, modern work pressure is high, and there is a general phenomenon of more static postures and less movement. For example, some normal people are born with more static postures and less movement, or they are born to speak slowly, with low volume and gentleness. Relying on single posture and movement information or single speech information increases the probability of misjudgment.

[0089] Therefore, the "fusion" of posture and movement information dimension and speech information dimension will be more efficient. Analyzing the two dimensions together will significantly improve the accuracy of auxiliary diagnosis and disease assessment. For example, the fusion of two-dimensional information can further distinguish common clinical subtypes of depression, such as melancholic depression and depression with anxiety and pain.

[0090] Melancholic depression: For a continuous period of time, calculations found that the posture and movements showed "inhibition" in all aspects, that is, a large number of semi-recumbent postures appeared during daytime working hours, movements and behaviors such as walking and running decreased, and the time for transitions from standing to sitting and from sitting to standing was prolonged, the walking frequency slowed down, the swaying of walking increased, and early awakening occurred; the amount of speech decreased, the speed slowed down, the pauses increased, the volume decreased, and the volume variation decreased; and this posture, movement and speech characteristics showed the characteristics of "worsening in the morning and easing in the afternoon"; for a continuous period of time, this state was manifested every day.

[0091] If, based on the changes in posture, movement and speech of depression, it is also calculated that there are more fragmentary standing-to-sitting and sitting-to-standing transitions, the time of posture transitions is shortened, the number of back-and-forth movements is increased, the amount of speech is increased, the speed of speech is increased, the pauses are reduced, and the volume is increased, then it is suggested that the patient should consider the "depression with anxiety and distress" subtype;

[0092] In more preferred examples, the posture action information includes posture action categories, wherein the posture action categories include: standing, sitting, lying, standing to sitting, sitting to standing, walking, running, going upstairs, going downstairs, step frequency, and more posture action information, such as switching between various lying positions and various lying positions, walking back and forth, etc.; the posture action information includes time information, wherein the time information includes the time, duration, cumulative duration of each category of posture action, and more time dimension statistical features, etc.;

[0093] More specifically, the speech information acquisition module: extracts speech information based on the first inertial measurement information, wherein the speech information represents the vibration information of the user's body caused by the vibration of the vocal cords when speaking; further extracts information caused by the vibration of the human body due to speaking through the sensor; such vibration is usually weak and has a higher frequency. In more preferred examples, the speech information includes speech features, wherein the speech features indicate: whether to speak or not, speech speed, pauses, amplitude, fundamental frequency, and more time domain, frequency domain, power spectrum, inverse spectrum and other features; the speech information includes speech time information, wherein the speech time information includes the time when each speech feature occurs, the duration of each speech, and more statistical parameter features of time information.

[0094] Preferred examples of the present invention are described below.

[0095] In a preferred example of the present invention, a multimodal neural network recognition that fuses gestures, movements and speech features may be used; for example, the onset characteristics of depression patients are different from those of normal people.

[0096] The present invention also provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the following steps are implemented:

[0097] A first inertial measurement information acquisition step: acquiring first inertial measurement information, wherein the first inertial measurement information represents inertial measurement information of the user's body;

[0098] Posture and motion information acquisition step: extracting posture and motion information according to the first inertial measurement information, wherein the posture and motion information represents the posture information and motion information of the user's body trunk;

[0099] Speech information acquisition step: extracting speech information according to the first inertial measurement information, wherein the speech information represents vibration information of the user's body caused by speaking;

[0100] The first processing step: analyzing the gesture information and the speech information to obtain a recognition result of a mental illness; the first processing step includes:

[0101] Benchmark acquisition step: acquiring a benchmark of gesture information and speech information;

[0102] Comparison step: comparing the gesture information with the reference, comparing the speech information with the reference, and obtaining two comparison results;

[0103] Fusion step: Analyze the two comparison results together to obtain the recognition result of mental illness.

[0104] The first inertial measurement information is information collected by an inertial measurement information sensor. The first inertial measurement information represents inertial measurement information of the user's chest area.

[0105] An application management system according to the present invention provides an application;

[0106] The application is configured to implement the following steps when executed, such as Figure 3 As shown:

[0107] A first inertial measurement information acquisition step: acquiring first inertial measurement information, wherein the first inertial measurement information represents inertial measurement information of the user's body;

[0108] Posture and motion information acquisition step: extracting posture and motion information according to the first inertial measurement information, wherein the posture and motion information represents the posture information and motion information of the user's body trunk;

[0109] Speech information acquisition step: extracting speech information according to the first inertial measurement information, wherein the speech information represents vibration information of the user's body caused by speaking;

[0110] The first processing step: analyzing and obtaining a recognition result of a mental illness according to the gesture information and the speech information;

[0111] The first display step: displaying the identification results of mental illness.

[0112] The first processing step comprises:

[0113] Benchmark acquisition step: acquiring a benchmark of gesture information and speech information;

[0114] Comparison step: comparing the gesture information with the reference, comparing the speech information with the reference, and obtaining two comparison results;

[0115] Fusion step: Analyze the two comparison results together to obtain the recognition result of mental illness.

[0116] An electronic device provided according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the following steps when executed by the processor:

[0117] A first inertial measurement information acquisition step: acquiring first inertial measurement information, wherein the first inertial measurement information represents inertial measurement information of the user's body;

[0118] Posture and motion information acquisition step: extracting posture and motion information according to the first inertial measurement information, wherein the posture and motion information represents the posture information and motion information of the user's body trunk;

[0119] Speech information acquisition step: extracting speech information according to the first inertial measurement information, wherein the speech information represents vibration information of the user's body caused by speaking;

[0120] The first processing step: analyzing and obtaining a recognition result of a mental illness according to the gesture information and the speech information;

[0121] Alternatively, the electronic device includes the digital assistance system applicable to mental illness;

[0122] Alternatively, the electronic device includes the computer-readable storage medium storing the computer program;

[0123] Alternatively, the electronic device includes the application management system.

[0124] The first processing step comprises:

[0125] Benchmark acquisition step: acquiring a benchmark of gesture information and speech information;

[0126] Comparison step: comparing the gesture information with the reference, comparing the speech information with the reference, and obtaining two comparison results;

[0127] Fusion step: Analyze the two comparison results together to obtain the recognition result of mental illness.

[0128] The electronic device further comprises: an inertial measurement information sensor; the inertial measurement information sensor can be attached to the skin surface of the user.

[0129] Those skilled in the art know that, in addition to realizing the system and its various devices, steps, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to implement the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, steps, and units provided by the present invention can be considered as a hardware component, and the devices, steps, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, steps, and units for realizing various functions can also be regarded as both software steps for realizing the method and structures within the hardware component.

[0130] The present invention uses the information from the user's body inertial measurement unit to achieve synchronous, long-term continuous monitoring of the user's posture, movements, and speech behaviors in natural life and work states. It can use a simple method to jointly depict the changes in digital behavioral characteristics of patients with mental illnesses, including depression, from two very important dimensions of posture, movement, and speech behavior, which are very important in social functions, to achieve digital-assisted diagnosis and disease assessment, and avoid the serious privacy leakage risks brought by conventional video and microphone recordings.

[0131] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A digital assistance system for mental illness, characterized in that: include: A first inertial measurement information acquisition module: acquires first inertial measurement information, wherein the first inertial measurement information represents inertial measurement information of the user's body; A posture and motion information acquisition module: extracting posture and motion information according to the first inertial measurement information, wherein the posture and motion information represents the posture information and motion information of the user's body trunk; A speech information acquisition module: extracting speech information according to the first inertial measurement information, wherein the speech information represents vibration information of the user's body caused by speaking; The first processing module: according to the posture action information and the speech information, analyze and obtain the recognition result about the mental illness.

2. The digital assistance system for mental illness according to claim 1, characterized in that: The first inertial measurement information includes three-axis acceleration information and / or three-axis angular velocity information; among the three-axis acceleration and three-axis angular velocity information, one axis acceleration or one axis angular velocity is sampled at a higher frequency, and the remaining axis accelerations and axis angular velocities are sampled at a lower frequency, and the acceleration of the axis perpendicular to the body contact surface is sampled at a higher frequency; The first inertial measurement information is collected from one or more parts of the user's body; the one part is the sternum part of the chest area.

3. The digital assistance system for mental illness according to claim 1, characterized in that: The gesture action information includes gesture action categories, wherein: The posture action categories include: standing, sitting, lying, standing to sitting, sitting to standing, walking, running, going upstairs, going downstairs, and step frequency; The gesture action information includes time information, wherein the time information includes the time, duration, and cumulative duration of each type of gesture action; The posture action information and speech information are respectively extracted from the lower frequency information and the higher frequency information of the first inertial measurement information.

4. The digital assistance system for mental illness according to claim 1, characterized in that the speech information includes speech features, wherein: The speech characteristics indicate: whether to speak, speech speed, pauses, amplitude, and fundamental frequency; The speech information includes speech time information, wherein the speech time information includes the time when each speech feature occurs and the duration of each speech.

5. According to claim 1, the digital assistance system suitable for mental illness is characterized in that, in the first processing module, through the recognition model, according to the characteristics and combined characteristics presented by the input posture action information and speech information, the recognition results of auxiliary diagnosis and / or condition assessment of the mental illness are output.

6. The digital assistance system for mental illness according to claim 1, characterized in that: The first inertial measurement information is information collected by an inertial measurement information sensor; the first inertial measurement information represents inertial measurement information of the user's front chest area body surface; The first processing module comprises: Benchmark acquisition module: obtains the benchmark of gesture information and speech information; Comparison module: compares the gesture information with the benchmark, compares the speech information with the benchmark, and obtains two comparison results; Fusion module: Based on the two comparison results, the recognition result of mental illness is obtained.

7. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the following steps are implemented: A first inertial measurement information acquisition step: acquiring first inertial measurement information, wherein the first inertial measurement information represents inertial measurement information of the user's body; Posture and motion information acquisition step: extracting posture and motion information according to the first inertial measurement information, wherein the posture and motion information represents the posture information and motion information of the user's body trunk; Speech information acquisition step: extracting speech information according to the first inertial measurement information, wherein the speech information represents vibration information of the user's body caused by speaking; The first processing step: analyzing the gesture information and the speech information to obtain recognition results about mental illness.

8. An application management system, characterized in that: Provide applications; The application is configured to perform the following steps when executed: A first inertial measurement information acquisition step: acquiring first inertial measurement information, wherein the first inertial measurement information represents inertial measurement information of the user's body; Posture and motion information acquisition step: extracting posture and motion information according to the first inertial measurement information, wherein the posture and motion information represents the posture information and motion information of the user's body trunk; Speech information acquisition step: extracting speech information according to the first inertial measurement information, wherein the speech information represents vibration information of the user's body caused by speaking; The first processing step: analyzing and obtaining a recognition result of a mental illness according to the gesture information and the speech information; The first display step: displaying the identification results of mental illness.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program implements the following steps when executed by the processor: A first inertial measurement information acquisition step: acquiring first inertial measurement information, wherein the first inertial measurement information represents inertial measurement information of the user's body; Posture and motion information acquisition step: extracting posture and motion information according to the first inertial measurement information, wherein the posture and motion information represents the posture information and motion information of the user's body trunk; Speech information acquisition step: extracting speech information according to the first inertial measurement information, wherein the speech information represents vibration information of the user's body caused by speaking; The first processing step: analyzing the gesture information and the speech information to obtain a recognition result of a mental illness; Alternatively, the electronic device comprises a digital assistance system for mental illness according to any one of claims 1 to 6; Alternatively, the electronic device comprises a computer-readable storage medium storing a computer program as claimed in claim 7; Alternatively, the electronic device includes the application management system as described in claim 8.

10. The electronic device according to claim 9, characterized in that: Also includes: Inertial measurement information sensor; The inertial measurement information sensor can be attached to the skin surface of the user.

Images