Wearable mental disorder automatic diagnosis system and method based on comparative learning

Abstract

The invention discloses a wearable mental disorder automatic diagnosis system and method based on comparative learning. The system is realized by adopting a wearable device, and comprises a data acquisition unit used for acquiring multi-modal physiological data of a user; the user registration unit is used for executing the following steps under the condition that the user is judged to be a new user: performing fine tuning on a first feature encoder subjected to offline pre-training in a self-supervised comparative learning manner by utilizing multi-modal physiological data to obtain a second feature encoder; extracting data features of the labeled multi-modal physiological data through a second feature encoder; training a personalized classifier by taking the data features as input to obtain a mental disorder recognition model; and the recognition unit is used for obtaining a recognition result by utilizing the mental disorder recognition model under the condition that the user is judged to be a non-new user. The psychological state of the user can be detected in real time, so that mental disorder symptoms are intervened and relieved as soon as possible.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology, and more specifically, to a wearable mental disorder automatic diagnosis system and method based on contrastive learning. Background Art

[0002] For diseases such as depression, anxiety, and mood disorders, current researchers use data of different modalities to identify mental disorders, such as text information in the form of questionnaires, behavioral information captured by mobile sensing technology and ecological instantaneous assessment, and deep physiological information obtained using dedicated medical equipment. However, these identification methods have problems such as privacy leakage, high cost of dedicated equipment, difficulty in obtaining data labels, and low real-time performance. In recent years, wearable devices have developed rapidly, and can obtain a large amount of shallow unlabeled physiological information of users anytime and anywhere, such as heart rate variability and blood oxygen saturation.

[0003] In the prior art, the schemes for identifying different mental disorders are mainly divided into three categories. The first category is the traditional identification method using questionnaires or special medical equipment. These methods face problems such as privacy leakage, high cost, and non-universality. The second category is to use ubiquitous equipment to collect user behavior information for mental disorder identification and assessment. This method generally obtains user activity, sleep status, etc., and has the problem of low real-time performance. The third category is to use self-supervised learning to identify mental disorders. This method usually obtains the user's unimodal signal for pre-training, and has problems such as single modality, difficulty in obtaining data labels, and no joint identification of multiple types of mental disorders.

[0004] Studies have shown that mental disorders usually cause physiological reactions, resulting in changes in shallow physiological information. Shallow physiological information directly reflects the body's immediate physiological state, so that the user's mental state can be monitored in a timely manner. Currently, the use of contrastive learning to identify mental disorders is developing rapidly. However, this method does not use multimodal fusion to construct positive and negative sample pairs, and these technical studies usually only target one disease. Because different mental disorders have common symptoms, it is difficult to identify which disease an individual suffers from.

[0005] After analysis, it is found that the existing solutions for identifying mental disorders have the following main defects:

[0006] 1) Most of the research on mental disorder identification focuses on one mental disorder, and rarely on multiple mental disorders. Therefore, users can only determine whether they have the mental disorder being studied, but cannot determine whether they have other mental disorders, which affects users' early detection and worsens their condition.

[0007] 2) Using dedicated equipment to collect users' deep physiological information is not only costly, complex to operate, and not universal, but also cannot meet the needs of real-time monitoring in daily life.

[0008] 3) Using mobile sensing technology to collect user behavior information usually requires collecting user behavior information for a long time, and then analyzing this information to obtain results. It has low real-time performance and it is difficult to capture the user's instantaneous psychological state.

[0009] 4) In the current schemes using self-supervised contrastive learning, the contrastive learning strategy used usually only targets one modality of data. This single modality has single physiological information and cannot mine deeper information, resulting in poor recognition performance.

[0010] In summary, there are few studies at home and abroad on the recognition of multiple mental disorders based on the physiological data collected by pervasive wearable devices under multimodal feature fusion. In addition, existing studies require a large amount of labeled data and do not support user personalization. Summary of the invention

[0011] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a wearable automatic diagnosis method and system for mental disorders based on contrastive learning.

[0012] According to a first aspect of the present invention, a wearable automatic diagnosis system for mental disorders based on contrastive learning is provided. The system is implemented using a wearable device and includes:

[0013] Data acquisition unit: used to obtain multimodal physiological data of users;

[0014] A user registration unit is used to, when determining that the user is a new user, execute: using the multimodal physiological data to fine-tune the first feature encoder pre-trained offline in a self-supervised contrastive learning manner to obtain a second feature encoder; extracting data features from the stored multimodal physiological data with labels through the second feature encoder; using the data features as input to train a personalized classifier to obtain a mental disorder recognition model, wherein the mental disorder recognition model is constructed based on the second feature encoder and the trained personalized classifier;

[0015] Identification unit: used to obtain identification results by using the stored mental disorder identification model when it is determined that the user is not a new user.

[0016] According to a second aspect of the present invention, a wearable automatic diagnosis method for mental disorders based on contrastive learning is provided. The method comprises the following steps:

[0017] Acquire the multimodal physiological data of the user and determine whether the user is a new user;

[0018] In the case where the user is determined to be a new user, executing: using the multimodal physiological data to fine-tune the first feature encoder pre-trained offline in a self-supervised contrastive learning manner to obtain a second feature encoder; extracting data features from the stored labeled multimodal physiological data via the second feature encoder; using the data features as input to train a personalized classifier to obtain a mental disorder recognition model, wherein the mental disorder recognition model is constructed based on the second feature encoder and the trained personalized classifier;

[0019] When it is determined that the user is not a new user, the stored mental disorder recognition model is used to obtain a recognition result.

[0020] Compared with the prior art, the advantage of the present invention is that the wearable automatic diagnosis scheme for mental disorders based on contrastive learning and multimodal physiological information fusion is generally divided into two stages. In the first stage, a large number of unlabeled shallow physiological information of patients and normal people are used to pre-train feature encoders in a way of integrating self-supervised contrastive learning; in the second stage, a very small amount of labeled shallow physiological information is used to fine-tune the feature encoder of the first stage and obtain a personalized model specific to each user. The present invention uses pervasive wearable devices to collect users' shallow unlabeled physiological information and integrates contrastive learning to identify multiple mental disorders, achieving the purpose of real-time monitoring, early detection and intervention.

[0021] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0023] Figure 1 is an overall framework diagram of a wearable automatic diagnosis solution for mental disorders based on contrastive learning according to an embodiment of the present invention;

[0024] Figure 2 is a framework diagram of a wristband-type smart wearable device according to an embodiment of the present invention;

[0025] Figure 3 is an appearance diagram of a wristband-type smart wearable device according to an embodiment of the present invention;

[0026] Figure 4 is an internal structure diagram of a wristband-type smart wearable device according to an embodiment of the present invention;

[0027] Figure 5 is a schematic diagram of a sampling rule used according to an embodiment of the present invention;

[0028] Figure 6 is a schematic diagram of a backbone network and a multi-modal fusion strategy according to an embodiment of the present invention;

[0029] Figure 7 is a new user registration flow chart according to an embodiment of the present invention;

[0030] Figure 8 is a schematic diagram of a process for identifying a mental disorder according to an embodiment of the present invention;

[0031] Fig. 9 is a schematic diagram of mental disorder identification results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0032] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0033] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0034] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0035] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0036] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0037] In general, the provided wearable automatic diagnosis system for mental disorders based on contrastive learning is implemented by a wearable device, including: a data acquisition unit, which is used to obtain multimodal physiological data of a user; a user registration unit, which is used to perform, when the user is judged to be a new user: using the multimodal physiological data to fine-tune the first feature encoder pre-trained offline in a self-supervised contrastive learning manner to obtain a second feature encoder; extracting data features from the stored multimodal physiological data with labels through the second feature encoder; using the data features as input to train a personalized classifier to obtain a mental disorder recognition model, which includes a second feature encoder and the trained personalized classifier; an identification unit, which is used to obtain a recognition result by using the stored mental disorder recognition model when the user is judged to be a non-new user. Each unit in the system can be implemented by a general-purpose processor, a dedicated processor or an FPGA, a general-purpose sensor, etc. in combination with software. In addition, each unit can be divided in a variety of ways, as long as the functions of the present invention can be achieved.

[0038] The present invention combines ubiquitous wearable devices and proposes a multimodal shallow physiological information fusion mental disorder identification system based on contrastive learning. The system is generally divided into a two-stage training process. In the first stage, the pre-training stage, the cloud pre-trains a feature encoder with a large amount of shallow unlabeled physiological information of different users in a contrastive learning manner. In the second stage, users collect unlabeled shallow physiological information anytime and anywhere through ubiquitous wearable devices. This information is used to personalize the user model through the feature encoder pre-trained in the first stage to identify whether the user suffers from a mental disorder. Contrastive learning learns high-quality and recognizable feature representations by maximizing the similarity of positive samples and minimizing the similarity of negative samples, so that the model can more clearly distinguish the subtle differences between different mental disorders.

[0039] Combination Figure 1As shown, the wearable automatic diagnosis of mental disorders based on contrastive learning is divided into two stages. In the first stage, a large amount of unlabeled shallow physiological information collected by smart wearable devices is used to pre-train a general feature encoder in the cloud in a multimodal fusion contrastive learning manner, and then the general feature encoder is deployed to the smart wearable device. In the second stage, when a new user uses the smart wearable device, a small amount of unlabeled shallow physiological information of the new user is first collected and combined with the model parameters and a small amount of data previously imported by the smart wearable device to quickly fine-tune the model suitable for the new user, so that the personalized model for the new user is customized. Next, the smart wearable device collects the user's shallow physiological information anytime and anywhere, and after data preprocessing, it is sent to the personalized model, and the user's physiological indicators and the results of identifying whether he suffers from mental disorders can be displayed on the smart wearable device. These data and results can also be transmitted to other devices for display through wireless transmission protocols (such as Bluetooth protocols). If the user suffers from a mental disorder, the voice module of the smart wearable device can also issue an alarm.

[0040] In one embodiment, a wristband-type smart wearable device is used, which uses STM32 as the main control board to realize the functions of accessing various physiological sensors and running system algorithms. The smart wearable device is equipped with a display screen to display recognition results and physiological indicators. This information can also be transmitted to a terminal device via Bluetooth for intuitive observation by the user, such as a mobile phone or tablet computer.

[0041] Figure 2 This is the overall framework diagram of the wristband-type smart wearable device, which generally includes a data acquisition function board and a main control board. The data acquisition function board is equipped with a bioelectrical impedance sensor, a blood oxygen sensor, a photoelectric volume pulse wave sensor, a skin charge sensor, a skin temperature sensor, and a display module. The main control board is equipped with a microcontroller, a voice module, a Bluetooth module, and a power module. The main control board has a communication connection with each module on the data acquisition function board.

[0042] For example, the bioelectrical impedance sensor is connected to the main control board through the analog input pin, and the analog-to-digital converter (ADC) of the main control board reads the value of the breathing information. The main control board and the blood oxygen sensor communicate asynchronously through the serial port of the main control board to read the blood oxygen saturation data. Between the main control board and the photoelectric volume pulse wave sensor, the analog-to-digital converter (ADC) in the main control board is used to read the value of the heart rate variability. Between the main control board and the skin electrical sensor, the analog-to-digital converter (ADC) in the main control board is used to read the value of the skin conductivity change, thereby achieving the purpose of monitoring the skin electrical response (EDA). Between the main control board and the epidermal temperature sensor, the analog-to-digital converter (ADC) in the main control board is used to read the epidermal temperature value. Between the main control board and the Bluetooth module, the UART serial port protocol is used to transmit the data of each sensor to the Bluetooth BLE module every 50ms, and the Bluetooth BLE module is transmitted to other terminals, for example, through the Bluetooth 5.3 protocol. The main control board and the display screen module use the I2C (or IIC) bus of the main control board for low-speed serial communication, so that the values ​​of each sensor and the processing results are displayed on the display screen. When the main control board receives the data from each sensor and the output results of the model, it sends instructions and data to the voice broadcast module through the UART serial port of the main control board to broadcast the overall data situation of the monitoring time period and the final recognition results.

[0043] Figure 3 This is the overall appearance of the wristband wearable device. Figure 4 The figure below is its internal structure diagram. A display screen is set on the top of the device to display the values ​​of each sensor, model recognition results, etc. processed by the main control board. Since these sensors need to be close to the skin, they can be embedded in the bottom of the device and close to the human wrist. The Bluetooth module and voice broadcast module are embedded in the circuit of the main control board.

[0044] In the following, the processes of pre-training, customizing personalized models, and model application will be described in detail.

[0045] 1. Pre-training feature encoder stage

[0046] In daily life, it is easy to obtain unlabeled physiological data of individuals, but it is difficult to obtain labeled multimodal physiological information of individuals with mental disorders. Different mental disorders have common symptoms, so it is difficult to identify which disease an individual suffers from. Contrastive learning learns high-quality, recognizable feature representations by maximizing the similarity of positive samples and minimizing the similarity of negative samples, allowing the model to more clearly distinguish the subtle differences between different mental disorders. Contrastive learning uses the model pre-trained in a self-supervised manner using unlabeled data in the source domain dataset, and can be quickly generalized to different individuals through supervised fine-tuning with a small amount of labeled data.

[0047] 1. Sampling of positive and negative sample pairs

[0048] A key aspect of contrastive learning is the sampling of positive and negative sample pairs. The selection of positive and negative sample pairs in standard contrastive learning only coarsely divides the data into positive and negative sample pairs in the time dimension, but does not fine-grainedly divide the data into positive and negative sample pairs in different modal physiological data. In one embodiment of the present invention, coarse-grained and fine-grained positive and negative sample pairs are combined to divide multimodal physiological data, thereby enhancing the representation capability and achieving more comprehensive feature capture.

[0049] Figure 5 The sampling rules of positive and negative samples in contrastive learning are illustrated. In the case of time alignment, a modality is selected as the anchor modality at time T1, and the physiological data of other modalities at the same time are regarded as positive samples. The samples of the anchor modality at different time points are regarded as strong negative samples, and the samples of other modalities at different time points are regarded as weak negative samples. The ultimate goal of contrastive learning is to maximize the similarity between positive samples and minimize the similarity between negative samples.

[0050] For example, coarse-grained sampling can be seen as being implemented on the time dimension of a 1s sample window. Positive samples are different modalities within the 1s window at the same moment, and negative samples are 1s windows at different moments. Fine-grained sampling is implemented on the internal modality of the 1s window. Positive samples are mutual sampling of different modalities at the same moment. Negative samples are divided into two types: one is sampling of the same modality at different moments, called strong negative samples, and the other is sampling of different modalities at different moments, called weak negative samples. Combining this sampling method with residual variable convolution fusion, coarse-grained sample representation along the time dimension and fine-grained representation across different physiological modalities are achieved. This sampling method significantly improves the accuracy of the recognition results.

[0051] 2. Backbone network and multimodal fusion strategy

[0052] Variable convolution learns the offset so that the convolution kernel can adaptively adjust its sampling position to capture local details. When processing time series data, variable convolution can flexibly adapt to the different rhythms and fluctuations in time series physiological data and capture more accurate time series features. When fusing multimodal physiological data, variable convolution can capture the dynamic changes between time points in different modalities or the same modality, thereby enhancing the feature extraction effect and more accurately fusing data from different modalities.

[0053] In combination with the above sampling method, in one embodiment, a residual variable convolutional network fusion method is proposed to obtain local fusion features of physiological data of different modalities and global unique features of each modality. Figure 6It is a schematic diagram of the backbone network and multimodal fusion strategy, where SPO2 is blood oxygen saturation, BIA is respiratory data, EDA is skin electrical response data, SKT is skin temperature data, and PPG is heart rate variability data. For example, a 1s window is used as the sample size for selecting physiological information of different modalities. First, each modality passes through a different bidirectional long short-term memory network (BiLSTM) layer to extract the unique time series features of each modality. After passing through the linear layer (Linear), these unique features are spliced ​​in the channel dimension to obtain a vector of (batch size, 5, 100). Then, the spliced ​​vector is sent to the residual variable convolutional network (DCN) to extract local fusion features and global unique features of each modality. Different modal physiological data are learned together with coarse and fine granularity, combined with residual variable convolutional network fusion, which is conducive to better representation of the characteristics of patients with each disease and the control group.

[0054] A large amount of unlabeled shallow physiological information is processed by the filter group to pre-train a feature encoder f in the first stage. This stage can be trained in the cloud or server, thus saving computing resources of smart wearable devices.

[0055] In summary, the present invention uses residual variable convolution for multimodal fusion, thereby extracting local fusion features and global unique features of each modality. The physiological information of different channels is fused together in the form of a residual variable convolutional network, which not only retains the global information of each channel, but also retains the local information of the fused modality. Physiological data of different modalities are jointly learned with coarse and fine granularity combined with residual variable convolutional network fusion, so that the system can better represent the characteristics of patients with each disease and the control group.

[0056] 2. New User Personalization Model Stage

[0057] After pre-training the feature encoder, the trained feature encoder and a small amount of labeled multimodal physiological data are deployed to the wearable device. In daily life scenarios, new users need to register when using this system to obtain a personalized customized model for the new user. The registration process is as follows: Figure 7As shown in the figure, in this process, the wearable device worn by the user dynamically collects multimodal physiological data for a period of time (such as a few minutes), and then the local software end preprocesses these data. The processed multimodal physiological data also fine-tunes the feature encoder f from the pre-training stage in a self-supervised comparative learning manner. After fine-tuning, a new feature encoder g is obtained. Next, the local software end randomly selects a small amount of labeled multimodal physiological data within the system, passes the labeled data through the encoder g to obtain features, and then inputs these features into the classifier composed of a lightweight transformer, a multi-layer perceptron (MLP), a Dropout layer, and a Softmax layer. Finally, the feature encoder g1 and classifier c1 belonging to the new user are personalized, so that the new user has completed the registration, and the user's physiological indicators and mental state can be monitored in real time on a daily basis to determine whether the user suffers from a mental disorder.

[0058] If the new user has been wearing the device during the customization process, then within a few minutes after the customization is completed, the user can get a conclusion on whether he or she is ill. If the user does not wear the device, then the user needs to wear the device to collect multimodal physiological data for ten minutes, for example, and then get a conclusion on whether he or she is ill.

[0059] In summary, the present invention uses a filter group combining Butterworth bandpass filtering and bandstop filtering to preprocess multimodal physiological information, and then uses a deep learning model to identify different types of mental disorders, which not only improves the recognition accuracy but also saves the resources of wearable devices.

[0060] III. Systematic Identification of Mental Disorders

[0061] The process of systematically identifying mental disorders is as follows Figure 8 As shown. First, in the cloud, with the help of a large amount of unlabeled multimodal shallow physiological information processed by the filter group, a feature encoder is pre-trained in a self-supervised contrastive learning manner, and this feature encoder and a small amount of labeled multimodal shallow physiological information processed by the filter group are embedded in the smart wearable device. Then, when a user uses the system, first determine whether it is a new user. If it is a new user, the new user must be registered, and the unlabeled shallow physiological information of the new user is collected through the smart wearable device to fine-tune the feature encoder inside the device. Then, a model belonging to the new user is trained with a small amount of labeled shallow physiological information inside, so that the physiological information of the new user can be monitored and whether he has a mental disorder can be identified; if it is not a new user, the physiological information is directly monitored and whether he has a mental disorder can be identified.

[0062] Through this system, users can understand their mental and physical states anytime and anywhere. Smart wearable devices can also send recognition results and physiological indicator information to the mobile phone via the Bluetooth module to better analyze changes in their mental and physical states; if the system recognizes that the result is a type of mental disorder, the recognition result can be broadcast through the voice module to remind the user in time. In this way, mental disorders can be detected and intervened at an early stage, thereby alleviating psychological and physical symptoms.

[0063] Accordingly, the present invention also provides a wearable automatic diagnosis method for mental disorders based on contrastive learning. The method includes the following steps: obtaining multimodal physiological data of a user and determining whether the user is a new user; if the user is determined to be a new user, executing: using the multimodal physiological data to fine-tune the first feature encoder pre-trained offline in a self-supervised contrastive learning manner to obtain a second feature encoder; extracting data features from the stored labeled multimodal physiological data through the second feature encoder; using the data features as input to train a personalized classifier to obtain a mental disorder recognition model, which is constructed based on the second feature encoder and the trained personalized classifier; if the user is determined to be a non-new user, using the stored mental disorder recognition model to obtain a recognition result.

[0064] Furthermore, in order to verify the feasibility of the present invention, a preliminary experiment was conducted. In the experiment, 18 patients with depression, 20 patients with anxiety, 16 patients with mood disorders, and 18 normal people were used to obtain the test accuracy of each person in each category using the leave-one-out method. The results are shown in Figure 2. Fig. 9 As shown, Fig. 9 (a) is the recognition accuracy of depressive disorders, Fig. 9 (b) is the recognition accuracy of anxiety disorders, Fig. 9 (c) is the recognition accuracy of mood disorders, Fig. 9 (d) is the recognition accuracy of normal people. As can be seen from Figure (9), the average recognition accuracy of depression disorder is 82.51%, the average recognition accuracy of anxiety disorder is 81.97%, the average recognition accuracy of mood disorder is 82.04%, and the average accuracy of normal people is 80.97%.

[0065] It should be noted that, without violating the spirit and scope of the present invention, those skilled in the art may make appropriate changes or modifications to the above embodiments. For example, for the preprocessing of the data set, various statistical features of physiological signals, such as mean, variance, standard deviation, median, mode, maximum, minimum, etc., can be used, and then these statistical features can be used as substitutes for mental disorder identification. Alternatively, the physiological signal can be converted into a time-frequency graph for processing, and the processed features can be transmitted to the system framework for mental disorder identification. For another example, for the sampling rules of positive and negative sample pairs, strong and weak negative sample pairs can also be combined without adding weight coefficients to construct negative samples. In addition, bidirectional GRU can also be used instead of BiLSTM. By improving the present invention, it is also applicable to other types of disease identification, such as cardiovascular diseases, as long as multimodal physiological data of other types of diseases can be collected.

[0066] In summary, compared with the prior art, the present invention has the following advantages:

[0067] 1) The present invention uses universal and convenient smart wearable devices to collect multimodal shallow physiological information of users. Since mental disorders usually cause physiological reactions, which cause shallow physiological information to change, and shallow physiological information directly reflects the body's immediate physiological state, universal wearable devices can better meet the real-time requirements of individuals in daily life than dedicated equipment and non-physiological collection equipment, and can detect and intervene in mental disorders at an early stage, and alleviate psychological and physical symptoms in a timely manner.

[0068] 2) The present invention uses a new contrastive learning method combined with residual variable convolution for multimodal fusion. Through the characteristics of contrastive learning, the similarity between positive samples is maximized, the similarity between negative samples is minimized, and multimodal fusion is performed using residual variable convolution. Local fusion features and global unique features of each modality can be extracted. Different modal physiological data are jointly learned with coarse and fine granularity combined with residual variable convolution network fusion, so that the system can better represent the characteristics of patients with each disease and the control group.

[0069] 3) The existing self-supervised fine-tuning strategy is to directly use a small amount of labeled data to train the classifier of the downstream task. The present invention first uses unlabeled data to fine-tune the feature encoder in a comparative learning manner for each user's different data, and then uses a small amount of labeled data to train a classifier dedicated to different users, thereby obtaining a personalized customized model for each user.

[0070] 4) Most of the existing work only identifies one type of mental disorder, and a few work identifies multiple types of mental disorders, but in terms of the amount of data labels consumed and system performance, such work is not as good as contrastive learning. The new contrastive learning method adopted by the present invention realizes the recognition of multiple categories of mental disorders and can solve the problem of a small number of labels. The trained model has better performance than the model trained using a large amount of labeled data.

[0071] 5) Different mental disorders have common symptoms, so it is difficult to identify which disease an individual suffers from. The present invention uses a new contrastive learning positive and negative sample pair construction strategy to deeply mine the coarse-grained and fine-grained multi-modal physiological data in the entire time series dimension and at a certain time point, and realize the common identification of multiple categories of mental disorders (such as depressive disorders, anxiety disorders, and mood disorders, etc.). Compared with existing research, the generalization of the system is significantly improved, and the user's psychological state can be more comprehensively identified.

[0072] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0073] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0074] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0075] The computer program instructions for performing the operation of the present invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, Python, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present invention.

[0076] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0077] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0078] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0079] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0080] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A wearable automatic diagnosis system for mental disorders based on contrastive learning, the system is implemented using a wearable device, comprising: Data acquisition unit: used to obtain multimodal physiological data of users; A user registration unit is used for, when it is determined that the user is a new user, executing: using the multimodal physiological data to fine-tune the first feature encoder pre-trained offline in a self-supervised contrastive learning manner to obtain a second feature encoder; Extracting data features from the stored labeled multimodal physiological data via a second feature encoder; using the data features as input to train a personalized classifier to obtain a mental disorder recognition model, wherein the mental disorder recognition model is constructed based on the second feature encoder and the trained personalized classifier; Identification unit: used to obtain identification results by using the mental disorder identification model when determining that the user is not a new user.

2. The system according to claim 1, characterized in that The personalized classifier includes a transformer network, a multi-layer perceptron, a Dropout layer and a Softmax layer.

3. The system according to claim 1, characterized in that The multimodal physiological data includes blood oxygen saturation, respiratory information, galvanic skin response, skin temperature and heart rate variability.

4. The system according to claim 1, characterized in that In the contrastive learning, the sampling rules of positive and negative samples are set as follows: for the multimodal physiological data, in the case of time alignment, one modality is selected as the anchor modality at a set time, and the physiological data of other modalities at the same time are regarded as positive samples, the sampling of the anchor modality at different time points is regarded as strong negative samples, and the sampling of other modalities at different time points is regarded as weak negative samples.

5. The system according to claim 1, characterized in that The first feature encoder includes multiple bidirectional long short-term memory networks, linear layers and residual variable convolutional networks, wherein each of the bidirectional long short-term memory networks is used to extract specific temporal features corresponding to a modality of physiological data, and after passing through the linear layer, the specific temporal features of each modality of physiological data are spliced ​​in the channel dimension to obtain a splicing vector, which is passed to the residual variable convolutional network to extract local fusion features and global features of each modality of physiological data.

6. The system according to claim 1, characterized in that The offline pre-training process of the first feature encoder is carried out in the cloud or server, and the unlabeled multimodal physiological data of different users is used for offline pre-training in a comparative learning manner.

7. The system according to claim 1, characterized in that The multiple modal physiological data monitored by the system and the recognition results of the mental disorder recognition model are transmitted to the terminal device via Bluetooth for display.

8. The system according to claim 3, characterized in that The system is a wristband wearable device, comprising a main control board, a bioelectrical impedance sensor, a blood oxygen sensor, a photoelectric volumetric pulse wave sensor, a skin charge sensor, a skin temperature sensor, a display screen module, a voice broadcast module and a Bluetooth module, wherein the bioelectrical impedance sensor is connected to the main control board through an analog input pin, and the analog-to-digital converter of the main control board is used to read the numerical value of respiratory information; the blood oxygen sensor performs asynchronous communication through the serial port of the main control board; the main control board and the Bluetooth module communicate using the UART serial port protocol; the main control board and the display screen module perform serial communication using the I2C bus; instructions and data are sent to the voice broadcast module through the UART serial port of the main control board to broadcast the physiological data information of the monitoring time period and the recognition result of the mental disorder recognition model.

9. A wearable automatic diagnosis method for mental disorders based on contrastive learning, comprising the following steps: Acquire the multimodal physiological data of the user and determine whether the user is a new user; In the case where the user is determined to be a new user, executing: using the multimodal physiological data to fine-tune the first feature encoder pre-trained offline in a self-supervised contrastive learning manner to obtain a second feature encoder; Extracting data features from the stored labeled multimodal physiological data via a second feature encoder; using the data features as input to train a personalized classifier to obtain a mental disorder recognition model, wherein the mental disorder recognition model is constructed based on the second feature encoder and the trained personalized classifier; When it is determined that the user is not a new user, the mental disorder recognition model is used to obtain a recognition result.

10. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to claim 9 are implemented.

Images