Mental health data acquisition method and system based on emotional reasoning normal form

Abstract

The invention provides a psychological health data acquisition method and system based on an emotional reasoning normal form, and relates to the technical field of artificial intelligence. The method comprises the steps that facial expression data, human body posture data and physiological signal data of a to-be-detected person are acquired and preprocessed, and multi-modal data are obtained; marking the multi-modal data to obtain a construction factor token; matching the construction factor token with a sample factor token in an emotion reasoning normal form database; if the matching is successful, obtaining mental health data corresponding to the successfully matched sample factor token from an emotion reasoning normal form database; if the matching is unsuccessful, performing feature extraction on the multi-modal data, constructing a cross-modal pair, and performing modal restoration on a missing modal to obtain a mental health quantification matrix of the person to be tested; and obtaining a conversion function, and taking a result calculated by the conversion function as psychological health data of the to-be-tested person. According to the invention, the problem of inaccurate and untimely acquisition of psychological health data at present is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for acquiring mental health data based on an emotion reasoning paradigm. Background Art

[0003] At present, most mental health data are interfered with based on questionnaire survey results. For this type of method, the fundamental premise of the validity of mental health data is that the students answer the questions carefully and truthfully. However, when conducting mental health tests, the students may intentionally or unintentionally conceal their mental health, resulting in false mental health data and biased questionnaire survey results, which makes it impossible to accurately and meticulously portray the students' mental state, resulting in untimely intervention in students' mental health. Summary of the invention

[0004] The present invention provides a method and system for acquiring mental health data based on an emotional reasoning paradigm, so as to solve the problem that the current acquisition of mental health data is inaccurate and untimely.

[0005] In one aspect, the present invention provides a method for acquiring mental health data based on an emotional reasoning paradigm, the method comprising: Acquire the facial expression data, human posture data and physiological signal data of the subject to be tested, perform preprocessing, and obtain multimodal data; Marking the multimodal data to obtain a construction factor token; matching the construction factor token with a sample factor token in a sentiment reasoning paradigm database; If the match is successful, the mental health data corresponding to the sample factor token that is successfully matched is obtained from the emotional reasoning paradigm database; If the match is unsuccessful, the multimodal data is subjected to feature extraction, and a cross-modal pair is constructed to perform modality repair on the missing modalities to obtain a psychological health quantification matrix of the subject; a conversion function is obtained based on the psychological health quantification matrix, and a result calculated by the conversion function is used as the psychological health data of the subject.

[0006] Furthermore, the emotional reasoning paradigm database is constructed based on the one-to-one correspondence between the construction factor token and the mental health data.

[0007] Furthermore, the facial expression data and the human body posture data are obtained by: The video dataset of the subject is collected simultaneously using an RGB camera, an infrared camera, and an infrared thermal imaging camera, and a visible image dataset, an infrared image dataset, and a thermal imaging image dataset are obtained accordingly; The visible image data set and the infrared image data set are fused to obtain a first image set; fusing the first image set with the thermal imaging image data set to obtain a second image set; The second image set is sliced ​​to obtain multiple frames of images, a face frame and a body frame are positioned in each frame of the image, and cropping, normalization, and data enhancement processing are performed to obtain the facial expression data and the body posture data.

[0008] Furthermore, the preprocessing includes: aligning the facial expression data, the human body posture data and the physiological signal data using timestamps.

[0009] Furthermore, the construction factor token includes a facial expression label, a human posture label and a physiological signal label.

[0010] Furthermore, The facial expression label includes: one or more of a mouth label, an eyebrow label, an eye label, and a cheek muscle label; The human body posture label includes: one or more of a head label, a limb label, and a torso label; The physiological signal label includes: one or more of an α wave label, a β wave label, a θ wave label, a δ wave label, and a γ wave label.

[0011] Furthermore, constructing cross-modal pairs and repairing the missing modalities includes: Construct cross-modal pairs of available modalities and missing modalities in triple format, and use ResEncoder and cycle consistency learning to perform forward and reverse modality restoration.

[0012] Further, using The function classifies the emotions of the subjects and obtains the mental health quantification matrix : ; in, Represents a classifier composed of multiple fully connected layers; Indicates connection The latent vector obtained from the ResEncoder, .

[0013] Furthermore, based on the mental health quantification matrix Get the conversion function include: The mental health quantification matrix Expressed as ; The conversion function It is expressed as: ; in, represents first mental health data obtained according to the physiological signal data; represents the second mental health data obtained according to the facial expression data; Represents the third psychological health data obtained based on human posture data; function, The range is ; , , are the learnable weights of each modality data, and .

[0014] On the other hand, the present invention also provides a mental health data acquisition system based on the emotional reasoning paradigm, comprising a memory, a processor and a computer program stored in the memory, characterized in that the processor executes the computer program to implement the steps of any of the above methods.

[0015] In general, the present invention provides a method and system for acquiring mental health data based on the emotional reasoning paradigm, which can achieve the following beneficial effects compared with the prior art: (1) The present invention uses the labeling of multimodal data to obtain the construction factor token; and matches the construction factor token with the sample factor token in the emotional reasoning paradigm database, so as to directly obtain the mental health data corresponding to the current multimodal data. In addition, the multimodal data is processed by feature extraction, modal repair, etc. to obtain the mental health quantification matrix of the subject; and the conversion function is obtained based on the mental health quantification matrix, and the result of the conversion function calculation is used as the mental health data of the subject. It can not only more accurately and deeply characterize the mental state of the subject and obtain mental health data, but also use the emotional reasoning paradigm to analyze the frontier consequences of psychological emotions, predict possible developments, and reversely infer the reasons for the psychological emotions of the subject, thereby providing more personalized, accurate and timely psychological intervention. Compared with questionnaire surveys and other methods, the present invention is more direct and objective, and greatly improves the efficiency and accuracy of obtaining mental health data.

[0016] (2) The present invention constructs an emotional reasoning paradigm database based on the one-to-one correspondence between construction factor tokens and mental health data, and stores the mental health data marked with construction factor tokens based on the data collected each time in the emotional reasoning paradigm database. As the database continues to grow, the success rate of matching construction factor tokens with sample factor tokens also continues to increase, which greatly improves the efficiency and accuracy of obtaining mental health data, and is also conducive to reverse reasoning about the reasons for the current mental state of the subject.

[0017] (3) The present invention achieves a comprehensive, objective, and multi-angle characterization of the psychological state of the subject by integrating multimodal data information in three dimensions: facial expression data, human posture data, and physiological signal data. By deeply exploring the complex causes and dynamic changes behind the psychological state of the subject from the internal physiological signals, as well as the external expressions and human body signals, the potential concealment behavior of the subject during the test is avoided. Compared with single-modal data, the diversified multimodal method of the present invention not only enriches the dimensions of psychological health data acquisition, but also significantly improves the comprehensiveness, accuracy, and reliability of psychological health data acquisition to a certain extent, providing a reliable tool for in-depth understanding and analysis of the psychological health status of the subject.

[0018] (4) The present invention constructs cross-modal pairs and performs modal repair on the missing modalities. It uses a polymorphic fusion reconstruction kernel to perform forward and backward repair on the missing modal data, effectively solving the problem of inaccurate classification caused by the missing data of a certain modality. To a certain extent, it improves the integrity of multimodal data and further enhances the accuracy and stability of mental health data results.

[0019] (5) The present invention uses an RGB camera, an infrared camera, and an infrared thermal imaging camera to simultaneously collect video data sets, fuses the visible image data set and the infrared image data set to obtain a first image set; and fuses the first image set with the thermal imaging image data set to obtain a second image set. By integrating multiple types of cameras for collection, such a diversified data collection strategy greatly suppresses the risk of overall classification errors caused by single camera collection errors, provides a more scientific and rigorous basis for the acquisition of students' mental health data, and is of great significance for detecting students' healthy growth and all-round development. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a schematic diagram of the method steps of a method and system for acquiring mental health data based on an emotional reasoning paradigm provided by the present invention; Figure 2 It is a data collection scenario diagram of a method and system for acquiring mental health data based on an emotional reasoning paradigm provided by the present invention; Figure 3 It is a schematic diagram of a matching database of a method and system for acquiring mental health data based on an emotional reasoning paradigm provided by the present invention; Figure 4 It is a schematic diagram of the principle of a multimodal detection model of a method and system for acquiring mental health data based on an emotional reasoning paradigm provided by the present invention; Figure 5 It is a schematic diagram of a face processor of a method and system for acquiring mental health data based on an emotion reasoning paradigm provided by the present invention; Figure 6 It is a schematic diagram of a human posture processor of a method and system for acquiring mental health data based on an emotional reasoning paradigm provided by the present invention; Figure 7 It is a schematic diagram of a physiological signal processor of a method and system for acquiring mental health data based on an emotional reasoning paradigm provided by the present invention; Figure 8 It is a schematic diagram of a polymorphic symbiotic reconstruction core of a mental health data acquisition method and system based on an emotional reasoning paradigm provided by the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings and embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0023] It should be noted that, in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method, step or system including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such method, step or system. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the method, step or system including the element.

[0024] The present invention provides a method for acquiring mental health data based on the emotional reasoning paradigm, such as Figure 1As shown, the method includes: obtaining facial expression data, human posture data and physiological signal data of the subject to be tested, performing preprocessing to obtain multimodal data; marking the multimodal data to obtain construction factor tokens; matching the construction factor tokens with sample factor tokens in the emotional reasoning paradigm database; if the match is successful, obtaining the mental health data corresponding to the matched sample factor tokens from the emotional reasoning paradigm database; if the match is unsuccessful, extracting features from the multimodal data, constructing cross-modal pairs, performing modal repair on the missing modalities, and obtaining the mental health quantification matrix of the subject to be tested; obtaining a conversion function based on the mental health quantification matrix, and using the result of the conversion function calculation as the mental health data of the subject to be tested.

[0025] The facial expression data, human posture data and physiological signal data of the subject are obtained, and preprocessed to obtain multimodal data.

[0026] It should be noted that the facial expression data includes information on the five senses and facial muscles, etc., which are used to represent the facial expressions of the subject, such as smile, excitement, surprise, sadness, fear, disgust, anger, etc.; the body posture data includes head information, limb information and torso information, etc., which are used to represent the body posture of the subject, such as holding the head, curling up the body, supporting the chin, looking up the head, walking with the head down, etc.; the physiological signal data E(t) includes electroencephalogram signals, electrocardiogram signals, electromyography signals, respiratory signals, blood oxygen signals, skin electricity signals, etc., which are used to represent the physiological signals of the subject.

[0027] like Figure 2 As shown, facial expression data and human posture data can be collected by one or more of an RGB camera, an infrared camera, and an infrared thermal imaging camera, or by a combination of the three cameras; the collected data can be video data or image data. The collection method of physiological signal data can be to use various types of wearable devices to extract the physiological signals of the subject in real time.

[0028] Since data collection using a single camera may result in incomplete data collection, which may lead to incorrect labeling of multimodal data and the risk of overall classification errors, as a preferred embodiment of the present invention, data collection is performed using a combination of multiple types of cameras. Such a diversified data collection strategy provides a more scientific and rigorous basis for obtaining students' mental health data.

[0029] Specifically, facial expression data and human body posture data are obtained in the following ways: The video dataset of the subject is collected simultaneously using an RGB camera, an infrared camera, and an infrared thermal imaging camera, and a visible image dataset, an infrared image dataset, and a thermal imaging image dataset are obtained accordingly; The visible image data set and the infrared image data set are fused to obtain a first image set; fusing the first image set with the thermal imaging image data set to obtain a second image set; The second image set is sliced ​​to obtain multiple frames of images, and the face frame and the body frame are located in each frame of the image, and cropping, normalization, and data enhancement processing are performed to obtain facial expression data and body posture data.

[0030] Since facial expressions are subtle and the acquisition error rate is high, as another embodiment, the facial expression data and human posture data can also be obtained in the following manner: The RGB camera and the infrared camera are used to collect the video data set of the subject at the same time, and the video data set is sliced ​​separately, and the face frame and the body frame are located respectively to obtain each instance of the face image and the body posture image, and each face and each body is cropped, normalized, and data enhanced to obtain the corresponding visible image data frame and infrared image data frame; the visible image data frame and the infrared image data frame are fused using the VIF network to obtain the face image and human posture data ; At the same time, an infrared thermal imaging camera is used to collect the video data set of the subject, the video data set is sliced, the face frame is located and cropped, each instance face image is obtained, and each face is normalized and data enhanced to obtain the corresponding facial thermal imaging image. ; The face image and thermal imaging of the face Fusion is performed to obtain facial expression data.

[0031] The fusion method may be to use a VIF network for fusion.

[0032] For example, using the VIF network, the visible image data frame and the infrared image data frame are input into the dual channels respectively, and multiple convolution layers are used respectively. After extracting the features, the features of the two channels are superimposed, and the convolution layers are used to reconstruct the features to obtain the face image. and human posture data . Its loss function is: .

[0033] ; ; ; ; ; ; in, , All represent learnable weights; Represents an RGB image; represents an infrared image; Indicates that the size of a sliding window is ; Indicates pixels; and Indicates the standard deviation of pixels; express The covariance of represents a constant; Indicates the total number of sliding windows; Indicates indicators; Indicates indicators; Represents an RGB image; An image representing a processed image of a facial expression or human posture.

[0034] It should be noted that the preprocessing includes: aligning facial expression data, human posture data and physiological signal data using timestamps.

[0035] It may also include one or more of the following processing: removing useless electrodes, re-referencing, filtering, downsampling, and removing artifacts on the physiological signal data.

[0036] Mark the multimodal data to obtain the construction factor token.

[0037] Labeling means labeling the expressions, movements, and physiological performances of the subjects according to the facial expression data, human posture data, and physiological signal data, so as to match the constructed factor tokens with the sample factor tokens in the emotional reasoning paradigm database.

[0038] As an example, the construction factor token may include a facial expression label, a human posture label, and a physiological signal label.

[0039] More specifically, the facial expression tags include: one or more of the mouth tags, eyebrow tags, eye tags, and cheek muscle tags. For example, the corners of the mouth are raised, the corners of the mouth are drooped, the mouth is tightly closed, the mouth is slightly open, and the mouth is wide open; the eyebrows are stretched, the eyebrows are slightly frowned, and the eyebrows are tightly wrinkled; the eyes are moist, with tears, the eyes are wide open, the eyes are closed, the eyes are narrowed, the corners of the eyes are drooped, and the corners of the eyes are raised; the cheek muscles are relaxed, and the cheek muscles are ferocious, etc.

[0040] Human posture labels include: one or more of head labels, limb labels, and body labels. For example, holding the head with four limbs, curling up, walking with head down, holding the head high, supporting the chin with one hand, supporting the chin with both hands, holding the head with one hand, holding both hands in front of the chest, and putting both hands on the waist.

[0041] Physiological signal labels include: one or more of alpha wave labels, beta wave labels, theta wave labels, delta wave labels, and gamma wave labels. For example, a high level of alpha wave indicates distraction, a normal level indicates relaxation, and a low level indicates anxiety; a high level of beta wave indicates anxiety, a normal level indicates relaxation, and a low level indicates depression; a high level of theta wave indicates depression, a normal level indicates relaxation, and a low level indicates anxiety; a high level of delta wave indicates inattention, a normal level indicates sleep, and a low level indicates poor sleep; a high level of gamma wave indicates anxiety, a normal level indicates sleep, and a low level indicates depression.

[0042] like Figure 3 As shown, the constructed factor token is matched with the sample factor token in the emotional reasoning paradigm database; if the match is successful, the mental health data corresponding to the matched sample factor token is obtained from the emotional reasoning paradigm database.

[0043] The emotional reasoning paradigm database is constructed based on a one-to-one correspondence between construction factor tokens and mental health data.

[0044] It should be noted that the emotional reasoning paradigm database includes multiple sets of paradigm data, each of which consists of a sample factor token (or a labeled construction factor token) and a corresponding mental health data. In other words, the emotional reasoning paradigm database includes multiple sample factor tokens and multiple mental health data, and one sample factor token corresponds to one mental health data.

[0045] For example, a certain multimodal data is labeled and the constructed factor token is obtained as follows: mouth corner raised + eyebrow stretched + head held high + normal alpha wave level + normal theta wave. The constructed factor token is matched with the sample factor token in the emotional reasoning paradigm database.

[0046] If a sample factor token is obtained from the emotional reasoning paradigm database, which is also: mouth corner raised + eyebrows stretched + head held high + normal alpha wave + normal theta wave, it means the match is successful. Since the mental health data corresponding to the sample factor token in the database is s1, the mental health data corresponding to the construction factor token that is successfully matched is also s1. Then the mental health data s1 corresponding to the sample factor token can be directly output.

[0047] If a sample factor token is obtained from the emotional reasoning paradigm database: raised corners of the mouth + eyes wide open + head held high and chest out + normal alpha wave level + normal theta wave; or raised corners of the mouth + relaxed eyebrows + raised corners of the eyes + head held high and chest out + normal alpha wave level + normal theta wave; it means that the match is unsuccessful, and the emotional reasoning paradigm model is reconstructed, and the psychological health data of the subject is calculated to be s2.

[0048] Reconstruct the emotional reasoning paradigm model. Specifically, perform feature extraction on multimodal data, construct cross-modal pairs, perform modal repair on missing modalities, and obtain the psychological health quantification matrix of the subject; obtain the conversion function based on the psychological health quantification matrix, and use the result of the conversion function calculation as the psychological health data of the subject.

[0049] It should be noted that after obtaining multimodal data and successfully matching, the psychological health data of the subject can be used for preliminary warning. Multimodal data can then be used for feature extraction, modality repair and other processing to obtain a psychological health quantification matrix, and then the psychological health data of the subject can be used for ultimate warning.

[0050] Extract features from multimodal data. For example, use the SPNDNet multimodal detection model to process the preprocessed multimodal data to obtain feature representations of different modalities of the subject.

[0051] It should be noted that if Figure 4 As shown in Figure 1, the SPNDNet multimodal detection model includes a face processor, a human posture processor, and a physiological signal processor.

[0052] As an example, Figure 5 As shown in Figure 1, the backbone network of the face processor includes multiple R3Ds, a dynamic attention mechanism module, and an MLP classification head. Among them, multiple R3Ds initially extract the features of each image frame of the subject. , the features of each frame are fused through the dynamic attention mechanism module, which includes multi-head self-attention mechanism, dynamic instance normalization, and one-dimensional convolution. The multi-head self-attention mechanism learns the relationship between each image of the instance and obtains the attention weight A.

[0053] Dynamic instance normalization maintains the temporal consistency of individual image features and fused features, and its formula is: ; in, and Respectively represent The image before and after normalization Channel values; and They represent the mean and variance values ​​estimated by the normalizer for a specific channel value of an instance, respectively; and represents the importance weight of the normalizer; and represents the learnable affine transformation parameters; A small number added for numerical stability.

[0054] Use one-dimensional convolution to get the target face features. For face images and thermal imaging of the face All use face processors, facial expression features , specifically: ; in, for The attention weights of each channel of the image after the attention connection is concatenated.

[0055] As an example, Figure 6 As shown in Figure 2, the human posture processor combines the spatial attention mechanism and the temporal attention mechanism to extract human posture features using the Transformer encoder. .

[0056] The attention module obtains the spatial dimension weights respectively and time dimension weight ;in, Indicates a query, Indicates the key, Represents a value. Fusion of two modules Finally, combined with the weight Encode to obtain human body posture features .

[0057] As an example, Figure 7 As shown in the figure, the physiological signal processor uses the Transformer network to extract physiological signal features from the physiological signal E(t) obtained after preprocessing. , specifically: .

[0058] The polymorphic co-integration reconstruction kernel is to construct a cross-modal pair, repair the missing modalities, and obtain the psychological health quantitative matrix of the subject. More specifically, Figure 8 As shown, cross-modal pairs of available modalities and missing modalities in triple format are constructed, and ResEncoder and cycle consistency learning are used to perform forward and reverse modality repair.

[0059] As a specific embodiment, taking the absence of EEG signals as an example, the extracted features are: ; ;in, Represents a zero vector, i.e., missing data.

[0060] ResEncoders and cycle consistency learning are used to perform forward and reverse modal restoration.

[0061] As an embodiment, the formula for modal repair is: ; in, Indicates The output of a ResEncoder; Indicates A ResEncoder.

[0062] The introduced forward loss function , inverse loss function and the cross entropy loss function Specifically, ; ; ; in, Indicates the number of ResEncoder; represents the true value representation extracted by the network; ; ; represents the total number of samples; Indicates the total number of categories; Table Data No. The label of the dimension; Indicates connection The potential vector obtained by ResEncoder: ; Indicates The samples belong to The probability of the class.

[0063] As a preferred embodiment, using The function classifies the emotions of the subjects and obtains the psychological health quantification matrix : ; in, Represents a classifier composed of multiple fully connected layers; Indicates connection The latent vector obtained from the ResEncoder, .

[0064] Based on the psychological health quantitative matrix Get the conversion function include: Quantifying Mental Health Expressed as ; The conversion function It is expressed as: ; in, represents first mental health data obtained according to the physiological signal data; represents the second mental health data obtained according to the facial expression data; Represents the third psychological health data obtained based on human posture data; function, The range is ; , , are the learnable weights of each modality data, and .

[0065] Finally, the result of the conversion function calculation is used as the psychological health data of the subject. . That is: It should be noted that mental health data The closer it is to 0, the healthier the person's mental health is and the lower the risk is.

[0066] It should be noted that the method also includes: using the mental health data obtained The corresponding construction factor token Stored in the emotional reasoning paradigm database In order to continuously increase the database, the success rate of matching the constructed factor token with the sample factor token is continuously increased, which greatly improves the efficiency and accuracy of obtaining mental health data, and is also conducive to reverse reasoning about the causes of the mental health data of the test subject; and the emotional reasoning paradigm can be used to analyze the frontier consequences of psychological emotions, predict possible developments, and reversely reason the causes of the mental health state of the test subject, thereby providing more personalized, accurate, and timely psychological intervention.

[0067] In addition, the present invention can perform corresponding interventions based on mental health data.

[0068] As an embodiment of the present invention, a plurality of mental health data thresholds are set to obtain an intervention control matrix, and corresponding intervention is performed on the subject according to the current mental health data and the intervention control matrix.

[0069] For example, setting a threshold for mental health data , and , the intervention is divided into primary warning, moderate imbalance, severe warning, and critical crisis. When , it means that the mental health of the person being tested is not at risk and no intervention is required; when When , it indicates that the psychological health of the person being tested needs to be initially warned, for example, psychological counseling services can be provided to help the person being tested recognize and deal with emotions; when When , it indicates that the mental health of the person being tested is moderately unbalanced and requires certain interventions, such as providing professional psychological counseling, regular psychological assessments, mental health education, and one-on-one counseling; when When , it means that a serious warning is needed for the mental health of the person being tested. For example, the student can be referred to a professional medical institution for emergency assessment and treatment; when When the test subject's mental health is at a critical crisis, certain intervention is required. For example, the psychological crisis intervention procedure can be initiated immediately to ensure the student's life safety.

[0070] On the other hand, the present invention also provides a mental health data acquisition system based on the emotional reasoning paradigm, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of any of the above methods. The technical features of the system are consistent with those of the above methods, and will not be described one by one here.

[0071] It should be noted that, for the above-mentioned various embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0072] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0073] In the several embodiments provided in this application, it should be understood that the disclosed method or system can be implemented in other ways. For example, the above-described embodiments are only schematic, and the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0074] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0075] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0076] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0077] Those skilled in the art will appreciate that all or part of the various circuits in the above embodiments may be completed by entering a program to instruct related hardware, and the program may be stored in a computer-readable memory, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0078] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

[0079] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0080] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for acquiring mental health data based on the emotional reasoning paradigm, characterized in that: The method comprises: Acquire the facial expression data, human posture data and physiological signal data of the subject to be tested, perform preprocessing, and obtain multimodal data; Marking the multimodal data to obtain a construction factor token; matching the construction factor token with a sample factor token in a sentiment reasoning paradigm database; If the match is successful, the mental health data corresponding to the sample factor token that is successfully matched is obtained from the emotional reasoning paradigm database; If the match is unsuccessful, the multimodal data is subjected to feature extraction, and a cross-modal pair is constructed to perform modality repair on the missing modalities to obtain a psychological health quantification matrix of the subject; a conversion function is obtained based on the psychological health quantification matrix, and a result calculated by the conversion function is used as the psychological health data of the subject.

2. According to the method for obtaining mental health data based on the emotional reasoning paradigm of claim 1, it is characterized in that: The emotional reasoning paradigm database is constructed based on the one-to-one correspondence between the construction factor token and the mental health data.

3. The method for acquiring mental health data based on the emotional reasoning paradigm according to claim 1 is characterized in that: The facial expression data and the human body posture data are obtained by: The video dataset of the subject is collected simultaneously using an RGB camera, an infrared camera, and an infrared thermal imaging camera, and a visible image dataset, an infrared image dataset, and a thermal imaging image dataset are obtained accordingly; The visible image data set and the infrared image data set are fused to obtain a first image set; fusing the first image set with the thermal imaging image data set to obtain a second image set; The second image set is sliced ​​to obtain multiple frames of images, a face frame and a body frame are positioned in each frame of the image, and cropping, normalization, and data enhancement processing are performed to obtain the facial expression data and the body posture data.

4. The method for acquiring mental health data based on the emotional reasoning paradigm according to claim 1, characterized in that: The preprocessing includes: aligning the facial expression data, the human body posture data and the physiological signal data using timestamps.

5. The method for acquiring mental health data based on the emotional reasoning paradigm according to claim 1, characterized in that: The construction factor token includes a facial expression label, a human posture label and a physiological signal label.

6. A method for acquiring mental health data based on the emotional reasoning paradigm according to claim 5, characterized in that: The facial expression label includes: one or more of a mouth label, an eyebrow label, an eye label, and a cheek muscle label; The human body posture label includes: one or more of a head label, a limb label, and a torso label; The physiological signal label includes: one or more of an α wave label, a β wave label, a θ wave label, a δ wave label, and a γ wave label.

7. The method for acquiring mental health data based on the emotional reasoning paradigm according to claim 1, characterized in that: Constructing cross-modal pairs and repairing the missing modalities includes: Construct cross-modal pairs of available modalities and missing modalities in triple format, and use ResEncoder and cycle consistency learning to perform forward and reverse modality restoration.

8. The method for acquiring mental health data based on the emotional reasoning paradigm according to claim 1, characterized in that: use The function classifies the emotions of the subjects and obtains the mental health quantification matrix : ； in, Represents a classifier composed of multiple fully connected layers; Indicates connection The latent vector obtained from the ResEncoder, .

9. The method for acquiring mental health data based on the emotional reasoning paradigm according to claim 1, characterized in that: Based on the mental health quantification matrix Get the conversion function include: The mental health quantification matrix Expressed as ; The conversion function It is expressed as: ; in, represents first mental health data obtained according to the physiological signal data; represents the second mental health data obtained according to the facial expression data; Represents the third psychological health data obtained based on human posture data; function, The range is ; , , are the learnable weights of each modality data, and .

10. A mental health data acquisition system based on an emotional reasoning paradigm, comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes a computer program to implement the steps of the method according to any one of claims 1 to 9.

Images