Task state thermal imaging and near-infrared manic emotion assessment method, system and device

Abstract

The invention discloses a manic emotion assessment method, system and device based on task state thermal imaging and near infrared. The method comprises the following steps: acquiring expression features, pupil features and behavioral features of a user in an irritable psychological experiment normal form assessment process; constructing a multi-modal fusion neural network; and inputting the expression features, the pupil features and the behavioral features into a multi-modal fusion neural network to obtain a manic emotion evaluation result of the user. Thermal imaging and eye movement tracking technologies are integrated to achieve real-time and accurate detection of emotional states such as manic emotional disorder, the limitation of a traditional method is broken through through the multi-modal fusion technology, a new solution is provided for screening and evaluation of emotional disorder, and particularly, the method has a good application prospect in emotional management and mental health monitoring of high-risk people. Wide application prospects are realized.

Description

Technical Field

[0001] The present application relates to the field of psychological emotion assessment, and specifically, to a method, system and device for assessing manic emotion using task-state thermal imaging and near-infrared. Background Art

[0002] With the increase of social pressure and the accelerated pace of life, the incidence of mental and psychological disorders has shown a significant upward trend worldwide, especially among children, adolescents and adults. The onset of mental and psychological disorders not only seriously affects the physical and mental health of individuals, but also has a negative impact on the quality of family life and socioeconomic development. Mood disorders, especially manic mood disorders, are one of the most serious mood disorders. They are characterized by irritability, high spirits and extreme excitement, and are extremely risky. Therefore, timely and accurate screening and evaluation of these mood disorders are particularly important in certain special industries. Traditional methods for evaluating mood disorders usually rely on subjective questionnaires and behavioral observations. These methods have problems such as low evaluation efficiency, reliance on human judgment, and single detection modality, which limits the accuracy of the results. Summary of the invention

[0003] In order to overcome at least one deficiency in the prior art, the present application provides a method, system and device for assessing manic mood using task-state thermal imaging and near-infrared.

[0004] In a first aspect, a method for assessing manic mood using task-state thermal imaging and near-infrared is provided, comprising:

[0005] Acquire the user's facial features, pupil features, and behavioral features during the irritability psychology experimental paradigm assessment process; facial features include real-time facial temperature and facial thermal imaging images, pupil features include real-time pupil size and near-infrared eye tracking images, and behavioral features include key reaction time and key judgment accuracy;

[0006] Construct a multimodal fusion neural network, which includes a thermal imaging image processing module, an eye tracking image processing module, a numerical matrix module, a cross-modal attention mechanism, a self-attention mechanism, a fusion module, and a classifier;

[0007] The facial expression features, pupil features, and behavioral features are input into the multimodal fusion neural network to obtain the user's manic emotion assessment results.

[0008] In one embodiment, the thermal imaging image processing module is used to obtain a feature representation of the thermal imaging image based on the facial thermal imaging image;

[0009] The eye tracking image processing module is used for obtaining feature representation of the eye tracking image based on the near infrared eye tracking image;

[0010] The numerical matrix module is used to obtain numerical feature representation based on real-time facial temperature, real-time pupil size, key reaction time and key judgment accuracy;

[0011] The feature representation of the thermal imaging image and the feature representation of the eye tracking image are input into the cross-modal attention mechanism for cross-modal attention calculation to obtain the cross-modal attention feature;

[0012] The numerical feature representation is input into the self-attention mechanism for self-attention calculation to obtain the self-attention feature;

[0013] The cross-modal attention features and self-attention features are input into the fusion module for fusion to obtain fusion features;

[0014] The fused features are input into the classifier to obtain the user's manic emotion assessment result.

[0015] In one embodiment, the thermal imaging image processing module includes an image cutting and embedding unit, a Transformer encoder, a residual connection unit, and a layer normalization unit; the Transformer encoder includes a plurality of Transformer units;

[0016] The facial thermal imaging image is input into the image cutting and embedding unit and cut into multiple image blocks, and each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension;

[0017] After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation. Each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result.

[0018] Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain the residual connection result;

[0019] The residual connection result is input into the layer normalization unit to obtain the feature representation of the thermal imaging image.

[0020] In one embodiment, the eye tracking image processing module includes an image preprocessing and enhancement unit, an image cutting and embedding unit, a Transformer encoder, a residual connection unit, and a layer normalization unit; the Transformer encoder includes a plurality of Transformer units;

[0021] The near-infrared eye tracking image is input into the image preprocessing and enhancement unit for noise suppression and edge feature extraction to obtain a processed image;

[0022] The processed image is input into the image cutting and embedding unit and cut into multiple image blocks, and each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension;

[0023] After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation. Each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result.

[0024] Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain the residual connection result;

[0025] The residual connection result is input into the layer normalization unit to obtain the feature representation of the eye tracking image.

[0026] In one embodiment, the numerical matrix module includes three fully connected convolutional layers, a token mixing unit, a channel mixing unit, and a global pooling layer connected in sequence;

[0027] Real-time facial temperature, real-time pupil size, key reaction time, and key judgment accuracy are input into three fully connected convolutional layers for feature dimension reduction and extraction to obtain numerical matrix data;

[0028] The numerical matrix data is input into the token mixing unit for token mixing to obtain the token mixing result;

[0029] The token mixing result is input into the channel mixing unit to obtain the channel mixing result;

[0030] The channel mixing result is input into the global pooling layer to obtain the numerical feature representation.

[0031] In a second aspect, a task-state thermal imaging and near-infrared manic mood assessment device is provided, comprising:

[0032] The feature acquisition module is used to obtain the user's facial features, pupil features, and behavioral features during the irritability psychology experimental paradigm assessment process; the facial features include real-time facial temperature and facial thermal imaging images, the pupil features include real-time pupil size and near-infrared eye tracking images, and the behavioral features include key reaction time and key judgment accuracy;

[0033] A network building module, used to build a multimodal fusion neural network, which includes a thermal imaging image processing module, an eye tracking image processing module, a numerical matrix module, a cross-modal attention mechanism, a self-attention mechanism, a fusion module and a classifier;

[0034] The evaluation module is used to input facial expression features, pupil features, and behavioral features into the multimodal fusion neural network to obtain the user's manic emotion evaluation results.

[0035] In one embodiment, the thermal imaging image processing module is used to obtain a feature representation of the thermal imaging image based on the facial thermal imaging image;

[0036] The eye tracking image processing module is used to obtain feature representation of the eye tracking image based on the near-infrared eye tracking image;

[0037] The numerical matrix module is used to obtain numerical feature representation based on real-time facial temperature, real-time pupil size, key reaction time and key judgment accuracy;

[0038] The feature representation of the thermal imaging image and the feature representation of the eye tracking image are input into the cross-modal attention mechanism for cross-modal attention calculation to obtain the cross-modal attention feature;

[0039] The numerical feature representation is input into the self-attention mechanism for self-attention calculation to obtain the self-attention feature;

[0040] The cross-modal attention features and self-attention features are input into the fusion module for fusion to obtain fusion features;

[0041] The fused features are input into the classifier to obtain the user's manic emotion assessment result.

[0042] In one embodiment, the thermal imaging image processing module includes an image cutting and embedding unit, a Transformer encoder, a residual connection unit, and a layer normalization unit; the Transformer encoder includes a plurality of Transformer units;

[0043] The facial thermal imaging image is input into the image cutting and embedding unit and cut into multiple image blocks, and each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension;

[0044] After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation. Each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result.

[0045] Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain the residual connection result;

[0046] The residual connection result is input into the layer normalization unit to obtain the feature representation of the thermal imaging image.

[0047] In one embodiment, the eye tracking image processing module includes an image preprocessing and enhancement unit, an image cutting and embedding unit, a Transformer encoder, a residual connection unit, and a layer normalization unit; the Transformer encoder includes a plurality of Transformer units;

[0048] The near-infrared eye tracking image is input into the image preprocessing and enhancement unit for noise suppression and edge feature extraction to obtain a processed image;

[0049] The processed image is input into the image cutting and embedding unit and cut into multiple image blocks, and each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension;

[0050] After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation. Each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result.

[0051] Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain the residual connection result;

[0052] The residual connection result is input into the layer normalization unit to obtain the feature representation of the eye tracking image.

[0053] In one embodiment, the numerical matrix module includes three fully connected convolutional layers, a token mixing unit, a channel mixing unit, and a global pooling layer connected in sequence;

[0054] Real-time facial temperature, real-time pupil size, key reaction time, and key judgment accuracy are input into three fully connected convolutional layers for feature dimension reduction and extraction to obtain numerical matrix data;

[0055] The numerical matrix data is input into the token mixing unit for token mixing to obtain the token mixing result;

[0056] The token mixing result is input into the channel mixing unit to obtain the channel mixing result;

[0057] The channel mixing result is input into the global pooling layer to obtain the numerical feature representation.

[0058] In a third aspect, a task-state thermal imaging and near-infrared manic mood assessment system is provided, comprising: a multi-modal thermal imaging and eye tracking acquisition platform and a task-state thermal imaging and near-infrared manic mood assessment device; the multi-modal thermal imaging and eye tracking acquisition platform comprises a thermal imaging device, an eye tracker and a keyboard;

[0059] The thermal imaging device is used to collect the facial expression characteristics of the user during the irritability psychology experimental paradigm assessment process, the eye tracker is used to collect the pupil characteristics of the user during the irritability psychology experimental paradigm assessment process, and the keyboard is used to collect the behavioral characteristics of the user during the irritability psychology experimental paradigm assessment process; the task-state thermal imaging and near-infrared manic emotion assessment device is the above-mentioned device.

[0060] Compared with the existing technology, the present application has the following beneficial effects: the task-state thermal imaging and near-infrared manic mood assessment method, system and device of the present application integrate thermal imaging and eye tracking technology to achieve real-time and accurate detection of emotional states such as manic mood disorders. This multimodal fusion technology breaks through the limitations of traditional methods and provides a new solution for the screening and assessment of emotional disorders, especially in the emotional management and mental health monitoring of high-risk groups, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The present application may be better understood by referring to the following description given in conjunction with the accompanying drawings, which together with the following detailed description are included in this specification and form a part of this specification. In the drawings:

[0062] Figure 1 A flowchart of a method for assessing manic mood using task-state thermal imaging and near-infrared imaging according to an embodiment of the present application is shown;

[0063] Figure 2 A schematic diagram of real-time facial temperature recognition under a thermal imaging image is shown, wherein (a) is when the irritable emotion is not activated, and (b) is when the irritable emotion is activated;

[0064] Figure 3 A schematic diagram of real-time pupil recognition under near-infrared images is shown, where (a) is when irritable emotions are not activated, and (b) is when irritable emotions are activated;

[0065] Figure 4 The structural block diagram of the multimodal fusion neural network is shown;

[0066] Figure 5 A structural block diagram of a task-state thermal imaging and near-infrared manic mood assessment device according to an embodiment of the present application is shown;

[0067] Figure 6 A schematic diagram of a multimodal thermal imaging and eye tracking acquisition platform is shown. DETAILED DESCRIPTION

[0068] The exemplary embodiments of the present application will be described below in conjunction with the accompanying drawings. For the sake of clarity and conciseness, not all features of the actual embodiments are described in the specification. However, it should be understood that many implementation-specific decisions can be made in the process of developing any such actual embodiments in order to achieve the specific goals of the developer, and these decisions may vary from embodiment to embodiment.

[0069] It is also necessary to explain here that, in order to avoid obscuring the present application due to unnecessary details, only the device structure closely related to the scheme according to the present application is shown in the drawings, while other details that are not closely related to the present application are omitted.

[0070] It should be understood that the present application is not limited to the described implementation forms due to the following description with reference to the accompanying drawings. In this article, where feasible, the embodiments can be combined with each other, features between different embodiments can be replaced or borrowed, and one or more features can be omitted in one embodiment.

[0071] The present application provides a task-state thermal imaging and near-infrared method for assessing mania. First, based on the irritability characteristics unique to manic disorder, irritability is triggered by constructing an irritability psychology experimental paradigm. Then, a multimodal thermal imaging and eye tracking acquisition platform is built to synchronously collect three external features after irritability is activated, including thermal imaging real-time image features of facial expressions, eye tracking features of pupil changes, and behavioral features triggered by buttons. Finally, a multimodal fusion neural network is independently constructed to fuse and quantify the features of these three dimensions, so as to quickly and accurately detect disorders such as mania, in order to solve the current problem of lack of objective evaluation methods.

[0072] Figure 1 A flowchart of a method for assessing manic mood using task-state thermal imaging and near-infrared imaging according to an embodiment of the present application is shown. Figure 1 , methods include:

[0073] Step S1, obtaining the user's facial expression characteristics, pupil characteristics, and behavioral characteristics during the irritability psychology experimental paradigm assessment process; the facial expression characteristics include real-time facial temperature and facial thermal imaging images, the pupil characteristics include real-time pupil size and near-infrared eye tracking images, and the behavioral characteristics include key reaction time and key judgment accuracy.

[0074] Irritability psychology experiment: Faces with expressions of joy and depression randomly appear on the screen, and the positions are randomly shuffled. After seeing the expression faces, the participants are required to identify the direction where the depression appears and press the corresponding left or right button. At the same time, there is a reward and punishment mechanism with points, and the points are displayed in real time in the upper right corner of the screen. Participants will be required to respond quickly and accurately throughout the experiment to get points. If the answer is correct, text feedback will be displayed and points will be increased. If the answer is wrong, text feedback will be displayed and points will be deducted.

[0075] The experimental paradigm evaluation process consists of two parts, namely experimental evaluation 1 and experimental evaluation 2.

[0076] In Experimental Evaluation 1, participants will receive feedback on whether their answers are accurate or not. That is, the influence of reaction speed is not considered at this time, and the corresponding points are increased or decreased based on the accuracy of the key press judgment. If the key is pressed correctly, "Correct Answer" will be displayed and points will be increased. If the key is pressed incorrectly, the text feedback "Incorrect Answer" will be displayed and points will be deducted.

[0077] In Experimental Evaluation 2, negative feedback trials occurred, activating irritable traits, that is, in 60% of their accurate experiments, text feedback of "answering too slowly" was given without reference to the subject's actual reaction time, and points were deducted, thereby causing frustration and activating irritable traits. In the other 40% of the experiments, accurate feedback was given according to whether the participant responded quickly and accurately. The reference reaction time was the average reaction time in Experimental Evaluation 1. When the judgment was correct and the reaction time was shorter than the average time in Experimental Evaluation 1, the text feedback "answered correctly" was displayed and points were increased; when the judgment was correct and the reaction time was longer than the average time in Experimental Evaluation 1, the text feedback "answered too slowly" was displayed and points were deducted. When the judgment was wrong, the text feedback "answered wrongly" was displayed and points were deducted.

[0078] In summary, the core of the irritability experimental paradigm is to fully stimulate the irritability traits unique to bipolar disorder through the negative feedback mechanism set in Assessment 2 in the paradigm.

[0079] When the irritable state is activated, blood circulation speeds up, and facial capillaries become congested, causing the facial temperature to rise. The system recognizes the real-time facial temperature based on the thermal imaging device. At the same time, the activation of the irritable state will cause characteristic emotional expressions such as anger and mania. The system recognizes the real-time emotions through the neural network fusion algorithm under thermal imaging. Here, the real-time facial temperature obtained includes the highest temperature, the lowest temperature and the average temperature of the face. Figure 2 A schematic diagram of real-time facial temperature recognition under a thermal imaging image is shown, where (a) is when the irritable emotion is not activated, and (b) is when the irritable emotion is activated.

[0080] When the irritable state is activated, pupil changes reflect the individual's emotional fluctuations. Eye tracking equipment is used to capture the pupil's response to external stimuli in real time, including pupil dilation and contraction, as well as the speed of reaction time. Real-time pupil size is used to quantify pupil changes under the influence of irritable emotions, and near-infrared eye tracking images capture the speed and magnitude of pupil response to stimuli by tracking eye movement data. Figure 3 A schematic diagram of real-time pupil recognition under near-infrared images is shown, where (a) is when irritable emotions are not activated, and (b) is when irritable emotions are activated.

[0081] Record the key reaction time and key judgment accuracy. Longer reaction time and lower judgment accuracy mean slower information processing speed and poorer executive function, which usually indicates a higher tendency towards emotional disorders. Key reaction time and key judgment accuracy can tap into the subject's ability to suppress manic emotions and information processing speed.

[0082] Step S2, construct a multimodal fusion neural network, which includes a thermal imaging image processing module, an eye tracking image processing module, a numerical matrix module, a cross-modal attention mechanism, a self-attention mechanism, a fusion module and a classifier. It should be noted that the constructed multimodal fusion neural network is a trained network, and the specific training process is a prior art, which will not be repeated here.

[0083] Step S3, inputting the facial expression features, pupil features, and behavioral features into the multimodal fusion neural network to obtain the user's manic emotion assessment result.

[0084] The thermal imaging image processing module, eye tracking image processing module, and numerical matrix module of this embodiment use different neural network modules to process their respective features, and then deeply fuse these features through a cross-modal attention mechanism and a fusion module, and finally achieve identification of people with irritable emotions through a classifier. This method breaks through the limitations of traditional methods and provides a new solution for the screening and assessment of emotional disorders, especially in emotional management and mental health monitoring involving high-risk groups, and has broad application prospects.

[0085] In one embodiment, Figure 4 shows the structural block diagram of the multimodal fusion neural network, see Figure 4 ,The thermal imaging image processing module is used to obtain the feature representation of the thermal imaging image based on the facial thermal imaging image;

[0086] The eye tracking image processing module is used for obtaining feature representation of the eye tracking image based on the near infrared eye tracking image;

[0087] The numerical matrix module is used to obtain numerical feature representation based on real-time facial temperature, real-time pupil size, key reaction time and key judgment accuracy;

[0088] The feature representation of the thermal imaging image and the feature representation of the eye tracking image are input into the cross-modal attention mechanism for cross-modal attention calculation to obtain the cross-modal attention feature;

[0089] The numerical feature representation is input into the self-attention mechanism for self-attention calculation to obtain the self-attention feature;

[0090] The cross-modal attention features and the self-attention features are input into the fusion module for fusion to obtain the fusion features; here, the fusion module concatenates the cross-modal attention features and the self-attention features, and performs regularization processing to obtain the fusion features.

[0091] The fusion features are input into the classifier to obtain the user's manic emotion assessment result. Here, the classifier can use a Softmax classifier to classify people with irritable emotions. The user's manic emotion assessment result refers to whether the user is an irritable person.

[0092] Specifically, the thermal imaging image processing module includes an image cutting and embedding unit, a Transformer encoder, a residual connection unit and a layer normalization unit; the Transformer encoder includes a plurality of Transformer units;

[0093] The facial thermal imaging image is input into the image cutting and embedding unit and cut into multiple image patches, and each image patch is flattened and linearly mapped to obtain an embedding vector of fixed dimension;

[0094] After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation. Each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result.

[0095] Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain the residual connection result;

[0096] The residual connection result is input into the layer normalization unit to obtain the feature representation of the thermal imaging image.

[0097] Specifically, the eye tracking image processing module includes an image preprocessing and enhancement unit, an image cutting and embedding unit, a Transformer encoder, a residual connection unit and a layer normalization unit; the Transformer encoder includes multiple Transformer units;

[0098] The near-infrared eye tracking image is input into the image preprocessing and enhancement unit for noise suppression and edge feature extraction to obtain a processed image; here, non-local means filtering (NLM) can be used to suppress noise in the near-infrared eye tracking image, and an adaptive Canny edge detection algorithm can be used to extract edge features.

[0099] The processed image is input into the image cutting and embedding unit and cut into multiple image blocks, and each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension;

[0100] After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation. Each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result.

[0101] Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain the residual connection result;

[0102] The residual connection result is input into the layer normalization unit to obtain the feature representation of the eye tracking image.

[0103] Specifically, the numerical matrix module includes three fully connected convolutional layers, a token mixing unit, a channel mixing unit, and a global pooling layer connected in sequence;

[0104] Real-time facial temperature, real-time pupil size, key reaction time, and key judgment accuracy are input into three fully connected convolutional layers for feature dimension reduction and extraction to obtain numerical matrix data;

[0105] The numerical matrix data is input into the token mixing unit for token mixing to obtain the token mixing result; the following formula is used:

[0106] u i =x num +W mix ·σ(W token ·x num )

[0107] Among them, u i is the token mixing result, x num is the numerical matrix data, W mix , W token is a learnable parameter and σ is an activation function.

[0108] The token mixing result is input into the channel mixing unit to obtain the channel mixing result; the following formula is used:

[0109] v i =ui +W channel ·σ(W channel_mix ·u i )

[0110] Among them, v i is the channel mixing result, W channel , W channel_mix is a learnable parameter and σ is an activation function.

[0111] The channel mixing result is input to the global pooling layer to obtain the numerical feature representation. In order to improve the representation ability of numerical features, a global pooling operation can be introduced after channel mixing to further compress the data dimension and retain key information by summarizing the overall statistical characteristics of the numerical features.

[0112] Based on the same inventive concept as the task-state thermal imaging and near-infrared manic emotion assessment method, this embodiment also provides a corresponding task-state thermal imaging and near-infrared manic emotion assessment device. Figure 5 The structure block diagram of the task-state thermal imaging and near-infrared manic mood assessment device according to an embodiment of the present application is shown, including:

[0113] The feature acquisition module 51 is used to acquire the user's facial features, pupil features, and behavioral features during the irritability psychology experimental paradigm assessment process; the facial features include real-time facial temperature and facial thermal imaging images, the pupil features include real-time pupil size and near-infrared eye tracking images, and the behavioral features include key reaction time and key judgment accuracy;

[0114] A network construction module 52, used to construct a multimodal fusion neural network, the multimodal fusion neural network including a thermal imaging image processing module, an eye tracking image processing module, a numerical matrix module, a cross-modal attention mechanism, a self-attention mechanism, a fusion module and a classifier;

[0115] The evaluation module 53 is used to input the facial expression features, pupil features, and behavioral features into the multimodal fusion neural network to obtain the user's manic emotion evaluation result.

[0116] The task-state thermal imaging and near-infrared manic mood assessment device of this embodiment has the same inventive concept as the task-state thermal imaging and near-infrared manic mood assessment method mentioned above. Therefore, the specific implementation method of the device can be seen in the embodiment part of the task-state thermal imaging and near-infrared manic mood assessment method mentioned above, and its technical effect corresponds to the technical effect of the above-mentioned method, which will not be repeated here.

[0117] The embodiment of the present application also provides a task-state thermal imaging and near-infrared manic emotion assessment system, comprising: a multi-modal thermal imaging and eye tracking acquisition platform and a task-state thermal imaging and near-infrared manic emotion assessment device; the multi-modal thermal imaging and eye tracking acquisition platform comprises a thermal imaging device, an eye tracker and a keyboard; Figure 6 A schematic diagram of a multimodal thermal imaging and eye tracking acquisition platform is shown.

[0118] The thermal imaging device is used to collect the facial features of the user during the irritability psychology experimental paradigm assessment process, the eye tracking device is used to collect the pupil features of the user during the irritability psychology experimental paradigm assessment process, and the keyboard is used to collect the behavioral features of the user during the irritability psychology experimental paradigm assessment process;

[0119] The task-state thermal imaging and near-infrared mania assessment device is the device in the aforementioned embodiment.

[0120] In summary, this application has the following technical effects:

[0121] 1. This application triggers irritable emotions by constructing an irritable psychology experimental paradigm, making full use of the irritable traits unique to manic disorder. Through the innovatively designed negative feedback mechanism, the irritable emotional traits of patients with bipolar disorder can be effectively stimulated. Combining quantitative features with network models, a more accurate assessment and detection of emotional disorders is achieved, providing a reliable basis for the scientific diagnosis of emotional disorders.

[0122] 2. The multimodal thermal imaging and eye tracking acquisition platform provided by this application solves the problem that it is difficult to capture emotional characteristics under visible light. With the help of thermal imaging technology, real-time capture of facial expressions and pupil changes is achieved, and changes in temperature and pupil size after irritable emotions are accurately monitored. At the same time, the platform also synchronously collects the behavioral characteristics of the subjects' key reactions to ensure the synchronization and consistency of the data. This platform not only reduces the workload and testing costs of physicians, but also avoids the uncertainty brought about by traditional subjective scale evaluations, and improves the objectivity and accuracy of emotional disorder detection.

[0123] 3. This application independently constructs a multimodal neural network based on thermal imaging and eye tracking, and realizes the deep fusion and rapid classification processing of multidimensional data. Through the end-to-end real-time processing capability of the neural network, the judgment result can be given quickly; and the accuracy comes from two aspects: one is to accurately trigger emotional changes through a carefully designed irritability experimental paradigm, and the other is to use a multimodal platform to analyze from three dimensions of facial expressions, pupil changes and behavioral reactions to ensure the accuracy and comprehensiveness of the evaluation results. This fusion algorithm can be effectively applied to the accurate assessment of emotional disorders, promoting the innovation and development of emotional disorder detection technology.

[0124] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for assessing manic mood using task-state thermal imaging and near-infrared imaging, characterized in that: include: Acquiring the user's facial features, pupil features, and behavioral features during the irritability psychology experimental paradigm assessment process; the facial features include real-time facial temperature and facial thermal imaging images, the pupil features include real-time pupil size and near-infrared eye tracking images, and the behavioral features include key reaction time and key judgment accuracy; Constructing a multimodal fusion neural network, wherein the multimodal fusion neural network includes a thermal imaging image processing module, an eye tracking image processing module, a numerical matrix module, a cross-modal attention mechanism, a self-attention mechanism, a fusion module and a classifier; The facial expression features, the pupil features, and the behavioral features are input into the multimodal fusion neural network to obtain a manic emotion assessment result of the user.

2. The method according to claim 1, characterized in that The thermal imaging image processing module is used to obtain a feature representation of the thermal imaging image based on the facial thermal imaging image; The eye tracking image processing module is used to obtain a feature representation of the eye tracking image based on the near-infrared eye tracking image; The numerical matrix module is used to obtain a numerical feature representation based on the real-time facial temperature, the real-time pupil size, the key reaction time and the key determination accuracy; The feature representation of the thermal imaging image and the feature representation of the eye tracking image are input into the cross-modal attention mechanism to perform cross-modal attention calculation to obtain a cross-modal attention feature; The numerical feature representation is input into the self-attention mechanism to perform self-attention calculation to obtain a self-attention feature; The cross-modal attention feature and the self-attention feature are input into the fusion module for fusion to obtain a fusion feature; The fused features are input into the classifier to obtain the user's manic emotion assessment result.

3. The method according to claim 1, characterized in that The thermal imaging image processing module includes an image cutting and embedding unit, a Transformer encoder, a residual connection unit and a layer normalization unit; the Transformer encoder includes a plurality of Transformer units; The facial thermal imaging image is input into the image cutting and embedding unit to be cut into multiple image blocks, and each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension; After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation, and each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result; Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain a residual connection result; The residual connection result is input into the layer normalization unit to obtain a feature representation of the thermal imaging image.

4. The method according to claim 1, characterized in that The eye tracking image processing module includes an image preprocessing and enhancement unit, an image cutting and embedding unit, a Transformer encoder, a residual connection unit and a layer normalization unit; the Transformer encoder includes a plurality of Transformer units; The near-infrared eye tracking image is input into the image preprocessing and enhancement unit for noise suppression and edge feature extraction to obtain a processed image; The processed image is input into the image cutting and embedding unit to be cut into multiple image blocks, and each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension; After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation, and each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result; Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain a residual connection result; The residual connection result is input into the layer normalization unit to obtain a feature representation of the eye tracking image.

5. The method according to claim 1, characterized in that The numerical matrix module includes three fully connected convolutional layers, a token mixing unit, a channel mixing unit and a global pooling layer connected in sequence; The real-time facial temperature, the real-time pupil size, the key reaction time and the key determination accuracy are input into the three fully connected convolutional layers to perform feature dimension reduction and extraction to obtain numerical matrix data; The numerical matrix data is input into the token mixing unit for token mixing to obtain a token mixing result; The token mixing result is input into the channel mixing unit to obtain a channel mixing result; The channel mixing result is input into the global pooling layer to obtain a numerical feature representation.

6. A task-state thermal imaging and near-infrared manic mood assessment device, characterized in that: include: A feature acquisition module is used to acquire the user's facial features, pupil features, and behavioral features during the irritability psychology experimental paradigm assessment process; the facial features include real-time facial temperature and facial thermal imaging images, the pupil features include real-time pupil size and near-infrared eye tracking images, and the behavioral features include key reaction time and key judgment accuracy; A network construction module, used to construct a multimodal fusion neural network, wherein the multimodal fusion neural network includes a thermal imaging image processing module, an eye tracking image processing module, a numerical matrix module, a cross-modal attention mechanism, a self-attention mechanism, a fusion module and a classifier; An evaluation module is used to input the facial expression features, the pupil features, and the behavioral features into the multimodal fusion neural network to obtain an evaluation result of the user's manic emotion.

7. The device according to claim 6, characterized in that The thermal imaging image processing module is used to obtain a feature representation of the thermal imaging image based on the facial thermal imaging image; The eye tracking image processing module is used to obtain a feature representation of the eye tracking image based on the near-infrared eye tracking image; The numerical matrix module is used to obtain a numerical feature representation based on the real-time facial temperature, the real-time pupil size, the key reaction time and the key determination accuracy; The feature representation of the thermal imaging image and the feature representation of the eye tracking image are input into the cross-modal attention mechanism to perform cross-modal attention calculation to obtain a cross-modal attention feature; The numerical feature representation is input into the self-attention mechanism to perform self-attention calculation to obtain a self-attention feature; The cross-modal attention feature and the self-attention feature are input into the fusion module for fusion to obtain a fusion feature; The fused features are input into the classifier to obtain the user's manic emotion assessment result.

8. The device according to claim 6, characterized in that The thermal imaging image processing module includes an image cutting and embedding unit, a Transformer encoder, a residual connection unit and a layer normalization unit; the Transformer encoder includes a plurality of Transformer units; The facial thermal imaging image is input into the image cutting and embedding unit to be cut into multiple image blocks, and each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension; After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation, and each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result; Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain a residual connection result; The residual connection result is input into the layer normalization unit to obtain a feature representation of the thermal imaging image.

9. The device according to claim 6, characterized in that The eye tracking image processing module includes an image preprocessing and enhancement unit, an image cutting and embedding unit, a Transformer encoder, a residual connection unit and a layer normalization unit; the Transformer encoder includes a plurality of Transformer units; The near-infrared eye tracking image is input into the image preprocessing and enhancement unit for noise suppression and edge feature extraction to obtain a processed image; The processed image is input into the image cutting and embedding unit to be cut into multiple image blocks, and each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension; After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation, and each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result; Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain a residual connection result; The residual connection result is input into the layer normalization unit to obtain a feature representation of the eye tracking image.

10. The device according to claim 6, characterized in that The numerical matrix module includes three fully connected convolutional layers, a token mixing unit, a channel mixing unit and a global pooling layer connected in sequence; The real-time facial temperature, the real-time pupil size, the key reaction time and the key determination accuracy are input into the three fully connected convolutional layers to perform feature dimension reduction and extraction to obtain numerical matrix data; The numerical matrix data is input into the token mixing unit for token mixing to obtain a token mixing result; The token mixing result is input into the channel mixing unit to obtain a channel mixing result; The channel mixing result is input into the global pooling layer to obtain a numerical feature representation.

11. A task-state thermal imaging and near-infrared manic mood assessment system, characterized in that: include: A multimodal thermal imaging and eye tracking acquisition platform and a task-based thermal imaging and near-infrared manic mood assessment device; the multimodal thermal imaging and eye tracking acquisition platform includes a thermal imaging device, an eye tracker and a keyboard; The thermal imaging device is used to collect the facial features of the user during the irritability psychology experimental paradigm assessment process, the eye tracker is used to collect the pupil features of the user during the irritability psychology experimental paradigm assessment process, and the keyboard is used to collect the behavioral features of the user during the irritability psychology experimental paradigm assessment process; The task-state thermal imaging and near-infrared manic mood assessment device is the device described in any one of claims 6-10.

Images