Task-based thermal imaging and near-infrared manic mood assessment method, system and device

Abstract

The task-based thermal imaging and near-infrared mania assessment method, system, and device of this application captures the user's facial features, pupil features, and behavioral characteristics during an irritability psychology experimental paradigm assessment; constructs a multimodal fusion neural network; and inputs the facial features, pupil features, and behavioral features into the multimodal fusion neural network to obtain the user's mania assessment results. This application integrates thermal imaging and eye tracking technology to achieve real-time and accurate detection of emotional states such as manic mood disorder. This multimodal fusion technology breaks through the limitations of traditional methods and provides a new solution for the screening and assessment of emotional disorders. It has broad application prospects, especially in emotional management and mental health monitoring involving high-risk populations.

Description

Technical Field

[0001] The present application relates to the field of psychological emotion assessment, and specifically, to a method, system and device for assessing manic emotions using task-state thermal imaging and near-infrared imaging. Background Art

[0002] With the increase in social pressure and the accelerated pace of life, the incidence of mental and psychological disorders has shown a significant upward trend worldwide, especially among children, adolescents and adults. The onset of mental and psychological disorders not only seriously affects the physical and mental health of individuals, but also has a negative impact on the quality of family life and socioeconomic development. Mood disorders, especially manic mood disorders, are one of the most serious mood disorders. They are characterized by irritability, high spirits and extreme excitement, and are extremely risky. Therefore, timely and accurate screening and assessment of these mood disorders are particularly important in certain special industries. Traditional methods for assessing mood disorders usually rely on subjective questionnaires and behavioral observations. These methods have problems such as low assessment efficiency, reliance on human judgment, and a single detection modality, which limits the accuracy of the results. Summary of the Invention

[0003] In order to overcome at least one deficiency in the prior art, the present application provides a method, system and device for assessing manic mood using task-state thermal imaging and near-infrared imaging.

[0004] In a first aspect, a method for assessing manic mood using task-based thermal imaging and near-infrared imaging is provided, comprising:

[0005] Acquire users' facial expressions, pupil characteristics, and behavioral characteristics during the irritability psychology experimental paradigm assessment process; facial expressions include real-time facial temperature and facial thermal imaging images; pupil characteristics include real-time pupil size and near-infrared eye tracking images; behavioral characteristics include key press reaction time and key press judgment accuracy;

[0006] Construct a multimodal fusion neural network, which includes a thermal imaging image processing module, an eye tracking image processing module, a numerical matrix module, a cross-modal attention mechanism, a self-attention mechanism, a fusion module, and a classifier;

[0007] The facial expression features, pupil features, and behavioral features are input into the multimodal fusion neural network to obtain the user's manic emotion assessment results.

[0008] In one embodiment, the thermal imaging image processing module is used to obtain a feature representation of the thermal imaging image based on the facial thermal imaging image;

[0009] The eye tracking image processing module is used to obtain a feature representation of the eye tracking image based on the near-infrared eye tracking image;

[0010] The numerical matrix module is used to obtain numerical feature representation based on real-time facial temperature, real-time pupil size, key reaction time and key judgment accuracy;

[0011] The feature representation of the thermal imaging image and the feature representation of the eye tracking image are input into the cross-modal attention mechanism for cross-modal attention calculation to obtain cross-modal attention features;

[0012] The numerical feature representation is input into the self-attention mechanism for self-attention calculation to obtain the self-attention feature;

[0013] The cross-modal attention features and self-attention features are input into the fusion module for fusion to obtain the fusion features;

[0014] The fused features are input into the classifier to obtain the user's manic mood assessment result.

[0015] In one embodiment, the thermal imaging image processing module includes an image cutting and embedding unit, a Transformer encoder, a residual connection unit, and a layer normalization unit; the Transformer encoder includes multiple Transformer units;

[0016] The facial thermal imaging image is input into the image cutting and embedding unit and cut into multiple image blocks. Each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension.

[0017] After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation. Each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result;

[0018] Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain the residual connection result;

[0019] The residual connection result is input into the layer normalization unit to obtain the feature representation of the thermal imaging image.

[0020] In one embodiment, the eye tracking image processing module includes an image preprocessing and enhancement unit, an image cutting and embedding unit, a Transformer encoder, a residual connection unit, and a layer normalization unit; the Transformer encoder includes multiple Transformer units;

[0021] The near-infrared eye tracking image is input into the image preprocessing and enhancement unit for noise suppression and edge feature extraction to obtain a processed image;

[0022] The processed image is input into the image cutting and embedding unit and cut into multiple image blocks. Each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension.

[0023] After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation. Each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result;

[0024] Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain the residual connection result;

[0025] The residual connection result is input into the layer normalization unit to obtain the feature representation of the eye tracking image.

[0026] In one embodiment, the numerical matrix module includes three fully connected convolutional layers, a token mixing unit, a channel mixing unit, and a global pooling layer connected in sequence;

[0027] Real-time facial temperature, real-time pupil size, key reaction time, and key judgment accuracy are input into three fully connected convolutional layers for feature dimensionality reduction and extraction to obtain numerical matrix data;

[0028] The numerical matrix data is input into the token mixing unit for token mixing to obtain the token mixing result;

[0029] The token mixing result is input into the channel mixing unit to obtain the channel mixing result;

[0030] The channel mixing result is input into the global pooling layer to obtain the numerical feature representation.

[0031] In a second aspect, a task-state thermal imaging and near-infrared manic mood assessment device is provided, comprising:

[0032] The feature acquisition module is used to obtain the user's facial features, pupil features, and behavioral features during the irritability psychology experimental paradigm assessment process; facial features include real-time facial temperature and facial thermal imaging images; pupil features include real-time pupil size and near-infrared eye tracking images; behavioral features include key press reaction time and key press judgment accuracy;

[0033] A network building module is used to construct a multimodal fusion neural network. The multimodal fusion neural network includes a thermal imaging image processing module, an eye tracking image processing module, a numerical matrix module, a cross-modal attention mechanism, a self-attention mechanism, a fusion module, and a classifier.

[0034] The evaluation module is used to input facial expression features, pupil features, and behavioral features into the multimodal fusion neural network to obtain the user's manic emotion evaluation results.

[0035] In one embodiment, the thermal imaging image processing module is used to obtain a feature representation of the thermal imaging image based on the facial thermal imaging image;

[0036] The eye tracking image processing module is used to obtain a feature representation of the eye tracking image based on the near-infrared eye tracking image;

[0037] The numerical matrix module is used to obtain numerical feature representation based on real-time facial temperature, real-time pupil size, key reaction time and key judgment accuracy;

[0038] The feature representation of the thermal imaging image and the feature representation of the eye tracking image are input into the cross-modal attention mechanism for cross-modal attention calculation to obtain cross-modal attention features;

[0039] The numerical feature representation is input into the self-attention mechanism for self-attention calculation to obtain the self-attention feature;

[0040] The cross-modal attention features and self-attention features are input into the fusion module for fusion to obtain the fusion features;

[0041] The fused features are input into the classifier to obtain the user's manic mood assessment result.

[0042] In one embodiment, the thermal imaging image processing module includes an image cutting and embedding unit, a Transformer encoder, a residual connection unit, and a layer normalization unit; the Transformer encoder includes multiple Transformer units;

[0043] The facial thermal imaging image is input into the image cutting and embedding unit and cut into multiple image blocks. Each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension.

[0044] After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation. Each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result;

[0045] Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain the residual connection result;

[0046] The residual connection result is input into the layer normalization unit to obtain the feature representation of the thermal imaging image.

[0047] In one embodiment, the eye tracking image processing module includes an image preprocessing and enhancement unit, an image cutting and embedding unit, a Transformer encoder, a residual connection unit, and a layer normalization unit; the Transformer encoder includes multiple Transformer units;

[0048] The near-infrared eye tracking image is input into the image preprocessing and enhancement unit for noise suppression and edge feature extraction to obtain a processed image;

[0049] The processed image is input into the image cutting and embedding unit and cut into multiple image blocks. Each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension.

[0050] After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation. Each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result;

[0051] Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain the residual connection result;

[0052] The residual connection result is input into the layer normalization unit to obtain the feature representation of the eye tracking image.

[0053] In one embodiment, the numerical matrix module includes three fully connected convolutional layers, a token mixing unit, a channel mixing unit, and a global pooling layer connected in sequence;

[0054] Real-time facial temperature, real-time pupil size, key reaction time, and key judgment accuracy are input into three fully connected convolutional layers for feature dimensionality reduction and extraction to obtain numerical matrix data;

[0055] The numerical matrix data is input into the token mixing unit for token mixing to obtain the token mixing result;

[0056] The token mixing result is input into the channel mixing unit to obtain the channel mixing result;

[0057] The channel mixing result is input into the global pooling layer to obtain the numerical feature representation.

[0058] In a third aspect, a task-based thermal imaging and near-infrared manic mood assessment system is provided, comprising: a multimodal thermal imaging and eye tracking acquisition platform and a task-based thermal imaging and near-infrared manic mood assessment device; the multimodal thermal imaging and eye tracking acquisition platform comprises a thermal imaging device, an eye tracker, and a keyboard;

[0059] The thermal imaging device is used to collect the user's facial expression characteristics during the irritability psychology experimental paradigm evaluation process, the eye tracker is used to collect the user's pupil characteristics during the irritability psychology experimental paradigm evaluation process, and the keyboard is used to collect the user's behavioral characteristics during the irritability psychology experimental paradigm evaluation process; the task-state thermal imaging and near-infrared manic emotion evaluation device is the above-mentioned device.

[0060] Compared to existing technologies, this application offers the following advantages: The proposed method, system, and device for task-based thermal imaging and near-infrared manic mood assessment integrates thermal imaging and eye tracking technology to enable real-time, accurate detection of emotional states such as manic mood disorder. This multimodal fusion technology transcends the limitations of traditional methods and provides a new solution for the screening and assessment of mood disorders, particularly in the areas of mood management and mental health monitoring for high-risk populations, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The present application may be better understood by referring to the following description in conjunction with the accompanying drawings, which together with the following detailed description are incorporated into and form a part of this specification. In the drawings:

[0062] Figure 1 A flowchart of a method for assessing manic mood using task-state thermal imaging and near-infrared imaging according to an embodiment of the present application is shown;

[0063] Figure 2 A schematic diagram of real-time facial temperature recognition under thermal imaging images is shown, where (a) is when irritability is not activated and (b) is when irritability is activated;

[0064] Figure 3 A schematic diagram of real-time pupil recognition under near-infrared images is shown, where (a) is when irritability is not activated and (b) is when irritability is activated;

[0065] Figure 4 Shows the structural block diagram of the multimodal fusion neural network;

[0066] Figure 5 The following is a structural block diagram of a task-state thermal imaging and near-infrared manic mood assessment device according to an embodiment of the present application;

[0067] Figure 6 A schematic diagram of a multimodal thermal imaging and eye tracking acquisition platform is shown. DETAILED DESCRIPTION

[0068] Exemplary embodiments of the present application are described below with reference to the accompanying drawings. For the sake of clarity and conciseness, not all features of actual embodiments are described in this specification. However, it should be understood that in the process of developing any such actual embodiment, many implementation-specific decisions may be made to achieve the developer's specific goals, and these decisions may vary from one implementation to another.

[0069] It is also necessary to explain here that, in order to avoid obscuring the present application due to unnecessary details, the accompanying drawings only show the device structure closely related to the solution according to the present application, while other details that are not closely related to the present application are omitted.

[0070] It should be understood that the present application is not limited to the described embodiments due to the following description with reference to the accompanying drawings. In this document, where feasible, the embodiments may be combined with each other, features between different embodiments may be replaced or borrowed, and one or more features may be omitted in one embodiment.

[0071] This application provides a task-based thermal imaging and near-infrared method for assessing manic mood. First, based on the irritability characteristic of manic mood disorder, irritability is triggered by constructing an irritability psychology experimental paradigm. Next, a multimodal thermal imaging and eye tracking acquisition platform is built to simultaneously capture three external features of irritability after activation: real-time thermal imaging features of facial expressions, eye tracking features of pupil changes, and behavioral features of key triggering. Finally, a multimodal fusion neural network is independently constructed to fuse and quantify these three dimensions of features, enabling rapid and accurate detection of disorders such as manic mood disorder, addressing the current lack of objective assessment methods.

[0072] Figure 1 A flowchart of the method for assessing manic mood using task-state thermal imaging and near-infrared imaging according to an embodiment of the present application is shown. Figure 1 , methods include:

[0073] Step S1, obtaining the user's facial expression characteristics, pupil characteristics, and behavioral characteristics during the irritability psychology experimental paradigm assessment process; facial expression characteristics include real-time facial temperature and facial thermal imaging images, pupil characteristics include real-time pupil size and near-infrared eye tracking images, and behavioral characteristics include key reaction time and key judgment accuracy.

[0074] Irritability Psychology Experiment: Faces with expressions of joy and frustration appeared randomly on a screen, positioned randomly. Participants were asked to identify the location of the frustrated face and press the corresponding left or right button. A reward and punishment system with points was also implemented, with points displayed in real time in the upper right corner of the screen. Participants were required to respond quickly and accurately throughout the experiment to earn points. Correct responses received text feedback and added points, while incorrect responses received text feedback and deducted points.

[0075] The experimental paradigm evaluation process consists of two parts, namely experimental evaluation 1 and experimental evaluation 2.

[0076] In Experimental Assessment 1, participants will receive feedback on their accuracy. This means that the accuracy of their keystrokes will be used to determine whether they are correct or not. Correct keystrokes will display "Correct Answer" and increase points. Incorrect keystrokes will display "Incorrect Answer" and deduct points.

[0077] In Experimental Assessment 2, negative feedback trials were administered to activate irritability. This involved providing textual feedback stating "Too slow to answer" in 60% of the correct responses, without reference to the participant's actual response time. Points were deducted, thereby eliciting frustration and activating irritability. In the remaining 40% of the responses, accurate feedback was given based on whether the participant responded both quickly and accurately. The reference response time was the average response time from Experimental Assessment 1. If the response was correct and the response time was shorter than the average in Experimental Assessment 1, the textual feedback "Correct" was displayed, and points were added. If the response was correct and the response time was longer than the average in Experimental Assessment 1, the textual feedback "Too slow to answer" was displayed, and points were deducted. If the response was incorrect, the textual feedback "Incorrect answer" was displayed, and points were deducted.

[0078] In summary, the core of the irritability experimental paradigm is to fully stimulate the irritability characteristics unique to bipolar disorder through the negative feedback mechanism set in Assessment 2 in the paradigm.

[0079] When an irritable state is activated, blood circulation increases, capillaries become congested, and facial temperature rises. The system uses thermal imaging to detect real-time facial temperature. Furthermore, irritability triggers characteristic emotional expressions such as anger and mania. The system uses a neural network fusion algorithm based on thermal imaging to detect these emotions in real time. Real-time facial temperature data includes the highest, lowest, and average temperatures. Figure 2 A schematic diagram of real-time facial temperature recognition under thermal imaging images is shown, where (a) shows deactivated irritable emotions and (b) shows activated irritable emotions.

[0080] When an irritable state is activated, pupil changes reflect an individual's emotional fluctuations. Eye tracking equipment is used to capture pupil responses to external stimuli in real time, including pupil dilation and constriction, as well as reaction times. Real-time pupil size is used to quantify pupil changes under the influence of irritability. Near-infrared eye tracking images capture the speed and magnitude of pupil responses to stimuli by tracking eye movement data. Figure 3 A schematic diagram of real-time pupil recognition under near-infrared images is shown, where (a) shows deactivated irritability and (b) shows activated irritability.

[0081] Keystroke reaction time and keystroke accuracy are recorded. Longer reaction times and lower accuracy indicate slower information processing and poorer executive function, which often indicate a higher predisposition to mood disorders. Keystroke reaction time and keystroke accuracy can reveal a subject's ability to suppress manic emotions and their information processing speed.

[0082] Step S2: Construct a multimodal fusion neural network. The multimodal fusion neural network includes a thermal imaging image processing module, an eye tracking image processing module, a numerical matrix module, a cross-modal attention mechanism, a self-attention mechanism, a fusion module, and a classifier. It should be noted that the constructed multimodal fusion neural network is a trained network. The specific training process is conventional and will not be detailed here.

[0083] Step S3: input the facial expression features, pupil features, and behavioral features into a multimodal fusion neural network to obtain the user's manic emotion assessment result.

[0084] The thermal imaging image processing module, eye tracking image processing module, and numerical matrix module of this embodiment use different neural network modules to process their respective features, and then deeply fuse these features through a cross-modal attention mechanism and a fusion module. Ultimately, the classifier is used to identify people with irritable emotions. This method breaks through the limitations of traditional methods and provides a new solution for the screening and assessment of emotional disorders. It has broad application prospects, especially in emotional management and mental health monitoring involving high-risk groups.

[0085] In one embodiment, Figure 4 The structural diagram of the multimodal fusion neural network is shown in Figure 4 ,The thermal imaging image processing module is used to obtain the feature representation of the thermal imaging image based on the facial thermal imaging image;

[0086] The eye tracking image processing module is used to obtain a feature representation of the eye tracking image based on the near-infrared eye tracking image;

[0087] The numerical matrix module is used to obtain numerical feature representation based on real-time facial temperature, real-time pupil size, key reaction time and key judgment accuracy;

[0088] The feature representation of the thermal imaging image and the feature representation of the eye tracking image are input into the cross-modal attention mechanism for cross-modal attention calculation to obtain cross-modal attention features;

[0089] The numerical feature representation is input into the self-attention mechanism for self-attention calculation to obtain the self-attention feature;

[0090] The cross-modal attention features and the self-attention features are input into the fusion module for fusion to obtain fusion features; here, the fusion module concatenates the cross-modal attention features and the self-attention features, and performs regularization processing to obtain fusion features.

[0091] The fused features are input into a classifier to obtain the user's manic mood assessment result. Here, the classifier can use a Softmax classifier to classify people with irritable emotions. The user's manic mood assessment result indicates whether the user is irritable.

[0092] Specifically, the thermal imaging image processing module includes an image cutting and embedding unit, a Transformer encoder, a residual connection unit, and a layer normalization unit; the Transformer encoder includes multiple Transformer units;

[0093] The facial thermal imaging image is input into the image cutting and embedding unit and cut into multiple image patches. Each image patch is flattened and linearly mapped to obtain an embedding vector of fixed dimension.

[0094] After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation. Each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result;

[0095] Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain the residual connection result;

[0096] The residual connection result is input into the layer normalization unit to obtain the feature representation of the thermal imaging image.

[0097] Specifically, the eye tracking image processing module includes an image preprocessing and enhancement unit, an image cutting and embedding unit, a Transformer encoder, a residual connection unit, and a layer normalization unit; the Transformer encoder includes multiple Transformer units;

[0098] The near-infrared eye tracking image is input into the image preprocessing and enhancement unit for noise suppression and edge feature extraction to obtain a processed image; here, non-local means filtering (NLM) can be used to suppress noise in the near-infrared eye tracking image, and an adaptive Canny edge detection algorithm can be used to extract edge features.

[0099] The processed image is input into the image cutting and embedding unit and cut into multiple image blocks. Each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension.

[0100] After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation. Each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result;

[0101] Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain the residual connection result;

[0102] The residual connection result is input into the layer normalization unit to obtain the feature representation of the eye tracking image.

[0103] Specifically, the numerical matrix module includes three fully connected convolutional layers, a token mixing unit, a channel mixing unit, and a global pooling layer connected in sequence;

[0104] Real-time facial temperature, real-time pupil size, key reaction time, and key judgment accuracy are input into three fully connected convolutional layers for feature dimensionality reduction and extraction to obtain numerical matrix data;

[0105] The numerical matrix data is input into the token mixing unit for token mixing to obtain the token mixing result; the following formula is used:

[0106] u i =x num +W mix ·σ(W token ·x num )

[0107] Among them, u i is the token mixing result, x num is the numerical matrix data, W mix 、W token is a learnable parameter and σ is an activation function.

[0108] The token mixing result is input into the channel mixing unit to obtain the channel mixing result; the following formula is used:

[0109] v i =ui +W channel ·σ(W channel_mix ·u i )

[0110] Among them, v i is the channel mixing result, W channel 、W channel_mix is a learnable parameter and σ is an activation function.

[0111] The channel mixing result is input to the global pooling layer to obtain the numerical feature representation. To improve the representation ability of numerical features, a global pooling operation can be introduced after channel mixing. By summarizing the overall statistical characteristics of the numerical features, it can further compress the data dimension and retain key information.

[0112] Based on the same inventive concept as the task-state thermal imaging and near-infrared manic mood assessment method, this embodiment also provides a corresponding task-state thermal imaging and near-infrared manic mood assessment device. Figure 5 The following is a block diagram of a task-state thermal imaging and near-infrared manic mood assessment device according to an embodiment of the present application, including:

[0113] Feature acquisition module 51 is used to acquire the user's facial features, pupil features, and behavioral features during the irritability psychology experimental paradigm assessment process; facial features include real-time facial temperature and facial thermal imaging images; pupil features include real-time pupil size and near-infrared eye tracking images; behavioral features include key press reaction time and key press judgment accuracy;

[0114] a network construction module 52 for constructing a multimodal fusion neural network, the multimodal fusion neural network including a thermal imaging image processing module, an eye tracking image processing module, a numerical matrix module, a cross-modal attention mechanism, a self-attention mechanism, a fusion module, and a classifier;

[0115] The evaluation module 53 is used to input facial expression features, pupil features, and behavioral features into the multimodal fusion neural network to obtain the user's manic emotion evaluation result.

[0116] The task-state thermal imaging and near-infrared manic mood assessment device of this embodiment has the same inventive concept as the above-mentioned task-state thermal imaging and near-infrared manic mood assessment method. Therefore, the specific implementation method of the device can be seen in the embodiment part of the task-state thermal imaging and near-infrared manic mood assessment method in the above text, and its technical effects correspond to the technical effects of the above-mentioned method, which will not be repeated here.

[0117] The present application also provides a task-state thermal imaging and near-infrared manic mood assessment system, comprising: a multimodal thermal imaging and eye tracking acquisition platform and a task-state thermal imaging and near-infrared manic mood assessment device; the multimodal thermal imaging and eye tracking acquisition platform comprises a thermal imaging device, an eye tracker, and a keyboard; Figure 6 A schematic diagram of a multimodal thermal imaging and eye tracking acquisition platform is shown.

[0118] The thermal imaging device is used to collect the user's facial features during the irritability psychology experimental paradigm assessment process, the eye tracker is used to collect the user's pupil features during the irritability psychology experimental paradigm assessment process, and the keyboard is used to collect the user's behavioral features during the irritability psychology experimental paradigm assessment process;

[0119] The task-state thermal imaging and near-infrared manic mood assessment device is the device in the aforementioned embodiment.

[0120] In summary, this application has the following technical effects:

[0121] 1. This application utilizes an irritability psychology experimental paradigm to trigger irritability, leveraging the unique irritability characteristic of manic mood disorder. Through an innovative negative feedback mechanism, it effectively stimulates irritability in bipolar patients. Combining quantitative features with network models enables more accurate assessment and detection of mood disorders, providing a reliable basis for the scientific diagnosis of mood disorders.

[0122] 2. The multimodal thermal imaging and eye tracking acquisition platform provided in this application solves the problem of difficulty in capturing emotional characteristics under visible light. With the help of thermal imaging technology, real-time capture of facial expressions and pupil changes is achieved, and changes in temperature and pupil size after irritable emotions are activated are accurately monitored. At the same time, the platform also synchronously collects the behavioral characteristics of the subjects' keystroke reactions to ensure the synchronization and consistency of the data. This platform not only reduces the workload and testing costs of physicians, but also avoids the uncertainty brought about by traditional subjective scale assessments, and improves the objectivity and accuracy of emotional disorder detection.

[0123] 3. This application independently constructed a multimodal neural network based on thermal imaging and eye tracking, achieving deep fusion and rapid classification processing of multidimensional data. Through the end-to-end real-time processing capabilities of the neural network, judgment results can be quickly generated; accuracy comes from two aspects: first, precise triggering of emotional changes through a carefully designed irritability experimental paradigm, and second, the use of a multimodal platform to analyze facial expressions, pupil changes, and behavioral responses from three dimensions to ensure the accuracy and comprehensiveness of the assessment results. This fusion algorithm can be effectively applied to the precise assessment of emotional disorders, promoting the innovation and development of emotional disorder detection technology.

[0124] The above descriptions are merely examples of various embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any modifications or substitutions that can be readily conceived by a person skilled in the art within the technical scope disclosed in the present application should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for assessing manic mood using task-based thermal imaging and near-infrared imaging, characterized in that: include: Acquiring the user's facial features, pupil features, and behavioral features during the irritability psychology experimental paradigm assessment process; the facial features include real-time facial temperature and facial thermal imaging images, the pupil features include real-time pupil size and near-infrared eye tracking images, and the behavioral features include key press reaction time and key press judgment accuracy; Constructing a multimodal fusion neural network, the multimodal fusion neural network including a thermal imaging image processing module, an eye tracking image processing module, a numerical matrix module, a cross-modal attention mechanism, a self-attention mechanism, a fusion module, and a classifier; Inputting the facial expression features, the pupil features, and the behavioral features into the multimodal fusion neural network to obtain a manic emotion assessment result of the user; The thermal imaging image processing module is used to obtain a feature representation of the thermal imaging image based on the facial thermal imaging image; The eye tracking image processing module is used to obtain a feature representation of the eye tracking image based on the near-infrared eye tracking image; The numerical matrix module is used to obtain a numerical feature representation based on the real-time facial temperature, the real-time pupil size, the key reaction time and the key determination accuracy; The feature representation of the thermal imaging image and the feature representation of the eye tracking image are input into the cross-modal attention mechanism to perform cross-modal attention calculation to obtain a cross-modal attention feature; The numerical feature representation is input into the self-attention mechanism to perform self-attention calculation to obtain a self-attention feature; The cross-modal attention feature and the self-attention feature are input into the fusion module for fusion to obtain a fusion feature; The fused features are input into the classifier to obtain the user's manic mood assessment result.

2. The method according to claim 1, wherein The thermal imaging image processing module includes an image cutting and embedding unit, a Transformer encoder, a residual connection unit and a layer normalization unit; the Transformer encoder includes multiple Transformer units; The facial thermal imaging image is input into the image cutting and embedding unit to be cut into multiple image blocks, and each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension; After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation. Each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result; Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain a residual connection result; The residual connection result is input into the layer normalization unit to obtain a feature representation of the thermal imaging image.

3. The method according to claim 1, wherein The eye tracking image processing module includes an image preprocessing and enhancement unit, an image cutting and embedding unit, a Transformer encoder, a residual connection unit and a layer normalization unit; the Transformer encoder includes multiple Transformer units; The near-infrared eye tracking image is input into the image preprocessing and enhancement unit for noise suppression and edge feature extraction to obtain a processed image; The processed image is input into the image cutting and embedding unit to be cut into multiple image blocks, and each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension; After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation. Each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result; Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain a residual connection result; The residual connection result is input into the layer normalization unit to obtain a feature representation of the eye tracking image.

4. The method according to claim 1, wherein The numerical matrix module includes three fully connected convolutional layers, a token mixing unit, a channel mixing unit and a global pooling layer connected in sequence; The real-time facial temperature, the real-time pupil size, the key reaction time, and the key determination accuracy are input into the three fully connected convolutional layers to perform feature dimensionality reduction and extraction to obtain numerical matrix data; The numerical matrix data is input into the token mixing unit for token mixing to obtain a token mixing result; The token mixing result is input into the channel mixing unit to obtain a channel mixing result; The channel mixing result is input into the global pooling layer to obtain a numerical feature representation.

5. A task-state thermal imaging and near-infrared manic mood assessment device, characterized in that: include: A feature acquisition module is used to acquire the user's facial features, pupil features, and behavioral features during the irritability psychology experimental paradigm assessment process; the facial features include real-time facial temperature and facial thermal imaging images, the pupil features include real-time pupil size and near-infrared eye tracking images, and the behavioral features include key press reaction time and key press judgment accuracy; A network construction module for constructing a multimodal fusion neural network, wherein the multimodal fusion neural network includes a thermal imaging image processing module, an eye tracking image processing module, a numerical matrix module, a cross-modal attention mechanism, a self-attention mechanism, a fusion module, and a classifier; An evaluation module, configured to input the facial expression features, the pupil features, and the behavioral features into the multimodal fusion neural network to obtain an evaluation result of the user's manic mood; The thermal imaging image processing module is used to obtain a feature representation of the thermal imaging image based on the facial thermal imaging image; The eye tracking image processing module is used to obtain a feature representation of the eye tracking image based on the near-infrared eye tracking image; The numerical matrix module is used to obtain a numerical feature representation based on the real-time facial temperature, the real-time pupil size, the key reaction time and the key determination accuracy; The feature representation of the thermal imaging image and the feature representation of the eye tracking image are input into the cross-modal attention mechanism to perform cross-modal attention calculation to obtain a cross-modal attention feature; The numerical feature representation is input into the self-attention mechanism to perform self-attention calculation to obtain a self-attention feature; The cross-modal attention feature and the self-attention feature are input into the fusion module for fusion to obtain a fusion feature; The fused features are input into the classifier to obtain the user's manic mood assessment result.

6. The device according to claim 5, characterized in that The thermal imaging image processing module includes an image cutting and embedding unit, a Transformer encoder, a residual connection unit and a layer normalization unit; the Transformer encoder includes multiple Transformer units; The facial thermal imaging image is input into the image cutting and embedding unit to be cut into multiple image blocks, and each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension; After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation. Each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result; Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain a residual connection result; The residual connection result is input into the layer normalization unit to obtain a feature representation of the thermal imaging image.

7. The device according to claim 5, characterized in that The eye tracking image processing module includes an image preprocessing and enhancement unit, an image cutting and embedding unit, a Transformer encoder, a residual connection unit and a layer normalization unit; the Transformer encoder includes multiple Transformer units; The near-infrared eye tracking image is input into the image preprocessing and enhancement unit for noise suppression and edge feature extraction to obtain a processed image; The processed image is input into the image cutting and embedding unit to be cut into multiple image blocks, and each image block is flattened and linearly mapped to obtain an embedding vector of fixed dimension; After the fixed-dimensional embedding vector is added to the position code, it is input into the Transformer encoder for multi-head self-attention calculation. Each Transformer unit of the Transformer encoder outputs a multi-head self-attention calculation result; Each multi-head self-attention calculation result is input into the residual connection unit for residual connection to obtain a residual connection result; The residual connection result is input into the layer normalization unit to obtain a feature representation of the eye tracking image.

8. The device according to claim 5, wherein The numerical matrix module includes three fully connected convolutional layers, a token mixing unit, a channel mixing unit and a global pooling layer connected in sequence; The real-time facial temperature, the real-time pupil size, the key reaction time, and the key determination accuracy are input into the three fully connected convolutional layers to perform feature dimensionality reduction and extraction to obtain numerical matrix data; The numerical matrix data is input into the token mixing unit for token mixing to obtain a token mixing result; The token mixing result is input into the channel mixing unit to obtain a channel mixing result; The channel mixing result is input into the global pooling layer to obtain a numerical feature representation.

9. A task-based thermal imaging and near-infrared manic mood assessment system, characterized in that: include: A multimodal thermal imaging and eye tracking acquisition platform and a task-based thermal imaging and near-infrared manic mood assessment device; the multimodal thermal imaging and eye tracking acquisition platform includes a thermal imaging device, an eye tracker, and a keyboard; The thermal imaging device is used to collect facial features of the user during the irritability psychology experimental paradigm assessment process, the eye tracker is used to collect pupil features of the user during the irritability psychology experimental paradigm assessment process, and the keyboard is used to collect behavioral features of the user during the irritability psychology experimental paradigm assessment process; The task-state thermal imaging and near-infrared manic mood assessment device is the device described in any one of claims 5 to 8.

Images