A method and system for analyzing facial pain expressions based on spatiotemporal information

Abstract

This invention discloses a method and system for facial pain expression analysis based on spatiotemporal information. By sampling facial expression videos using a sliding window, several video segments are obtained, improving dataset utilization. Discrete cosine transform is applied to the video frames, followed by sawtooth scanning for enhancement, resulting in detailed feature enhancement maps that enhance detail information. Then, all video frames and their corresponding detailed feature enhancement maps from each sampled video segment are combined and input into a Transformer model. The Transformer model uses a concatenated cross-processing method to extract and fuse temporal and spatial features. Finally, the pain analysis result is obtained based on the pain level corresponding to each sampled video segment. This method effectively extracts feature information from facial expression videos, resulting in high accuracy in pain assessment.

Description

Technical Field

[0001] This invention relates to the field of pain assessment technology, and in particular to a method and system for analyzing facial pain expressions based on spatiotemporal information. Background Technology

[0002] Internationally, pain assessments are typically conducted manually by specially trained healthcare professionals familiar with various monitoring indicators. These assessments rely heavily on individual knowledge and experience, and are influenced by subjective factors such as personal emotions, thus failing to fully and objectively reflect the patient's pain level. Therefore, developing a facial expression-based pain analysis method to provide objective and accurate pain assessments for clinical practice is of paramount importance and value, enabling healthcare professionals to implement timely and appropriate analgesic measures.

[0003] Current pain assessment methods primarily analyze facial expression images to obtain pain levels, failing to consider that facial expressions are a dynamic process with both temporal and spatial sequence information. Therefore, feature extraction is insufficient, leading to low accuracy in pain assessment. Furthermore, current video segmentation methods merely divide the video into segments, increasing the number of segments without fundamentally improving the overall dataset size or utilization. Summary of the Invention

[0004] The main objective of this invention is to provide a method, system, smart terminal, and storage medium for facial pain expression analysis based on spatiotemporal information, which can solve the problems of insufficient feature information extraction, low accuracy of pain assessment, and low utilization of datasets.

[0005] To achieve the above objectives, a first aspect of the present invention provides a method for facial pain expression analysis based on spatiotemporal information, the method comprising:

[0006] Get facial expression videos;

[0007] The facial expression video is sampled using a sliding window to obtain several sampled video segments;

[0008] For each video frame of each sampled video segment, a discrete cosine transform is performed to obtain the frequency domain matrix corresponding to each video frame; the frequency domain matrix is ​​enhanced by a sawtooth scan and then an inverse discrete cosine transform is performed to obtain the detail feature enhancement map corresponding to each video frame.

[0009] All video frames and corresponding detail feature enhancement maps of each sampled video are combined and input into the Transformer model. Temporal and spatial features are extracted in the Transformer model in a concatenated and cross-processed manner to obtain the pain level corresponding to each sampled video.

[0010] The pain analysis results are obtained and output based on the pain levels corresponding to all sampled videos.

[0011] Optionally, the step of sampling the facial expression video using a sliding window to obtain several sampled video segments includes:

[0012] The sliding window is initialized based on the preset start frame position, total number of sampling frames, and interval frame number of the first sampled video segment; the video frames of the facial expression video are sampled within the sliding window according to the interval frame number to obtain the first sampled video segment;

[0013] Set the next sampled video segment as the target video segment;

[0014] The sliding window is slid according to the movement step size;

[0015] When the preset total number of sampling frames and interval frame count of the target video segment are different from the preset total number of sampling frames and interval frame count of the previous sampled video segment, the sliding window is updated according to the preset total number of sampling frames and interval frame count of the target video segment.

[0016] The target video segment is obtained by sampling video frames of the facial expression video within a sliding window according to a preset interval frame number.

[0017] Return to set the next sampled video segment as the target video segment to obtain the next sampled video segment, and continue until all sampled video segments are obtained.

[0018] Optionally, the enhancement of the frequency domain matrix using a sawtooth scan includes:

[0019] A sawtooth scanning method is used, starting from the lower right corner of the frequency domain matrix, to sequentially obtain frequency domain feature information. If the current scanning position number is even, the frequency domain feature information of the previous position is used to replace the frequency domain feature information of the current scanning position.

[0020] Optionally, a discrete cosine transform is performed on the video frames to obtain the frequency domain matrix corresponding to the video frames, including:

[0021] Based on a preset image block size, the video frame is divided into several image blocks;

[0022] For each image block, the spatial domain image feature information is transformed into the frequency domain frequency feature information by using discrete cosine transform, thus obtaining the frequency domain frequency feature information corresponding to each image block.

[0023] Based on the position of the image block in the video frame, all frequency domain feature information is combined to obtain the frequency domain matrix.

[0024] Optionally, all video frames of the sampled video and their corresponding detail feature enhancement maps are combined and input into the Transformer model, including:

[0025] Each video frame of the sampled video and its corresponding detail feature enhancement map are combined and segmented according to a preset image block size to obtain an image block sequence;

[0026] The image block sequence is flattened, linearly projected, and has position markers added before being input into the Transformer encoder.

[0027] Optionally, the extraction of temporal and spatial features in the Transformer model using a concatenated crossover method includes:

[0028] The spatial features extracted by all attention heads in the multi-head self-attention module are fused with the temporal features extracted by the previous network layer and then input into the multilayer perceptron and residual connection are used to obtain the spatial features.

[0029] The temporal features extracted by all attention heads in the multi-head self-attention module are fused with the spatial features extracted by the previous network layer and then input into the multilayer perceptron and residual connection are used to obtain the temporal features.

[0030] Optionally, obtain the temporal or spatial features extracted by the attention heads in the multi-head self-attention module, including:

[0031] Perform a dot product operation on the query matrix and the keyword matrix to obtain similarity weights used to characterize the relevance between feature elements;

[0032] Based on the similarity weights and numerical matrices, the temporal or spatial features extracted by the attention head of the self-attention module are obtained.

[0033] A second aspect of the present invention provides a facial pain expression analysis system based on spatiotemporal information, wherein the system comprises:

[0034] The data acquisition module is used to acquire facial expression videos;

[0035] A sliding window module is used to sample the facial expression video using a sliding window to obtain several sampled video segments;

[0036] The detail feature enhancement map module is used to perform discrete cosine transform on each video frame of each sampled video segment to obtain the frequency domain matrix corresponding to each video frame; after enhancing the frequency domain matrix by sawtooth scanning, inverse discrete cosine transform is performed to obtain the detail feature enhancement map corresponding to each video frame.

[0037] The pain level analysis module is used to combine all video frames and corresponding detail feature enhancement maps of each sampled video and input them into the Transformer model. In the Transformer model, temporal and spatial features are extracted in a concatenated and cross-processed manner to obtain the pain level corresponding to each sampled video.

[0038] The results module is used to obtain and output pain analysis results based on the pain levels corresponding to all sampled videos.

[0039] A third aspect of the present invention provides a smart terminal, the smart terminal including a memory, a processor, and a spatiotemporal information-based facial pain expression analysis program stored in the memory and executable on the processor, wherein the spatiotemporal information-based facial pain expression analysis program, when executed by the processor, implements any of the steps of the spatiotemporal information-based facial pain expression analysis method.

[0040] A fourth aspect of the present invention provides a computer-readable storage medium storing a facial pain expression analysis program based on spatiotemporal information, wherein the facial pain expression analysis program based on spatiotemporal information, when executed by a processor, implements any of the steps of the aforementioned facial pain expression analysis method based on spatiotemporal information.

[0041] As can be seen from the above, compared with the prior art, this invention improves the utilization of the dataset by using a sliding window to sample facial expression videos to obtain several sampled video segments. Discrete cosine transform is applied to the video frames, and zigzag scanning is used for enhancement to obtain detail feature enhancement maps, which enhance detail information. Then, all video frames of each sampled video segment and the corresponding detail feature enhancement maps are combined and input into a Transformer model. In the Transformer model, temporal and spatial features are extracted and fused using a concatenated cross-processing method. Finally, pain analysis results are obtained based on the pain level corresponding to each sampled video segment. This invention can fully extract feature information from facial expression videos, achieves high accuracy in pain assessment, and demonstrates strong temporal predictive power in sequence prediction. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the facial pain expression analysis method based on spatiotemporal information provided in an embodiment of the present invention;

[0044] Figure 2 yes Figure 1 A schematic diagram of the sliding window sampling in the embodiment;

[0045] Figure 3 yes Figure 1 A schematic diagram illustrating the specific process of obtaining the frequency domain matrix corresponding to the video frame in step S300 of the embodiment;

[0046] Figure 4 yes Figure 1 Schematic diagram of sawtooth scan enhancement in the embodiment;

[0047] Figure 5 yes Figure 1 Implementation diagram of Transformer architecture;

[0048] Figure 6 yes Figure 1 A schematic diagram illustrating the specific process of extracting temporal or spatial features in step S400 of the embodiment;

[0049] Figure 7 It is a pain level assessment curve plotted from an experiment;

[0050] Figure 8 This is a schematic diagram of the structure of the facial pain expression analysis system based on spatiotemporal information provided in an embodiment of the present invention;

[0051] Figure 9 This is a block diagram illustrating the internal structure of a smart terminal provided in an embodiment of the present invention. Detailed Implementation

[0052] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0053] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0054] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0055] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0056] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0059] Scientific pain relief remains a challenge in clinical medicine. Accurate pain assessment is the first step in pain management and is crucial throughout the entire treatment process. Scientific pain monitoring helps aphasic patients express pain and discomfort during medical and surgical procedures in real time, enabling physicians to rationally control subsequent medication and medical procedures. Therefore, quantitative monitoring of pain levels is of paramount importance in healthcare.

[0060] Current research primarily focuses on image classification, failing to consider that facial expressions are a dynamic process with both temporal and spatial sequence information. This leads to insufficient feature extraction and low accuracy in pain assessment. Furthermore, current video segmentation methods merely divide the video into segments, increasing the number of segments without fundamentally improving the overall dataset size and utilization.

[0061] To address the aforementioned technical challenges, this invention provides a facial pain expression analysis method based on spatiotemporal information. It introduces a sliding window approach to segment videos, improving dataset utilization, and performs high-frequency data augmentation on video frames to enhance detail. The Transformer model employs a concatenated and cross-functional approach to extract and fuse temporal and spatial features, achieving spatiotemporal feature extraction and fusion in O(n) time complexity. This method provides comprehensive feature extraction, high accuracy in pain assessment, and enables end-to-end information processing and detection, effectively addressing complex real-world clinical application scenarios.

[0062] Exemplary methods

[0063] This invention provides a facial pain expression analysis method based on spatiotemporal information, used to assess pain based on facial expression videos. It can be deployed on hospital servers or on various mobile terminals, PCs, and other electronic devices. Specifically, for example... Figure 1 As shown, this embodiment includes the following steps:

[0064] Step S100: Acquire facial expression video;

[0065] Specifically, facial expression videos refer to videos obtained by capturing images of the face. These videos can be pre-recorded and saved, or they can be real-time videos captured by a camera. The source of the facial expression video is not limited; it can be obtained from a hospital's video server or from videos captured by mobile devices such as smartphones. The facial expression video needs to be of a certain length to ensure that the required number of video segments can be obtained during sampling.

[0066] Step S200: Use a sliding window to sample facial expression videos to obtain several sampled video segments;

[0067] Traditional video segmentation simply divides facial expression videos into segments, increasing the number of segments and obtaining multiple sampled videos, without fundamentally improving the overall dataset size and utilization. This invention employs a sliding window approach to segment facial expression videos, which not only expands the number of video segments but also improves the utilization of each frame to some extent, resulting in higher overall dataset utilization.

[0068] In this embodiment, when acquiring the first sampled video segment, the sliding window is initialized according to the preset starting frame position, total number of sampled frames, and interval frame number of the first sampled video segment; that is, the starting frame position is set as the starting position of the sliding window, and the length of the sliding window is calculated based on the total number of sampled frames and the interval frame number. Figure 2As shown, for example, if the total number of sampling frames is 9 and the interval frame number is 4, then the length of the sliding window must contain at least (9-1)*4+1=33 frames. Then, within the sliding window, video frames of the facial expression video are sampled according to the interval frame number to obtain the first sampled video segment. Afterwards, the next sampled video segment is used as the target video segment (i.e., the sampled video to be obtained), and the movement is performed according to a preset step size (…). Figure 2 As shown in Figure 2), the sliding window is moved to the next starting position, and video frames of the facial expression video are sampled within the sliding window according to the interval frame number to obtain the target video segment; then the sliding window is moved to the next starting position again for resampling until all video segments are obtained.

[0069] In another embodiment, if the total number of sampling frames in the next sampled video segment is different from the total number of sampling frames in the previous sampled video segment, or if the interval frame number in the next sampled video segment is different from the interval frame number in the previous sampled video segment, then the length of the sliding window in the next sampled video segment is different from the length of the sliding window in the previous sampled video segment. Therefore, after sliding the sliding window to the next starting position, the length of the sliding window needs to be updated according to the preset total number of sampling frames and interval frame number of the target video segment. Then, within the sliding window, video frames of the facial expression video are sampled according to the interval frame number to obtain the target video segment.

[0070] Assume the total number of frames in the sampled video is n, which is the total number of frames selected on the sliding window (the length of which does not exceed the length of the facial expression video). Each time a complete sliding window is hovering over a segment of facial expression video, the selected video frames automatically form a sampled video segment F, where F = {f1, f2, ..., f...}. n The position of the first frame of the sampled video is the starting frame position P. begin P begin =Position(f1), where the distance between adjacent frames selected to form the sampled video is the number of frames g between adjacent frames, and g = |Position(f1)|. i+1 )-Position(f i When the sliding window moves across the facial expression video, the number of frames between the starting frames of two consecutive sampled video sequences is the step size s of the sliding window in each movement.

[0071] Figure 2 The image shown is a 35-frame video of facial expressions, V = {v1, v2, ..., v...}. 35Two video samples, each containing 9 frames, were acquired using a sliding window. The narrow dashed box represents the initial position of the sliding window. The starting position of the sliding window is Position(v1), and the interval between adjacent frames within the sliding window is g = 4. The relevant parameters of this sliding window are as follows: total number of samples n = 9, sampled video F... blue ={f1, f2,...,f9}={v1, v5, v9, v 13 v 17 v 21 v 25 v 29 v 33}; Start frame position P begin =Position(f1)=Position(v1), the number of frames between adjacent frames g=|Position(f2)-Position(f1)|=|Position(v5)-Position(v1)|=4. The wide dashed box is the sliding window after the above sliding window slides a distance s=2, its starting position is Position(v3), the number of frames between adjacent frames is the same as the previous sliding window, also g=4, and a sampled video F with n=9 frames can also be extracted from this sliding window. yellow The relevant parameters for this sliding window are as follows: total number of sampling frames n = 9, sampling video F yellow ={f1, f2,...,f9}={v3, v7, v 11 v 15 v 19 v 23 v 27 v 31 v 35}; Start frame position P begin =Position(f1)=Position(v3), the number of frames between adjacent frames g=|Position(f2)-Position(f1)|=|Position(v7)-Position(v3)|=4.

[0072] By using different parameters such as the total number of sampling frames and the interval frame count for the sliding window, the utilization rate of each frame in the video can be improved. (Still using...) Figure 2 For example, the first sliding window has an interval of 4 frames, and the sampled videos are v1, v5, v9, v 13 ,v 17 ,v 21 Assuming the second sliding window has an interval of 3 frames and a movement step of 2, then the second sampled video segment is v3, v6, v9, v 12 ,v 15 ,v18 ,v 21 The frame number in the sliding window indicates that video frame v9,v 21 It was sampled twice, improving the utilization of the dataset.

[0073] Step S300: Perform discrete cosine transform on each video frame of each sampled video segment to obtain the frequency domain matrix corresponding to each video frame; enhance the frequency domain matrix by sawtooth scanning and then perform inverse discrete cosine transform to obtain the detail feature enhancement map corresponding to each video frame.

[0074] Specifically, to extract as many detailed features as possible from facial expressions of pain, it is necessary to effectively separate these detailed features. While the spatial distribution of image information follows the structure of the image content without a discernible pattern, the distribution of low-frequency and high-frequency information in the frequency space exhibits certain regularities. This facilitates the effective separation of detailed features. By referencing the lossy image compression method of JPEG, a discrete cosine transform algorithm can be used to convert spatial domain image features into frequency domain features.

[0075] In this embodiment, the frequency domain matrix corresponding to the video frame is obtained by performing a discrete cosine transform on the video frame, such as... Figure 3 As shown, the specific steps include:

[0076] Step S310: Divide the video frame into several image blocks based on the preset image block size;

[0077] Step S320: For each image block, the spatial domain image feature information is transformed into the frequency domain frequency feature information by applying discrete cosine transform to obtain the frequency domain frequency feature information corresponding to each image block.

[0078] Step S330: Based on the position of the image block in the video frame, combine all the frequency domain feature information to obtain the frequency domain matrix.

[0079] Specifically, assuming the original video frame size is M*N and the preset image block size is B*B, the original single video frame is cut into (M / B*N / B) small image blocks. Then, a two-dimensional DCT (Discrete Cosine Transform) transformation is performed on each block to convert the spatial domain image feature information into the frequency domain frequency feature information, resulting in (M / B*N / B) DCT frequency domain data blocks. After arranging the DCT frequency domain data blocks in order according to the spatial position relationship of the original image blocks, all the DCT frequency domain data blocks are assembled into a complete DCT frequency domain matrix. At this time, the size of the DCT frequency domain matrix is ​​consistent with the size of the original video frame.

[0080] Facial expression videos are typically stored in RGB encoding, meaning they represent specific pain expressions using red, green, and blue chromaticity. However, this RGB method of expressing information through chromaticity neglects the crucial information of brightness. According to the Pain Intensity Scale (PSPI) formula, the degree of pain expression can be quantified by the movement of facial features such as eyebrows, cheeks, mouth, and nose. These minute facial unit movements involve facial muscle movements, resulting in contrasts in brightness between different areas at the same time, or between the same area at different times. Therefore, brightness is an indispensable piece of information for extracting facial pain expression features and is a very important quantitative indicator. Preferably, the RGB-encoded video frames are converted to YC encoding before performing a discrete cosine transform on the video frames. b C r Encoding format.

[0081] YC b C r The color model and the RGB color model can be quantified into a linear relationship, with the specific formula as follows:

[0082]

[0083] In computers, YC b C r The encoding format is the same as that of RGB, both using 8-bit encoding to represent the information of the corresponding channel. However, the difference is that its value space is not the uniform 256-level range of [0, 255] for RGB. The value range of the luminance component Y is [16, 235], and the value range of the chrominance component C... b and C r The values ​​for all components are in the range [16, 240]. Therefore, if the calculated value of the luminance component Y exceeds 235, it will be truncated to 235; similarly, if the value of the chrominance component C... b and C r If the calculated value exceeds 240, it will be truncated to 240.

[0084] Considering that detailed feature information is stored in high-frequency features, and that frequency domain feature information increases sequentially along the diagonal from the top left to the bottom right, a zigzag scanning method is adopted. The scan begins from the bottom right corner of the frequency domain matrix, sequentially acquiring frequency domain feature information. If the current scan position number is even, the frequency domain feature information of the previous position is used to replace the current scan position's frequency domain feature information. That is, a "Z"-shaped scan is performed sequentially from the bottom right corner of the frequency domain matrix, and high-frequency features are enhanced according to the scan order. For the frequency domain feature information at the 2i-th position (where i's value does not exceed half the number of pixels in the cut image block), it is replaced by the frequency domain feature information of the previous position (2i-1) in the scan path. After one complete "Z"-shaped scan, a two-dimensional frequency domain data is obtained that has undergone overall high-frequency enhancement while maintaining its original frequency distribution. For a better understanding of this enhancement rule, please refer to [link to relevant documentation]. Figure 4 ,in accordance with Figure 4 The coloring order of the rightmost color bar follows the Zigzag scanning path, where the high-frequency feature data of the previous position is overlaid on the frequency domain data of the next position.

[0085] A sawtooth scan is used to enhance the frequency domain matrix, followed by a discrete cosine inverse transform to obtain the enhanced detail feature map corresponding to each video frame.

[0086] Step S400: Combine all video frames and corresponding detail feature enhancement maps of each sampled video segment and input them into the Transformer model respectively. Extract temporal and spatial features in the Transformer model in a concatenated and cross-processed manner to obtain the pain level corresponding to each sampled video segment.

[0087] Specifically, the size of the video frame image is H*W*3, and the size of the detail enhancement map is H*W*1. For ease of processing, they are uniformly considered as image frame data of size H*W*4 per frame. However, when actually input into the Transformer model, the interaction between a video frame image and its corresponding detail enhancement map is maintained, ensuring that images from the same source are adjacent. Therefore, the data input into the Transformer model is of size X∈R. H×W×4×F The video sequence.

[0088] Each sampled video segment is processed separately. Each video frame of the sampled video is combined with the corresponding detail feature enhancement map. The combined result of each sampled video segment is then input into the Transformer model and segmented according to the preset image patch size to obtain an image patch sequence. The image patch sequence is then flattened, linearly projected, and positional markers are added before being input into the Transformer encoder for extraction of temporal and spatial features.

[0089] Specifically, the combined size of the sampled video frames and the detail feature enhancement map is X∈R. H×W×4×F The image is divided into image blocks according to a preset image block size (P*P), resulting in N image block sequences of size P*P*4*F. Preferably, the image block size P is the same as the image block size B used in step S300 when performing the discrete cosine transform on the video frame. N is the number of image block sequences after division, i.e., N = (H*W) / P. 2 .

[0090] Since the standard Transformer model is often used to process long sequences of one-dimensional data, it is necessary to perform one-dimensional processing on the input image patch sequence of size P*P*4*F, that is, to flatten the image patch sequence to obtain one-dimensional tensor data. Where p represents the position identifier on the two-dimensional plane, with a value range of [1, N]; t represents the frame-level index of the one-dimensional tensor, with a value range of [1, F]. This module flattens the N*P*P*4*F image block sequence into N*4*P. 2 Sequence tensor data.

[0091] Because a simple flattening operation results in an excessively long dimension for the flattened one-dimensional tensor, if the image patch size is 10*10, the flattened tensor data dimension would be 1*4*10*10, or 1*400. However, in the Transformer model, all network layers use a constant latent tensor dimension *d*. Therefore, by adding a trainable linear mapping layer to the Transformer model, the dimension of the one-dimensional tensor is mapped to a specified embedding dimension *d*, which is called the embedding of the image patch representation sequence, similar to word vectors in natural language processing.

[0092] Considering that different features in two-dimensional image data exhibit different information at different locations, and that the Transformer model cannot recognize positional information from the transformed one-dimensional tensor information of the input, making it difficult to understand the positional relationships between input sequences, it is necessary to add positional markers to the input one-dimensional tensor information, such as... Figure 5 A vector that carries numbers.

[0093] When training the Transformer model, in addition to adding location markers, a data label tensor is also needed to compare the results with the model's predictions, thereby calculating the loss value and optimizing the model training. Since the input video data consists of F frames, the label tensor should contain the labels corresponding to these F video frames, i.e., the pain level of the facial expressions. Then, the dimensions of this label tensor are padded to the specified embedding dimension d.

[0094] Traditional 3D convolutional neural networks for video prediction require significant memory and computational resources, have a large number of parameters, and are time-consuming, making it difficult to meet real-time clinical requirements. Therefore, the Transformer model was used. Furthermore, the Transformer model in this embodiment is based on the Vision Transformer (visual self-attention model), and its encoder has been improved.

[0095] The Transformer encoder is the most critical structure in the entire model. It consists of multiple identical encoder modules stacked repeatedly. That is, the output of the previous layer serves as the input of the next layer. Through the transmission of encoder modules layer by layer, the output is finally fed into the fully connected network layer for result prediction.

[0096] The encoder module is mainly composed of alternating multi-headed self-attention (MSA) sub-modules and multi-layer perceptron (MLP) sub-modules. Each sub-module first performs layer normalization (LN) to accelerate convergence and improve training speed. Residual connections are also added after each sub-module to prevent overfitting during training.

[0097] Multi-head self-attention modules can simultaneously extract spatial and temporal information features. For example... Figure 6 As shown, extracting temporal or spatial features specifically includes the following steps:

[0098] Step S410: Perform a dot product operation on the query matrix and the keyword matrix to obtain the similarity weights used to characterize the relevance between feature elements;

[0099] Step S420: Based on the similarity weights and numerical matrix, obtain the temporal or spatial features extracted by the attention head of the self-attention module.

[0100] Specifically, when extracting spatial features, to better capture facial pain expression features and make the model more efficient, a self-attention mechanism is guided to extract pain feature information from the sampled video in the spatial dimension. When only the spatial dimension is considered, feature extraction is performed within the same frame. During model training, the self-attention module involves N+1 query-key pairs, comprising N image patch sequences and one separate data label block. During model usage, the self-attention module involves N query-key pairs, i.e., N image patch sequences.

[0101] When each self-attention module's attention head extracts spatial features, it first performs a dot product operation on the query matrix Q and the keyword matrix K to obtain the relevance between the target feature element and the other feature elements, i.e., the similarity weight.

[0102]

[0103] Next, features from the numerical matrix V (Value) are selected based on the calculated similarity weights, thus completing the spatial feature extraction process for each attention head of the self-attention module. The specific formula is:

[0104]

[0105] Spatial feature extraction is limited to single-frame spatial attention, ignoring cross-frame dependencies, i.e., temporal series features. Therefore, temporal features are jointly extracted from video blocks at the same location in each frame of the sampled video. Similar to considering only the spatial dimension, during model training, the self-attention module involves F+1 query-key pairs, containing F sequences of image blocks at a specific location at different times and one separate data label block. When using the model, the self-attention module involves F query-key pairs, i.e., F sequences of image blocks at a specific location at different times.

[0106] When each self-attention module's attention head extracts temporal features, it first performs a dot product operation on the query matrix Q and the keyword matrix K to obtain the relevance between the target feature element and the other feature elements, i.e., the similarity weight:

[0107]

[0108] Next, features in the numerical matrix V (Value) are selected based on the calculated similarity weights, thus completing the spatial feature extraction process for each attention head of the self-attention module. The specific calculation is shown in the following formula:

[0109]

[0110] If the two features are extracted in parallel within the same module, the time complexity will increase to O(n^2). 2 The time complexity remains at the O(n) level if the operations are performed sequentially and alternately. Therefore, to ensure fast model execution and keep the computational load at a low level, this embodiment selects the following in the Transformer model: Figure 5As shown in the lower right corner, spatiotemporal features are extracted using a concatenated crossover method. Specifically, the spatial features extracted by all attention heads in the multi-head self-attention module are fused with the temporal features extracted by the previous network layer, then input into a multilayer perceptron and connected using residual connections to obtain spatial features; conversely, the temporal features extracted by all attention heads in the multi-head self-attention module are fused with the spatial features extracted by the previous network layer, then input into a multilayer perceptron and connected using residual connections to obtain temporal features.

[0111] Specifically, in the multi-head attention module, self-attention feature extraction is performed jointly by multiple heads. Therefore, it is necessary to integrate the computation results of all heads and combine them with residual shortcut connections to obtain intermediate results.

[0112] The integration formula for extracting spatial features is:

[0113]

[0114] in It is the input of the previous module. Since the two features are extracted in a cross-processing manner, the input of the previous layer is the extracted time feature.

[0115] Next, the output from the previous step is subjected to layer normalization, and then processed through a method such as... Figure 5 The MLP structure shown is used, and spatial features are obtained by combining residual shortcut connections. The specific formula is as follows:

[0116]

[0117] The integration formula for extracting time features is:

[0118]

[0119] in It is the input of the previous module. Since the two features are extracted in a cross-processing manner, the input of the previous layer is the extracted spatial features.

[0120] Then perform layer normalization on the output, and pass it through a process such as... Figure 5 The MLP structure shown is combined with residual shortcut joins to obtain the result, and the specific formula is as follows:

[0121]

[0122] Finally, the features output from the Transformer encoder module are input into the prediction output module, which consists of a multi-layer fully connected network, to obtain the pain level corresponding to each sampled video. During model training, the pain level obtained in each frame of the sampled video is compared with the label tensor corresponding to that frame to calculate the loss value of each frame. Then, the loss value of the sampled video is calculated based on the loss values ​​of all video frames, such as by squaring the loss values ​​of all video frames, summing them, and averaging them to obtain the loss value of the sampled video. The Transformer model is then trained and optimized based on the loss value of the sampled video.

[0123] Step S500: Obtain and output the pain analysis results based on the pain levels corresponding to all sampled videos.

[0124] Specifically, after obtaining the pain level corresponding to each sampled video, the pain levels of all sampled videos are summed and averaged to obtain the pain level corresponding to facial pain expressions, which is then output. Optionally, a weighted average can be calculated based on the weights of each sampled video to obtain the pain level corresponding to facial pain expressions; further processing can also be performed on the pain levels corresponding to facial pain expressions to obtain pain analysis results.

[0125] During model training, this embodiment divides the 200 video sequences provided by the UNBC dataset into training, testing, and validation sets in a 165:30:5 ratio. To verify the model's effectiveness, a 255-frame video sequence from the validation set is evaluated at 6 and 16 levels. Furthermore, to enable the model to represent changes in facial pain levels with finer granularity and continuity, the model's predictions are rounded to two decimal places, and then plotted as shown below. Figure 7 The pain level assessment curve shown.

[0126] Through experiments, after training and validation on publicly available datasets, the models were quantitatively evaluated based on the three model evaluation metrics mentioned above. For coarse-grained 6-level pain intensity monitoring, the accuracy (Acc) was 98.03%, the root mean square error (RMSE) was 0.16, the Pearson correlation coefficient (PCC) was 0.95, and the intra-class correlation (ICC) was 0.73. For fine-grained 16-level pain intensity monitoring, the accuracy (Acc) was 97.06%, the root mean square error (RMSE) was 0.56, the Pearson correlation coefficient (PCC) was 0.97, and the intra-class correlation (ICC) was 0.86.

[0127] The calculated results show that the accuracy drops slightly when the pain level is estimated with finer granularity. However, the Pearson coefficient and intraclass correlation of the 16-level pain level assessment are both higher than those of the coarse-grained 6-level pain assessment. Therefore, it is clear that the fusion of temporal features helps in the sequential estimation of pain level.

[0128] As described above, by using a sliding window method to sample long videos into multiple short videos, data utilization is improved; and a discrete cosine transform is performed on each video frame, and high-frequency features are enhanced through zigzag scanning to enhance detailed feature information; and feature extraction is directionally planned in the self-attention module of the Transformer model, automatically extracting the spatiotemporal features of each short video sequence and performing feature fusion in O(n) time complexity, which can fully extract pain information features from facial expression videos, thereby accurately assessing pain levels.

[0129] Exemplary System

[0130] like Figure 8 As shown, corresponding to the above-mentioned facial pain expression analysis method based on spatiotemporal information, this embodiment of the invention also provides a facial pain expression analysis system based on spatiotemporal information, the system comprising:

[0131] Data acquisition module 600 is used to acquire facial expression videos;

[0132] The sliding window module 610 is used to sample the facial expression video using a sliding window to obtain several sampled video segments.

[0133] The detail feature enhancement map module 620 is used to perform discrete cosine transform on each video frame of each sampled video segment to obtain the frequency domain matrix corresponding to each video frame; after enhancing the frequency domain matrix by sawtooth scanning, inverse discrete cosine transform is performed to obtain the detail feature enhancement map corresponding to each video frame.

[0134] The pain level analysis module 630 is used to combine all video frames and corresponding detail feature enhancement maps of each sampled video and input them into the Transformer model. In the Transformer model, temporal and spatial features are extracted in a serial cross-process to obtain the pain level corresponding to each sampled video.

[0135] Results module 640 is used to obtain and output pain analysis results based on the pain levels corresponding to all sampled videos.

[0136] Specifically, in this embodiment, the specific functions of each module of the above-mentioned spatiotemporal information-based facial pain expression analysis system can be referred to the corresponding description in the above-mentioned spatiotemporal information-based facial pain expression analysis method, and will not be repeated here.

[0137] Based on the above embodiments, the present invention also provides a smart terminal, the principle block diagram of which can be as follows: Figure 9 As shown. The aforementioned smart terminal includes a processor, memory, network interface, and display screen connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a spatiotemporal information-based facial pain expression analysis program. The internal memory provides an environment for the operation of the operating system and the spatiotemporal information-based facial pain expression analysis program stored in the non-volatile storage medium. The network interface of the smart terminal is used for communication with external terminals via a network connection. When the spatiotemporal information-based facial pain expression analysis program is executed by the processor, it implements the steps of any of the aforementioned spatiotemporal information-based facial pain expression analysis methods. The display screen of the smart terminal can be a liquid crystal display (LCD) or an e-ink display.

[0138] Those skilled in the art will understand that Figure 9 The block diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the smart terminal to which the present invention is applied. A specific smart terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0139] In one embodiment, a smart terminal is provided, the smart terminal including a memory, a processor, and a spatiotemporal information-based facial pain expression analysis program stored in the memory and executable on the processor, wherein the spatiotemporal information-based facial pain expression analysis program performs the following operation instructions when executed by the processor:

[0140] Get facial expression videos;

[0141] The facial expression video is sampled using a sliding window to obtain several sampled video segments;

[0142] For each video frame of each sampled video segment, a discrete cosine transform is performed to obtain the frequency domain matrix corresponding to each video frame; the frequency domain matrix is ​​enhanced by a sawtooth scan and then an inverse discrete cosine transform is performed to obtain the detail feature enhancement map corresponding to each video frame.

[0143] All video frames and corresponding detail feature enhancement maps of each sampled video are combined and input into the Transformer model. Temporal and spatial features are extracted in the Transformer model in a concatenated and cross-processed manner to obtain the pain level corresponding to each sampled video.

[0144] The pain analysis results are obtained and output based on the pain levels corresponding to all sampled videos.

[0145] Optionally, the step of sampling the facial expression video using a sliding window to obtain several sampled video segments includes:

[0146] The sliding window is initialized based on the preset start frame position, total number of sampling frames, and interval frame number of the first sampled video segment; the video frames of the facial expression video are sampled within the sliding window according to the interval frame number to obtain the first sampled video segment;

[0147] Set the next sampled video segment as the target video segment;

[0148] The sliding window is slid according to the movement step size;

[0149] When the preset total number of sampling frames and interval frame count of the target video segment are different from the preset total number of sampling frames and interval frame count of the previous sampled video segment, the sliding window is updated according to the preset total number of sampling frames and interval frame count of the target video segment.

[0150] The target video segment is obtained by sampling video frames of the facial expression video within a sliding window according to a preset interval frame number.

[0151] Return to set the next sampled video segment as the target video segment to obtain the next sampled video segment, and continue until all sampled video segments are obtained.

[0152] Optionally, the enhancement of the frequency domain matrix using a sawtooth scan includes:

[0153] A sawtooth scanning method is used, starting from the lower right corner of the frequency domain matrix, to sequentially obtain frequency domain feature information. If the current scanning position number is even, the frequency domain feature information of the previous position is used to replace the frequency domain feature information of the current scanning position.

[0154] Optionally, a discrete cosine transform is performed on the video frames to obtain the frequency domain matrix corresponding to the video frames, including:

[0155] Based on a preset image block size, the video frame is divided into several image blocks;

[0156] For each image block, the spatial domain image feature information is transformed into the frequency domain frequency feature information by using discrete cosine transform, thus obtaining the frequency domain frequency feature information corresponding to each image block.

[0157] Based on the position of the image block in the video frame, all frequency domain feature information is combined to obtain the frequency domain matrix.

[0158] Optionally, all video frames of the sampled video and their corresponding detail feature enhancement maps are combined and input into the Transformer model, including:

[0159] Each video frame of the sampled video and its corresponding detail feature enhancement map are combined and segmented according to a preset image block size to obtain an image block sequence;

[0160] The image block sequence is flattened, linearly projected, and has position markers added before being input into the Transformer encoder.

[0161] Optionally, the extraction of temporal and spatial features in the Transformer model using a concatenated crossover method includes:

[0162] The spatial features extracted by all attention heads in the multi-head self-attention module are fused with the temporal features extracted by the previous network layer and then input into the multilayer perceptron and residual connection are used to obtain the spatial features.

[0163] The temporal features extracted by all attention heads in the multi-head self-attention module are fused with the spatial features extracted by the previous network layer and then input into the multilayer perceptron and residual connection are used to obtain the temporal features.

[0164] Optionally, obtain the temporal or spatial features extracted by the attention heads in the multi-head self-attention module, including:

[0165] Perform a dot product operation on the query matrix and the keyword matrix to obtain similarity weights used to characterize the relevance between feature elements;

[0166] Based on the similarity weights and numerical matrices, the temporal or spatial features extracted by the attention head of the self-attention module are obtained.

[0167] This invention also provides a computer-readable storage medium storing a facial pain expression analysis program based on spatiotemporal information. When executed by a processor, the spatiotemporal information-based facial pain expression analysis program implements the steps of any spatiotemporal information-based facial pain expression analysis method provided in this invention.

[0168] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0169] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0170] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0171] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0172] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of the above modules or units is merely a logical functional division, and in actual implementation, it can be divided in other ways. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0173] If the integrated modules / units described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction.

[0174] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not mean that the essence of the corresponding technical solutions deviates from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A facial pain expression analysis method based on spatiotemporal information, characterized by, The method includes: Get facial expression videos; The facial expression video is sampled using a sliding window to obtain several sampled video segments; For each video frame of each sampled video segment, a discrete cosine transform is performed to obtain the frequency domain matrix corresponding to each video frame; the frequency domain matrix is ​​enhanced by a sawtooth scan and then an inverse discrete cosine transform is performed to obtain the detail feature enhancement map corresponding to each video frame. All video frames and corresponding detail feature enhancement maps of each sampled video are combined and input into the Transformer model. Temporal and spatial features are extracted in the Transformer model in a concatenated and cross-processed manner to obtain the pain level corresponding to each sampled video. The pain analysis results are obtained and output based on the pain levels corresponding to all sampled videos; The enhancement of the frequency domain matrix by sawtooth scanning includes: using a sawtooth scanning method, scanning from the lower right corner of the frequency domain matrix, sequentially acquiring frequency domain frequency feature information; if the current scanning position number is even, replacing the frequency domain frequency feature information of the current scanning position with the frequency domain frequency feature information of the previous position. The Transformer model includes a linear mapping layer, which maps the dimensions of a one-dimensional tensor to a specified embedding dimension and uses the embedding dimension as the embedding of an image patch representation sequence.

2. The facial pain expression analysis method based on spatio-temporal information according to claim 1, characterized in that, The process of sampling the facial expression video using a sliding window yields several sampled video segments, including: The sliding window is initialized based on the preset start frame position, total number of sampling frames, and interval frame number of the first sampled video segment; the video frames of the facial expression video are sampled within the sliding window according to the interval frame number to obtain the first sampled video segment; Set the next sampled video segment as the target video segment; The sliding window is slid according to the movement step size; When the preset total number of sampling frames and interval frame count of the target video segment are different from the preset total number of sampling frames and interval frame count of the previous sampled video segment, the sliding window is updated according to the preset total number of sampling frames and interval frame count of the target video segment. The target video segment is obtained by sampling video frames of the facial expression video within a sliding window according to a preset interval frame number. Return to set the next sampled video segment as the target video segment to obtain the next sampled video segment, until all sampled video segments are obtained.

3. The facial pain expression analysis method based on spatio-temporal information according to claim 1, characterized in that, Performing a discrete cosine transform on the video frames yields the corresponding frequency domain matrix, including: Based on a preset image block size, the video frame is divided into several image blocks; For each image block, the spatial domain image feature information is transformed into the frequency domain frequency feature information by applying discrete cosine transform, thus obtaining the frequency domain frequency feature information corresponding to each image block. Based on the position of the image block in the video frame, all frequency domain feature information is combined to obtain the frequency domain matrix.

4. The facial pain expression analysis method based on spatiotemporal information as described in claim 1, characterized in that, The sampled video frames and their corresponding detail feature enhancement maps are combined and input into the Transformer model, including: Each video frame of the sampled video and its corresponding detail feature enhancement map are combined and segmented according to a preset image block size to obtain an image block sequence; The image block sequence is flattened, linearly projected, and has position markers added before being input into the Transformer encoder.

5. The facial pain expression analysis method based on spatiotemporal information as described in claim 1, characterized in that, The extraction of temporal and spatial features in the Transformer model using a concatenated and interleaved approach includes: The spatial features extracted by all attention heads in the multi-head self-attention module are fused with the temporal features extracted by the previous network layer and then input into the multilayer perceptron and residual connection are used to obtain the spatial features. The temporal features extracted by all attention heads in the multi-head self-attention module are fused with the spatial features extracted by the previous network layer and then input into the multilayer perceptron and residual connection are used to obtain the temporal features.

6. The facial pain expression analysis method based on spatiotemporal information as described in claim 5, characterized in that, Obtain the temporal or spatial features extracted by the attention heads in the multi-head self-attention module, including: Perform a dot product operation on the query matrix and the keyword matrix to obtain similarity weights used to characterize the relevance between feature elements; Based on the similarity weights and numerical matrices, the temporal or spatial features extracted by the attention head of the self-attention module are obtained.

7. A facial pain expression analysis system based on spatiotemporal information, characterized in that, The system includes: The data acquisition module is used to acquire facial expression videos; A sliding window module is used to sample the facial expression video using a sliding window to obtain several sampled video segments; The detail feature enhancement map module is used to perform discrete cosine transform on each video frame of each sampled video segment to obtain the frequency domain matrix corresponding to each video frame; after enhancing the frequency domain matrix by sawtooth scanning, inverse discrete cosine transform is performed to obtain the detail feature enhancement map corresponding to each video frame. The pain level analysis module is used to combine all video frames and corresponding detail feature enhancement maps of each sampled video and input them into the Transformer model. In the Transformer model, temporal and spatial features are extracted in a concatenated and cross-processed manner to obtain the pain level corresponding to each sampled video. The results module is used to obtain and output pain analysis results based on the pain levels corresponding to all sampled videos. The detailed feature enhancement map module is also used to: scan from the lower right corner of the frequency domain matrix using a sawtooth scanning method, and sequentially obtain frequency domain frequency feature information. If the current scanning position number is even, the frequency domain frequency feature information of the current scanning position is replaced with the frequency domain frequency feature information of the previous position. The Transformer model includes a linear mapping layer, which maps the dimensions of a one-dimensional tensor to a specified embedding dimension and uses the embedding dimension as the embedding of an image patch representation sequence.

8. A smart terminal, characterized in that, The smart terminal includes a memory, a processor, and a spatiotemporal information-based facial pain expression analysis program stored in the memory and executable on the processor. When the spatiotemporal information-based facial pain expression analysis program is executed by the processor, it implements the steps of the spatiotemporal information-based facial pain expression analysis method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a facial pain expression analysis program based on spatiotemporal information, which, when executed by a processor, implements the steps of the facial pain expression analysis method based on spatiotemporal information as described in any one of claims 1-6.

Images