Pressure detection method and device based on deep learning

Abstract

The invention provides a pressure detection method and device based on deep learning, and the method comprises the steps: obtaining a to-be-detected video, and extracting a face frame of a target user in the to-be-detected video; inputting the face frame of the target user into a pre-trained pressure detection model to obtain a pressure detection result output by the pressure detection model; wherein the pressure detection model is obtained by training based on a face frame sample, and a sample pressure detection result, a sample emotion guide description feature and a sample group classification result corresponding to the face frame sample; the face frame sample is extracted from a video sample; the emotion-oriented description features are obtained by superposing representation features extracted from the face frame sample through a multi-layer attention mechanism; the group classification result is obtained by extracting the face frame sample through a group attention mechanism. The technical problem that in the prior art, a non-contact detection means is poor in detection accuracy is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a pressure detection method and device based on deep learning. Background Art

[0002] With the accelerated pace of life and the intensified social competition, people are under unprecedented psychological pressure. Before the adverse consequences caused by excessive stress occur, it is also crucial to detect stress in time through stress detection methods.

[0003] Traditional stress detection methods rely on psychological questionnaires or professional psychological counseling, which are usually only suitable for smaller groups of people and cannot effectively respond to large numbers of detection needs. In addition, since the results of the questionnaire survey depend on the consultant's answers to relevant questions, the measurement of stress levels is relatively subjective, especially when the consultant selectively expresses his or her own mental state, there will be large deviations in the detection results.

[0004] In order to overcome the limitations of questionnaire surveys, some existing technologies use dedicated sensors or wearable devices (such as mobile phones with embedded sensors) to detect psychological stress by sensing human physiological signals (such as heart rate variability, electrocardiogram, electroencephalogram, electromyogram, blood pressure, skin current response, etc.). However, although this detection method has high accuracy, it requires users to wear wearable devices or dedicated sensors, making it difficult to achieve non-contact measurement, and the equipment cost is high.

[0005] In view of this, providing a pressure detection method and device based on deep learning, in order to perform more accurate contactless detection of the target user's psychological stress level, realize contactless detection of pressure, and improve the accuracy of pressure detection, has become an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0006] The present invention provides a pressure detection method and device based on deep learning, which are used to solve the technical problem of poor detection accuracy of contactless detection means in the prior art, so as to perform more accurate contactless detection of the psychological stress level of the target user, realize contactless detection of pressure, and improve the accuracy of pressure detection.

[0007] The present invention provides a pressure detection method based on deep learning, comprising:

[0008] Obtain a video to be detected, and extract a facial frame of a target user in the video to be detected;

[0009] Inputting the target user's facial frame into a pre-trained pressure detection model to obtain a pressure detection result output by the pressure detection model;

[0010] The stress detection model is trained based on facial frame samples, sample stress detection results corresponding to the facial frame samples, sample emotion-oriented description features, and sample group classification results;

[0011] The facial frame samples are extracted from video samples; the emotion-oriented description features are obtained by superimposing the representation features extracted from the facial frame samples through a multi-layer attention mechanism; and the group classification results are obtained by extracting the facial frame samples through a group attention mechanism.

[0012] According to the deep learning-based pressure detection method provided by the present invention, extracting facial frame samples from video samples specifically includes:

[0013] Acquire video samples, and extract frames that meet preset image quality requirements from the video samples;

[0014] Based on the preset emotion classification, the probability of the emotion classification detected in each frame is calculated to obtain a sequence of emotion classification probabilities;

[0015] Extracting a preset number of emotion classification probabilities from the sequence of emotion classification probabilities;

[0016] The frames corresponding to the preset number of emotion classification probabilities obtained are used as the facial frame samples.

[0017] According to the stress detection method based on deep learning provided by the present invention, the multi-layer attention mechanism at least includes an emotion expression attention mechanism, a continuous negative emotion attention mechanism and a video frame self-attention mechanism with emotion information;

[0018] The facial frame samples are superimposed with the representation features extracted by the multi-layer attention mechanism to obtain the emotion-oriented description features, which specifically include:

[0019] Based on the emotion expression attention mechanism, feature representation is extracted from the facial frame sample to obtain a frame representation result containing emotion information.

[0020] Based on the negative emotion cycle and continuous negative emotion attention mechanism, the result is represented in the frame containing emotional information. Extract the video representation results containing enhanced negative emotion frames

[0021] Based on the self-attention mechanism of video frames with emotional information, the video representation results containing negative emotional frames are enhanced. Embed position information in the video frame to obtain the self-attention result with emotion

[0022] According to the stress detection method based on deep learning provided by the present invention, based on the emotion expression attention mechanism, feature representation is extracted from the facial frame sample to obtain a frame representation result containing emotion information. Specifically include:

[0023] Construct facial frame sample X F And the sample pressure test result X C A vector of

[0024] Normalize each element in the vector A to [0, 1], and use the normalized result to represent the attention weight relative to the target emotion category;

[0025] According to the attention weight relative to the target emotion category and the vector, the classification emotion attention X is obtained. A ;

[0026] Classify the sentiment attention X A Sample X with face frames F Align each frame in the image, and calculate the frame representation containing emotional information through a linear function

[0027] According to the stress detection method based on deep learning provided by the present invention, based on the negative emotion cycle and the continuous negative emotion attention mechanism, the result is represented in the frame containing emotional information. Extract the video representation results containing enhanced negative emotion frames Specifically include:

[0028] Extract the frame representation result containing emotional information Negative emotion segments p of consecutive facial frames in ;

[0029] According to the duration of the negative emotion segment p and its arrangement position in the segment, the frame representation result of each frame in the negative emotion segment is weighted enhanced to obtain a video representation result of the facial frame sample containing negative emotion frame enhancement.

[0030] According to the stress detection method based on deep learning provided by the present invention, based on the self-attention mechanism of video frames with emotional information, the video representation result containing the enhanced negative emotion frame is Embed position information in the video frame to obtain the self-attention result with emotion Specifically include:

[0031] Based on the preset frame position information parameters, the video representation result containing the negative emotion frame enhancement is Each frame in the representation is embedded in position to obtain the embedding result.

[0032] Based on the self-attention mechanism of video frames with emotional information, the embedding representation results Perform frame self-attention calculation to obtain the video frame self-attention result with emotion

[0033] The present invention also provides a pressure detection device based on deep learning, comprising:

[0034] A video acquisition unit, used to acquire a video to be detected and extract a facial frame of a target user in the video to be detected;

[0035] A result generating unit, used for inputting the facial frame of the target user into a pre-trained pressure detection model to obtain a pressure detection result output by the pressure detection model;

[0036] The stress detection model is trained based on facial frame samples, sample stress detection results corresponding to the facial frame samples, sample emotion-oriented description features, and sample group classification results;

[0037] The facial frame samples are extracted from video samples; the emotion-oriented description features are obtained by superimposing the representation features extracted from the facial frame samples through a multi-layer attention mechanism; and the group classification results are obtained by extracting the facial frame samples through a group attention mechanism.

[0038] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the pressure detection method based on deep learning as described above is implemented.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the pressure detection method based on deep learning as described above.

[0040] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the pressure detection method based on deep learning as described above.

[0041] The present invention provides a stress detection method based on deep learning. The method comprises the following steps: acquiring a video to be detected and extracting a facial frame of a target user in the video to be detected; inputting the facial frame of the target user into a pre-trained stress detection model to obtain a stress detection result output by the stress detection model; wherein the stress detection model is trained based on facial frame samples, and sample stress detection results, sample emotion-oriented description features and sample group classification results corresponding to the facial frame samples; the facial frame samples are extracted from video samples; the emotion-oriented description features are obtained by superimposing representation features extracted from the facial frame samples through a multi-layer attention mechanism; and the group classification results are obtained by extracting the facial frame samples through a group attention mechanism.

[0042] In this way, the stress detection method provided by the present invention can realize a personalized stress detection method based on ubiquitous video and deep learning. It adopts a three-layer attention mechanism, pays attention to the user's specific emotional facial expressions, the duration of continuous negative emotions, and the conversion and self-association of emotional frames in the video, and can accurately identify the stress level, thereby solving the technical problem of poor detection accuracy of contactless detection means in the prior art, and can perform relatively accurate contactless detection of the target user's psychological stress level, thereby realizing contactless detection of stress and improving the accuracy of stress detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 It is one of the flow charts of the pressure detection method based on deep learning provided by the present invention;

[0045] Figure 2 This is the second flow chart of the pressure detection method based on deep learning provided by the present invention;

[0046] Figure 3 This is the third flow chart of the pressure detection method based on deep learning provided by the present invention;

[0047] Figure 4 This is the fourth flow chart of the pressure detection method based on deep learning provided by the present invention;

[0048] Figure 5 This is the fifth flow chart of the pressure detection method based on deep learning provided by the present invention;

[0049] Figure 6 This is the sixth flow chart of the pressure detection method based on deep learning provided by the present invention;

[0050] Figure 7 This is the seventh flow chart of the pressure detection method based on deep learning provided by the present invention;

[0051] Figure 8 This is the eighth flow chart of the pressure detection method based on deep learning provided by the present invention;

[0052] Fig. 9 This is the ninth flow chart of the pressure detection method based on deep learning provided by the present invention;

[0053] Fig.10 It is a structural schematic diagram of a pressure detection device based on deep learning provided by the present invention;

[0054] Fig.11 It is a structural schematic diagram of the electronic device provided by the present invention.

[0055] Reference numerals:

[0056] 1001: video acquisition unit; 1002: result generation unit. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0059] Combine the following Figure 1-Figure 9 The deep learning-based pressure detection method of the present invention is described.

[0060] In a specific embodiment, Figure 1 As shown, the pressure detection method based on deep learning provided by the present invention includes the following steps:

[0061] S110: Obtain the video to be detected, and extract the facial frame of the target user in the video to be detected. In a specific usage scenario, the video to be detected may be a surveillance video, such as a surveillance video in a target area during a target period of time; with the widespread use of contactless cameras in public spaces and specific places, it is possible to detect psychological stress through human facial expressions and movements captured by video; compared with traditional stress gauges and specific sensors, surveillance cameras have advantages such as convenience, low cost, and non-invasiveness.

[0062] S120: Input the facial frame of the target user into a pre-trained stress detection model to obtain the stress detection result output by the stress detection model; wherein the stress detection model is trained based on facial frame samples, and sample stress detection results, sample emotion-oriented description features and sample group classification results corresponding to the facial frame samples; the facial frame samples are extracted from video samples; the emotion-oriented description features are obtained by superimposing the representation features extracted from the facial frame samples through a multi-layer attention mechanism; and the group classification results are obtained by extracting the facial frame samples through a group attention mechanism.

[0063] In addition to the multi-layer attention mechanism, in order to improve the prediction accuracy, the model provided in this embodiment also integrates the group classification results. In other words, it learns and mines the user's personalized features from the video, such as the user's gender, age, and extroversion characteristics, and establishes user groups to further enhance the video-based stress detection performance.

[0064] When learning and building groups based on user personalized features, we build a group user graph based on RDF based on the user demographic and personality characteristics automatically inferred from the video. Since the user's gender, age, and personality characteristics will affect their facial expressions, in addition to the video content, the user's gender, age, and personality characteristics are inferred from the video. Specifically, through the group attention mechanism, the stress recognition ability is enhanced, including the following steps:

[0065] S1: Use RDF (Resource Description Framework) statements to describe user information and construct an RDF-based user graph;

[0066] S2: Detect and identify user age, gender, and extroversion based on the video;

[0067] S3: Group attention mechanism based on user personality traits: In addition to the user's individual video content, it also pays attention to other users who share the same personality traits as the user, and enriches the user representation with group information.

[0068] Since human facial expressions contain rich information that can reflect the stress state of the human body, and there is a close connection between stress and negative emotions such as anger, disgust, fear, sadness, etc. Therefore, in theory, the stress detection task can be converted into instantaneous negative emotional state detection, and then the detection results of each frame of video can be summarized to determine the stress level according to the percentage of frames showing negative emotions. Compared with traditional technologies that only consider the negative emotions of users, in this embodiment, all positive, neutral and negative emotions of users are combined, and stress assessment is performed by analyzing the dynamic changes of users' emotions instead of simply aggregating and counting negative emotions. Particular attention is paid to the duration of users' negative emotions and the facial expression characteristics of users associated with various emotions. For example, these emotions can include eight types of emotions such as happiness, surprise, fear, sadness, anger, disgust, contempt and neutrality.

[0069] In actual usage scenarios, the video source may be a surveillance video. Since there is a lot of noise in the surveillance video, in order to ensure the accuracy of the results, the surveillance video data needs to be preprocessed to provide data support for the subsequent generation of video representations.

[0070] In some embodiments, according to the deep learning-based pressure detection method provided by the present invention, facial frame samples are extracted from video samples, such as Figure 2 As shown, the specific steps include:

[0071] S210: Obtain video samples, and extract frames that meet preset image quality requirements from the video samples; that is, obtain a high-quality user facial frame sequence from the input surveillance video.

[0072] S220: Based on the preset emotion classification, the probability of the emotion classification detected in each frame is calculated to obtain a sequence of emotion classification probabilities; the user's facial frame is encoded, and different classified emotions are identified from each facial frame.

[0073] S230: extracting a preset number of emotion classification probabilities from the sequence of emotion classification probabilities;

[0074] S240: The frames corresponding to the preset number of emotion classification probabilities are used as the facial frame samples. Specifically, for each type of emotion (such as happiness, surprise, fear, sadness, anger, disgust, contempt and neutrality, etc.), a most characteristic facial frame is selected from the user's video, which can also be understood as the facial frame corresponding to the emotion classification with the highest weight.

[0075] In a specific usage scenario, the user's surveillance video is preprocessed and an internal video representation is generated for further processing. The face detection technology MTCNN is used to capture the face area in each frame of the original video, and the face recognition system Face Net is used to recognize the user's face; the user's low-quality facial frames (such as face occlusion, poor lighting or blur, etc.) are filtered out through the SER-FIQ model. F user's facial frames are obtained from the input surveillance video, denoted as X F Theoretically, the high correlation between stress and emotions, as well as the excellent facial emotion recognition performance, motivates us to build stress detection models on high-level emotion features, rather than starting from low-level features such as facial landmarks and facial action units, in order to improve model performance.

[0076] When encoding each facial frame, models such as ResNet, Self-Cure Network (SCN), or Vision Transformer (ViT) can be utilized and 8 types of emotions can be identified from each facial frame, which can include happiness, surprise, fear, sadness, anger, disgust, contempt, and neutral.

[0077] Specifically, firstly, the emotion-oriented facial frame is represented as:

[0078] X F =(x1,x2,…,x F )

[0079] in (i=1,2,…,F).

[0080] Assume from X F The sequence of probabilities of the categorical emotions detected in each facial frame is:

[0081] X E =(e1,e2,…,e F )

[0082] Among them, e i =(e i,1 ,e i,2 ,…,e i,C ), e i,j ∈[0,1]; i=1,2,…,F, j=1,2…,C.

[0083] Furthermore, from X F The eight most emotional facial frames are selected:

[0084]

[0085] in, Based on X E In XF The facial frames corresponding to the i-th emotion are selected from the filter, and the selection rules are as follows:

[0086]

[0087] In this way, the selected 8 most emotional facial frame representations will be used as references to provide data support for distinguishing frame-based emotions in the subsequent triple attention mechanism.

[0088] The method provided by the present invention adopts a three-layer attention mechanism to focus on the user's specific emotional facial expressions, the duration of continuous negative emotions, and the conversion and self-association of emotional frames in the video. That is to say, when performing emotion-oriented video learning, the user's facial frames can be mined for information based on the deep learning framework by focusing on the user's emotional expression characteristics, the duration of continuous negative emotions, and emotional dynamics through three attention mechanisms, thereby improving accuracy. Among them, the three attention mechanisms specifically include an emotional expression attention mechanism, a continuous negative emotion attention mechanism, and a video frame self-attention mechanism with emotions; the emotional expression attention mechanism is used to focus on the emotional expression of each frame; the continuous negative emotion attention mechanism is used to enhance the weight of the user's continuous negative emotions based on the concept of negative emotion cycle; it should be understood that the negative emotion period is a segment containing a series of continuous frames, which meets the following conditions: first, each frame shows negative emotions, such as fear, sadness, anger, disgust and / or contempt, etc.; second, the time distance between any two continuous frames is less than a certain threshold, such as a 30-minute time interval or a 15-minute time interval in monitoring. The video frame self-attention mechanism with emotions is used to implant an emotional frame self-attention layer to model the frame sequence as a video representation.

[0089] More specifically, in some embodiments, according to the deep learning-based stress detection method provided by the present invention, the multi-layer attention mechanism includes at least an emotion expression attention mechanism, a continuous negative emotion attention mechanism, and a video frame self-attention mechanism with emotion information;

[0090] The facial frame samples are superimposed with the representation features extracted by the multi-layer attention mechanism to obtain the emotion-oriented description features, such as Figure 3 As shown, the specific steps include:

[0091] S310: Based on the emotion expression attention mechanism, extract feature representation from the facial frame sample to obtain a frame representation result containing emotion information

[0092] S320: Based on the negative emotion cycle and the continuous negative emotion attention mechanism, the result of representing the frame containing the emotion information is Extract the video representation results containing enhanced negative emotion frames

[0093] S330: Based on the self-attention mechanism of the video frame with emotional information, the video representation result containing the enhanced negative emotion frame is Embed position information in the video frame to obtain the self-attention result with emotion

[0094] In step S310, according to the stress detection method based on deep learning provided by the present invention, based on the emotion expression attention mechanism, feature representation is extracted from the facial frame sample to obtain a frame representation result containing emotion information. like Figure 4 As shown, the specific steps include:

[0095] S410: Constructing facial frame sample X F And the sample pressure test result X C A vector of

[0096] S420: normalizing each element in the vector A to [0, 1], and using the normalized result to represent the attention weight relative to the target emotion category;

[0097] S430: Obtain the classification emotion attention X according to the attention weight relative to the target emotion category and the vector A ;

[0098] S440: Classify emotional attention X A Sample X with face frames F Align each frame in the image, and calculate the frame representation containing emotional information through a linear function

[0099] Specifically, the emotional expression attention mechanism is used to obtain the frame representation result containing emotional information In the process, Figure 5 As shown, taking eight commonly used emotion types as an example, For each frame in the The closeness of is calculated as an 8-dimensional vector A. After the Softmax function, Each element in is normalized to [0,1] and is used to represent the attention weight relative to a specific sentiment category.

[0100] Set [Q, K, V] = [X F ×W q ,X C ×W k ,X C ×W v ]; among them, W q,W k ,W v is a trainable parameter used to transform X F and X C Projection maps to the same dimension for multiplication, d k is the size parameter of the scale.

[0101]

[0102] X A =A×V

[0103] Classify the sentiment attention X A With X F Each frame in is aligned and the dimension is reduced by a linear function to reduce the computational complexity:

[0104]

[0105]

[0106] in, is a trainable parameter, and the frame containing emotional information represents the result It will provide a reference dimension for further emotion identification.

[0107] In step S320, according to the stress detection method based on deep learning provided by the present invention, based on the negative emotion cycle and the continuous negative emotion attention mechanism, the result of the frame representation containing the emotional information is Extract the video representation results containing enhanced negative emotion frames Specifically include:

[0108] Extract the frame representation result containing emotional information Negative emotion segments p of consecutive facial frames in ;

[0109] According to the duration of the negative emotion segment p and its arrangement position in the segment, the frame representation result of each frame in the negative emotion segment is weighted enhanced to obtain a video representation result of the facial frame sample containing negative emotion frame enhancement. In principle, based on the concept of negative emotion cycle, the weight of the user's continuous negative emotions is enhanced. A negative emotion period is a segment containing a series of consecutive frames, satisfying condition (a) each frame shows negative emotions (fear, sadness, anger, disgust and / or contempt), and condition (b) the time distance between any two consecutive frames is less than a certain threshold (for example, 30 minutes in monitoring). This is because some low-quality frames are discarded during data preprocessing, and here, the time distance between each two consecutive frames may be uneven. Condition (b) requires that any two consecutive frames should not be too far apart. In addition, frames with positive or neutral emotions will not appear in any negative emotion segment.

[0110] Based on this, by using the continuous negative emotion attention mechanism, the video representation result containing the enhanced negative emotion frame of the facial frame sample is obtained. In the process, Figure 6 As shown, represents a negative emotion segment containing k consecutive facial frames, where the frames containing emotional information are represented as The fragment length |p|=k(1 <k≤F)。

[0111] The representation of each frame in the segment is transformed into Enhanced to

[0112]

[0113]

[0114] Λ(p)=|p|·W2+b2

[0115] The function Λ(p) returns the added frame-by-frame weight according to the duration of the time period p, and the function λ(p,i) returns the added frame-by-frame weight of the i-th frame in p according to Λ(p). and b2∈[0,1] are all trainable parameters.

[0116] From the above formula, we can see that the longer the negative emotion cycle is, the greater the added cycle weight is, and the later the frame is in the negative emotion cycle, the greater the added frame weight is. F For any other non-negative emotion frame in , since it does not exist in any negative emotion segment, its frame representation is consistent with the original frame representation containing emotion information. Therefore, we can use to represent the video representation containing enhanced negative emotion frames.

[0117] In step S330, according to the stress detection method based on deep learning provided by the present invention, based on the self-attention mechanism of the video frame with emotional information, the video representation result containing the enhanced negative emotion frame is Embed position information in the video frame to obtain the self-attention result with emotion Specifically include:

[0118] Based on the preset frame position information parameters, the video representation result containing the negative emotion frame enhancement is Each frame in the representation is embedded in position to obtain the embedding result.

[0119] Based on the self-attention mechanism of video frames with emotional information, the embedding representation results Perform frame self-attention calculation to obtain the video frame self-attention result with emotion

[0120] In a specific usage scenario, the video frame self-attention mechanism with emotional information is used to obtain the video frame self-attention result with emotion. In the process of video frame self-attention mechanism, an emotional frame self-attention layer is implanted to model the frame sequence as a video representation. Each frame in represents a position embedding operation, such as Figure 7 shown.

[0121]

[0122] in, is the frame position information parameter, Add a trainable parameter matrix In the frame sequence header as the category CLS_TOKEN, at this time

[0123] set up in It is used to map Perform frame self-attention calculation and calculate the frame self-attention weight matrix Then, the self-attention result of the video frame with emotion is obtained according to the following formula:

[0124]

[0125] in, By q i ′ ∈Q ′ and k j ′ ∈K ′The calculated video expression after the sub-attention mechanism is

[0126]

[0127] After normalization, Figure 8 As shown, the video representation is learned using a deep learning network It is then sent to an MLP block, which includes two fully connected network layers (for feature projection), a GELU nonlinear activation layer, and a dropout layer (for preventing overfitting); in steps S310-S330, M stacking (where M = 16) is performed to extract As a video representation, the video is expressed as Will be integrated with other attributes of the user base.

[0128] Furthermore, in the process of personalized feature learning and group attention, since users’ personalities and personality traits affect their facial expressions, in addition to the video content, efforts are made to infer other attribute values ​​of users from the video, such as gender, age, personality traits, etc., and incorporate them to enhance stress recognition capabilities.

[0129] First, a user graph based on RDF is constructed; Figure 8 As shown, Figure 8 This is an example of an RDF-based user graph, where straight edges represent "inclusion" or "weight" relationships, and curved edges represent "value" relationships. To avoid crossing lines, the graph only plots the information corresponding to user 1. The user graph contains four attributes (video, gender, age, and personality). The value of the attribute "video" is the head output of the video representation after being processed by the attention mechanism. For example, the value of the attribute "gender" is "male" or "female", and the value of the attribute "age" is "young", "middle-aged" or "old". In the Big Five personality trait classification (extraversion, agreeableness, conscientiousness, neuroticism, and openness), due to the feasibility of video inference and the potential impact on stress detection, only the "extraversion" dimension of the attribute "personality" is considered in this embodiment. The weights of the three attributes (gender, age, and personality) are applied to all users to indicate their importance in the stress detection task, where the sum of the values ​​of each attribute for each user is 1.0.

[0130] Then, the user's age and gender information is detected and recognized based on the video. Specifically, in many practical applications, human gender and age have been regarded as two important biometric features. Estimating age and gender from the facial expressions of people shown in photos or videos has been widely studied and has achieved fruitful results. Inspired by the powerful global relationship modeling ability of Vision Transformer (ViT) in computer vision tasks, a deep learning architecture based on PS-ViT and TimeSformer is constructed. The architecture uses PS-ViT to divide each frame of video into a series of significant patches, and then calculates the patch-level attention mechanism through TimeSformer, which includes self-attention of all patches in the same frame, and attention of patches at the same position in different frames. The model is trained for age and gender estimation respectively.

[0131] When detecting the personality traits of users based on videos, the user's personality traits (extraversion) are estimated based on the aforementioned PS-ViT and patch-level attention. Considering that personality estimation requires mining more detailed information than age and gender estimation, using models such as TNT (Transformer in Transformer), for each significant patch, we evenly divide it into several smaller sub-patches and calculate the attention of each sub-patch together with other local sub-patches, clustering each significant patch with its locally participating sub-patches to enhance the representation ability. The estimated probabilities of extroversion, gender, and age are edge weights between the corresponding values ​​in users and user groups.

[0132] When using the group attention mechanism, in addition to the user's personal video content, attention is also paid to other users with the same attribute values ​​as the user (for example, "female", "young", etc.), and the user representation is enriched with group information. To this end, user groups are defined by specific attribute values ​​"gender", "age" or "personality". For example, all "male" users form one group, and all "young" users form another group. In the study, a total of 6 groups were set up (corresponding to "male", "female", "young", "middle-aged", "old" and "extrovert"), which can aggregate the information of group users and define them as representations of users.

[0133] Specifically, let R(u) be the representation of user u, and its initial value is the multi-head video representation

[0134] Let R(g) be the representation of group g. After semantic encoding, its initial attribute value is Project both into the same feature space to be consistent with the video content representation;

[0135] The initial representation of user u and group g is: in is a trainable parameter,

[0136] For each pair of user u and group g, iteratively compute and aggregate group information into user representation:

[0137]

[0138]

[0139]

[0140]

[0141] Among them, || represents the Concat splicing operation, is a trainable parameter, is the aggregate information of the group to which user u belongs and the aggregate information of users in the same group g. USet(g) represents group g. * The user set and GSet(u) in the above table represent the set of groups to which user u belongs. * ,u) and β(u * ,g) are respectively for user u * The attention score of group g and user u in group g is calculated as follows:

[0142]

[0143]

[0144] where w e (u * ,g) is user u * The edge weight between group g, w p (g) is the weight of group g (equal to the weight of the corresponding attribute node in pressure detection).

[0145] like Fig. 9 As shown in the figure, the GAT-based graph neural network is used to perform group attention calculation. After (l+1) propagations, the obtained group user expression R (l+1) (u) is input to the fully connected layer, and then the Softmax activation function is used to obtain the probabilities of different stress levels, corresponding to no stress, low stress, and high stress, respectively. The item with the highest probability is the detected stress level:

[0146] [y1,…,y n ]=Softmax(R (l+1) (u)×W s )

[0147] in And the cross entropy loss function is used to adjust the model weights during the entire model training process.

[0148] In the above specific implementation manner, the deep learning-based stress detection method provided by the present invention obtains the video to be detected, extracts the facial frame of the target user in the video to be detected; inputs the facial frame of the target user into a pre-trained stress detection model to obtain the stress detection result output by the stress detection model; wherein the stress detection model is trained based on facial frame samples, and sample stress detection results, sample emotion-oriented description features and sample group classification results corresponding to the facial frame samples; the facial frame samples are extracted from video samples; the emotion-oriented description features are obtained by superimposing the representation features extracted from the facial frame samples through a multi-layer attention mechanism; the group classification results are obtained by extracting the facial frame samples through a group attention mechanism.

[0149] In this way, the stress detection method provided by the present invention can realize a personalized stress detection method based on ubiquitous video and deep learning. It adopts a three-layer attention mechanism, pays attention to the user's specific emotional facial expressions, the duration of continuous negative emotions, and the conversion and self-association of emotional frames in the video, and can accurately identify the stress level, thereby solving the technical problem of poor detection accuracy of contactless detection means in the prior art, and can perform relatively accurate contactless detection of the target user's psychological stress level, thereby realizing contactless detection of stress and improving the accuracy of stress detection.

[0150] In addition to the above method, the present invention also provides a pressure detection device based on deep learning, such as Fig.10 As shown, the device comprises:

[0151] The video acquisition unit 1001 is used to acquire a video to be detected and extract a facial frame of a target user in the video to be detected;

[0152] A result generating unit 1002 is used to input the face frame of the target user into a pre-trained pressure detection model to obtain a pressure detection result output by the pressure detection model;

[0153] The stress detection model is trained based on facial frame samples, sample stress detection results corresponding to the facial frame samples, sample emotion-oriented description features, and sample group classification results;

[0154] The facial frame samples are extracted from video samples; the emotion-oriented description features are obtained by superimposing the representation features extracted from the facial frame samples through a multi-layer attention mechanism; and the group classification results are obtained by extracting the facial frame samples through a group attention mechanism.

[0155] According to the deep learning-based stress detection device provided by the present invention, the multi-layer attention mechanism at least includes an emotion expression attention mechanism, a continuous negative emotion attention mechanism, and a video frame self-attention mechanism with emotion information;

[0156] The facial frame samples are superimposed with the representation features extracted by the multi-layer attention mechanism to obtain the emotion-oriented description features, which specifically include:

[0157] Based on the emotion expression attention mechanism, feature representation is extracted from the facial frame sample to obtain a frame representation result containing emotion information.

[0158] Based on the negative emotion cycle and continuous negative emotion attention mechanism, the result is represented in the frame containing emotional information. Extract the video representation results containing enhanced negative emotion frames

[0159] Based on the self-attention mechanism of video frames with emotional information, the video representation results containing negative emotional frames are enhanced. Embed position information in the video frame to obtain the self-attention result with emotion

[0160] According to the deep learning-based pressure detection device provided by the present invention, based on the emotion expression attention mechanism, feature representation is extracted from the facial frame sample to obtain a frame representation result containing emotion information. Specifically include:

[0161] Construct facial frame sample X F And the sample pressure test result X C A vector of

[0162] Normalize each element in the vector A to [0, 1], and use the normalized result to represent the attention weight relative to the target emotion category;

[0163] According to the attention weight relative to the target emotion category and the vector, the classification emotion attention X is obtained. A ;

[0164] Classify the sentiment attention X A Sample X with face frames FAlign each frame in the image, and calculate the frame representation containing emotional information through a linear function

[0165] According to the deep learning-based stress detection device provided by the present invention, based on the negative emotion cycle and the continuous negative emotion attention mechanism, the result is represented in the frame containing the emotional information. Extract the video representation results containing enhanced negative emotion frames Specifically include:

[0166] Extract the frame representation result containing emotional information Negative emotion segments p of consecutive facial frames in ;

[0167] According to the duration of the negative emotion segment p and its arrangement position in the segment, the frame representation result of each frame in the negative emotion segment is weighted enhanced to obtain a video representation result of the facial frame sample containing negative emotion frame enhancement.

[0168] According to the deep learning-based stress detection device provided by the present invention, based on the self-attention mechanism of the video frame with emotional information, the video representation result containing the enhanced negative emotion frame is Embed position information in the video frame to obtain the self-attention result with emotion Specifically include:

[0169] Based on the preset frame position information parameters, the video representation result containing the negative emotion frame enhancement is Each frame in the representation is embedded in position to obtain the embedding result.

[0170] Based on the self-attention mechanism of video frames with emotional information, the embedding representation results Perform frame self-attention calculation to obtain the video frame self-attention result with emotion

[0171] In the above-mentioned specific implementation manner, the deep learning-based pressure detection device provided by the present invention obtains the video to be detected, extracts the facial frame of the target user in the video to be detected; inputs the facial frame of the target user into a pre-trained pressure detection model to obtain the pressure detection result output by the pressure detection model; wherein, the pressure detection model is trained based on facial frame samples, and the sample pressure detection results, sample emotion-oriented description features and sample group classification results corresponding to the facial frame samples; the facial frame samples are extracted from the video samples; the emotion-oriented description features are obtained by superimposing the representation features extracted from the facial frame samples through a multi-layer attention mechanism; the group classification results are obtained by extracting the facial frame samples through a group attention mechanism.

[0172] In this way, the stress detection device provided by the present invention can realize a personalized stress detection method based on ubiquitous video and deep learning, and adopts a three-layer attention mechanism to focus on the user's specific emotional facial expressions, the duration of continuous negative emotions, and the conversion and self-association of emotional frames in the video. It can accurately identify the stress level, thereby solving the technical problem of poor detection accuracy of contactless detection means in the prior art, and can perform relatively accurate contactless detection of the target user's psychological stress level, thereby realizing contactless detection of stress and improving the accuracy of stress detection.

[0173] Fig.11 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.11 As shown, the electronic device may include: a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120 and the memory 1130 communicate with each other through the communication bus 1140. The processor 1110 may call the logic instructions in the memory 1130 to execute the pressure detection device based on deep learning, and the method includes: obtaining a video to be detected, and extracting a facial frame of a target user in the video to be detected;

[0174] Inputting the target user's facial frame into a pre-trained pressure detection model, the pressure detection result output by the pressure detection model can be obtained;

[0175] The stress detection model is trained based on facial frame samples, sample stress detection results corresponding to the facial frame samples, sample emotion-oriented description features, and sample group classification results;

[0176] The facial frame samples are extracted from video samples; the emotion-oriented description features are obtained by superimposing the representation features extracted from the facial frame samples through a multi-layer attention mechanism; and the group classification results are obtained by extracting the facial frame samples through a group attention mechanism.

[0177] In addition, the logic instructions in the above-mentioned memory 1130 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0178] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute a pressure detection device based on deep learning, the method includes: obtaining a video to be detected, extracting a facial frame of a target user in the video to be detected;

[0179] Inputting the target user's facial frame into a pre-trained pressure detection model, the pressure detection result output by the pressure detection model can be obtained;

[0180] The stress detection model is trained based on facial frame samples, sample stress detection results corresponding to the facial frame samples, sample emotion-oriented description features, and sample group classification results;

[0181] The facial frame samples are extracted from video samples; the emotion-oriented description features are obtained by superimposing the representation features extracted from the facial frame samples through a multi-layer attention mechanism; and the group classification results are obtained by extracting the facial frame samples through a group attention mechanism.

[0182] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the pressure detection device based on deep learning provided by the above methods, the method comprising: obtaining a video to be detected, extracting a facial frame of a target user in the video to be detected;

[0183] Inputting the target user's facial frame into a pre-trained pressure detection model, the pressure detection result output by the pressure detection model can be obtained;

[0184] The stress detection model is trained based on facial frame samples, sample stress detection results corresponding to the facial frame samples, sample emotion-oriented description features, and sample group classification results;

[0185] The facial frame samples are extracted from video samples; the emotion-oriented description features are obtained by superimposing the representation features extracted from the facial frame samples through a multi-layer attention mechanism; and the group classification results are obtained by extracting the facial frame samples through a group attention mechanism.

[0186] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0187] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A pressure detection method based on deep learning, characterized in that: include: Obtain a video to be detected, and extract a facial frame of a target user in the video to be detected; Inputting the target user's facial frame into a pre-trained pressure detection model to obtain a pressure detection result output by the pressure detection model; The stress detection model is trained based on facial frame samples, sample stress detection results corresponding to the facial frame samples, sample emotion-oriented description features, and sample group classification results; The facial frame samples are extracted from video samples; the emotion-oriented description features are obtained by superimposing the representation features extracted from the facial frame samples through a multi-layer attention mechanism; and the group classification results are obtained by extracting the facial frame samples through a group attention mechanism.

2. The pressure detection method based on deep learning according to claim 1, characterized in that: Extracting facial frame samples from video samples includes: Acquire video samples, and extract frames that meet preset image quality requirements from the video samples; Based on the preset emotion classification, the probability of the emotion classification detected in each frame is calculated to obtain a sequence of emotion classification probabilities; Extracting a preset number of emotion classification probabilities from the sequence of emotion classification probabilities; The frames corresponding to the preset number of emotion classification probabilities and the frames meeting the preset image quality requirements are taken as the facial frame samples.

3. The pressure detection method based on deep learning according to claim 1, characterized in that: The multi-layer attention mechanism includes at least an emotion expression attention mechanism, a continuous negative emotion attention mechanism, and a video frame self-attention mechanism with emotion information; The facial frame samples are superimposed with the representation features extracted by the multi-layer attention mechanism to obtain the emotion-oriented description features, which specifically include: Based on the emotion expression attention mechanism, feature representation is extracted from the facial frame sample to obtain a frame representation result containing emotion information. Based on the negative emotion cycle and continuous negative emotion attention mechanism, the result is represented in the frame containing emotional information. Extract the video representation results containing enhanced negative emotion frames Based on the self-attention mechanism of video frames with emotional information, the video representation results containing enhanced negative emotional frames are Embed position information in the video frame to obtain the self-attention result with emotion 4. The pressure detection method based on deep learning according to claim 3 is characterized in that: Based on the emotion expression attention mechanism, feature representation is extracted from the facial frame sample to obtain a frame representation result containing emotion information. Specifically include: Construct facial frame sample X F And the sample pressure test result X C A vector of Normalize each element in the vector A to [0, 1], and use the normalized result to represent the attention weight relative to the target emotion category; According to the attention weight relative to the target emotion category and the vector, the classification emotion attention X is obtained. A ; Classify the sentiment attention X A Sample X with face frames F Align each frame in the image, and calculate the frame representation containing emotional information through a linear function 5. The pressure detection method based on deep learning according to claim 3 is characterized in that: Based on the negative emotion cycle and continuous negative emotion attention mechanism, the result is represented in the frame containing emotional information. Extract the video representation results containing enhanced negative emotion frames Specifically include: Extract the frame representation result containing emotional information Negative emotion segments p of consecutive facial frames in ; According to the duration of the negative emotion segment p and its arrangement position in the segment, the frame representation result of each frame in the negative emotion segment is weighted to enhance the video representation result of the facial frame sample including the negative emotion frame enhancement.

6. The pressure detection method based on deep learning according to claim 3 is characterized in that: Based on the self-attention mechanism of video frames with emotional information, the video representation results containing enhanced negative emotional frames are Embed position information in the video frame to obtain the self-attention result with emotion Specifically include: Based on the preset frame position information parameters, the video representation result containing the negative emotion frame enhancement is Each frame in the representation is embedded in position to obtain the embedding result. Based on the self-attention mechanism of video frames with emotional information, the embedding representation results Perform frame self-attention calculation to obtain the video frame self-attention result with emotion 7. The pressure detection method based on deep learning according to claim 1, characterized in that: The facial frame samples are extracted through a group attention mechanism to obtain the group classification result, which specifically includes: Acquire sample users, and construct a user graph based on attribute information of the sample users; Based on the user graph, the features of the sample users are identified and classified through a group attention mechanism to obtain a group classification result.

8. A pressure detection device based on deep learning, characterized in that: include: A video acquisition unit, used to acquire a video to be detected and extract a facial frame of a target user in the video to be detected; A result generating unit, used for inputting the facial frame of the target user into a pre-trained pressure detection model to obtain a pressure detection result output by the pressure detection model; The stress detection model is trained based on facial frame samples, sample stress detection results corresponding to the facial frame samples, sample emotion-oriented description features, and sample group classification results; The facial frame samples are extracted from video samples; the emotion-oriented description features are obtained by superimposing the representation features extracted from the facial frame samples through a multi-layer attention mechanism; and the group classification results are obtained by extracting the facial frame samples through a group attention mechanism.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the deep learning-based pressure detection method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the deep learning-based pressure detection method as described in any one of claims 1 to 7 is implemented.

Images