Non-contact stress state recognition method and system

Abstract

The present invention provides a non-contact stress state recognition method and system, which relates to the field of stress state recognition. The method includes: performing convolution processing on the first data to be processed in the spatial dimension and the time dimension based on a preset feature network to obtain the second data to be processed; performing weighted fusion on the second data to be processed based on the cross-attention weight vector W′ to obtain the weighted fused feature descriptor Y′; performing dimensionality reduction processing on the feature descriptor Y′, and performing stress state recognition on the feature descriptor and HRV signal after dimensionality reduction processing based on the preset multi-classification activation function softmax to obtain the corresponding stress state level. In this way, not only the inconvenience of traditional wearable devices is overcome, but also the multi-dimensional facial data is feature-fused, the key features are selectively selected during the convolution process, the features on each dimension are extracted separately, and the feature fusion of four dimensions is performed to improve the accuracy of stress state level assessment.

Description

Technical Field

[0001] The present invention relates to the technical field of stress state recognition, and in particular to a non-contact stress state recognition method and system. Background Art

[0002] There is a close correlation between user emotions and user stress states. Excessive stress states can cause users to have inattention, irritability and other emotions. Emotions are the result of multiple physiological reactions within an individual, and can be manifested through external physical expressions such as the user's expression, posture, tone, and language. Therefore, in the prior art, the user's emotional state can be obtained through the user's skin electrical signals to identify the user's stress state.

[0003] However, the collection of data such as skin electrical signals is mostly based on wearable devices, such as finger-clip oximeters, wristbands, portable blood pressure monitors, etc. This type of contact state recognition method will cause physiological or psychological interference and discomfort to users, and cannot be promoted for special groups such as newborns and patients with large skin injuries.

[0004] Therefore, a non-contact stress state recognition method is urgently needed to solve the above problems. Summary of the invention

[0005] 1. Technical issues to be solved

[0006] In view of the deficiencies of the prior art, the present invention provides a non-contact stress state recognition method and system, which solves the problem that the contact state recognition method in the prior art brings discomfort to the user.

[0007] (II) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] In a first aspect of the present invention, a non-contact stress state recognition method is provided, the method comprising:

[0010] Obtain visible light video of the face;

[0011] Based on the facial visible light video, first data to be processed is obtained; wherein the first data to be processed includes: a facial motion unit intensity sequence, a gaze angle coordinate sequence, a head posture coordinate sequence and an rPPG signal;

[0012] Based on the preset feature network, the first data to be processed is convolved in the spatial dimension and the temporal dimension to obtain the second data to be processed after the convolution process; wherein the second data to be processed includes: the spatiotemporal feature sequence ST of the facial motion unit intensity sequence A, the spatiotemporal feature sequence ST of the gaze angle coordinate sequence E , the spatiotemporal feature sequence ST of the head posture coordinate sequence H and rPPG signal spatiotemporal feature sequence ST R ; The feature network consists of a VGG16 convolutional neural network and a long short-term memory network;

[0013] Based on the preset query vector Q, key K, value V, gradient d m And softmax function to obtain the cross attention weight vector W′;

[0014] Based on the cross attention weight vector W′, weighted fusion is performed on the second to-be-processed data to obtain a weighted fused feature descriptor Y′;

[0015] The feature descriptor Y′ is subjected to dimensionality reduction processing, and stress state identification is performed on the feature descriptor and HRV signal after dimensionality reduction processing based on a preset multi-classification activation function softmax to obtain a corresponding stress state level; wherein the HRV signal is obtained based on the rPPG signal.

[0016] Optionally, performing convolution processing on the first data to be processed in spatial dimension and time dimension based on a preset feature network to obtain second data to be processed after convolution processing includes:

[0017] Performing convolution processing on the first data to be processed in the spatial dimension based on a preset VGG16 network, and performing convolution processing on the first data to be processed in the temporal dimension based on a preset long short-term memory network, and performing feature fusion on the convolution-processed data to obtain second data to be processed;

[0018] The performing convolution processing on the first to-be-processed data in the spatial dimension based on the preset VGG16 network includes:

[0019] Based on the convolution layer in the preset VGG16 network, the first to-be-processed data is subjected to convolution processing, hierarchical features in each pooling layer in the VGG16 network are obtained, and a weight value is set for each hierarchical feature;

[0020] Based on the weight values, feature fusion processing is performed on each hierarchical feature of the first data to be processed, and the features after the feature fusion processing are input into the fully connected layer in the preset VGG16 network.

[0021] Optionally, obtaining first data to be processed based on the facial visible light video includes:

[0022] Acquire each image frame in the facial visible light video as a first image to be processed;

[0023] Convert the RGB values ​​of the pixels in the first image to be processed into a color representation in a YCbCr space as a second image to be processed;

[0024] Based on the second image to be processed and the openface open source toolkit, a facial motion unit intensity sequence, a gaze angle coordinate sequence and a head posture coordinate sequence are obtained.

[0025] Optionally, after acquiring each image frame in the facial visible light video as the first image to be processed, the method further includes:

[0026] Dividing a face region of interest in the first image to be processed based on the flooding method;

[0027] The rPPG signal in the face region of interest is extracted based on a planar orthogonal skin algorithm.

[0028] In a second aspect of the present invention, a non-contact stress state recognition system is provided, the system comprising:

[0029] A first acquisition module is used to acquire facial visible light video;

[0030] A second acquisition module is used to acquire first data to be processed based on the facial visible light video; wherein the first data to be processed includes: a facial motion unit intensity sequence, a gaze angle coordinate sequence, a head posture coordinate sequence and an rPPG signal;

[0031] A convolution processing module is used to perform convolution processing of the first data to be processed in the spatial dimension and the temporal dimension based on a preset feature network to obtain the second data to be processed after the convolution processing; wherein the second data to be processed includes: the spatiotemporal feature sequence ST of the facial motion unit intensity sequence A , the spatiotemporal feature sequence ST of the gaze angle coordinate sequence E , the spatiotemporal feature sequence ST of the head posture coordinate sequence H and rPPG signal spatiotemporal feature sequence ST R ; The feature network consists of a VGG16 convolutional neural network and a long short-term memory network;

[0032] The third acquisition module is used to obtain the value of the query vector Q, key K, value V, and gradient d based on the preset query vector Q, key K, value V, and gradient d. m And softmax function to obtain the cross attention weight vector W′;

[0033] A weighted fusion module, used for performing weighted fusion on the second to-be-processed data based on the cross attention weight vector W′ to obtain a weighted fused feature descriptor Y′;

[0034] The state recognition module is used to perform dimensionality reduction processing on the feature descriptor Y′, and to identify the stress state of the feature descriptor and HRV signal after dimensionality reduction processing based on a preset multi-classification activation function softmax to obtain a corresponding stress state level; wherein the HRV signal is obtained based on the rPPG signal.

[0035] Optionally, the convolution processing module includes:

[0036] A feature fusion submodule, configured to perform convolution processing on the first data to be processed in the spatial dimension based on a preset VGG16 network, and perform convolution processing on the first data to be processed in the temporal dimension based on a preset long short-term memory network, and perform feature fusion on the convolution-processed data to obtain second data to be processed;

[0037] The performing convolution processing on the first to-be-processed data in the spatial dimension based on the preset VGG16 network includes:

[0038] Based on the convolution layer in the preset VGG16 network, the first to-be-processed data is subjected to convolution processing, hierarchical features in each pooling layer in the VGG16 network are obtained, and a weight value is set for each hierarchical feature;

[0039] Based on the weight values, feature fusion processing is performed on each hierarchical feature of the first data to be processed, and the features after the feature fusion processing are input into the fully connected layer in the preset VGG16 network.

[0040] Optionally, the second acquisition module includes:

[0041] A first acquisition submodule is used to acquire each image frame in the facial visible light video as a first image to be processed;

[0042] A first conversion submodule, used for converting the RGB values ​​of the pixels in the first image to be processed into a color representation in a YCbCr space as a second image to be processed;

[0043] The second acquisition submodule is used to acquire a facial motion unit intensity sequence, a gaze angle coordinate sequence and a head posture coordinate sequence based on the second image to be processed and the openface open source toolkit.

[0044] Optionally, the convolution processing module further includes:

[0045] A first segmentation submodule is used for segmenting a human face region of interest in the first image to be processed based on a flooding method after taking each image frame in the facial visible light video as a first image to be processed;

[0046] The first extraction submodule is used to extract the rPPG signal in the face region of interest based on a planar orthogonal skin algorithm.

[0047] In a third aspect of the present invention, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0048] Memory, used to store computer programs;

[0049] The processor is used to implement any of the steps of the above-mentioned non-contact stress state identification method when executing the program stored in the memory.

[0050] In a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any of the above-mentioned non-contact stress state identification methods are implemented.

[0051] (III) Beneficial effects

[0052] The present invention provides a non-contact stress state recognition method and system. Compared with the prior art, it has the following beneficial effects:

[0053] The method comprises: obtaining a facial visible light video; obtaining first data to be processed based on the facial visible light video; wherein the first data to be processed comprises: a facial motion unit intensity sequence, a gaze angle coordinate sequence, a head posture coordinate sequence and an rPPG signal; performing convolution processing of the first data to be processed in spatial dimension and time dimension based on a preset feature network to obtain second data to be processed after convolution processing; and obtaining a second data to be processed based on a preset query vector Q, a key K, a value V, a gradient d m and softmax function to obtain a cross-attention weight vector W′; based on the cross-attention weight vector W′, weighted fusion is performed on the second data to be processed to obtain a weighted fused feature descriptor Y′; dimension reduction is performed on the feature descriptor Y′, and stress state recognition is performed on the feature descriptor and HRV signal after dimension reduction based on a preset multi-classification activation function softmax to obtain a corresponding stress state level.

[0054] Based on the above processing, not only the inconvenience and high cost of traditional wearable devices are overcome, but also the multi-dimensional facial data is fused to perform non-contact stress state recognition. Not only can key features be selected in the convolution process to realize and distinguish the extraction of features in each dimension, but also the four-dimensional feature fusion can be performed in the multi-dimensional feature fusion process, thereby effectively improving the accuracy of stress state level assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0056] Figure 1 A flow chart of a non-contact stress state identification method provided by the present invention;

[0057] Figure 2 A schematic diagram of a VGG16 network and a cross attention module provided by the present invention;

[0058] Figure 3 A flowchart of another non-contact stress state identification method provided by the present invention;

[0059] Figure 4 A flowchart of another non-contact stress state identification method provided by the present invention;

[0060] Figure 5 An architecture diagram of a non-contact stress state recognition system provided by the present invention;

[0061] Figure 6 A hardware architecture diagram of a non-contact stress state recognition system provided by the present invention;

[0062] Figure 7 A software architecture diagram of a non-contact stress state recognition system provided by the present invention;

[0063] Figure 8 This is a structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] The embodiment of the present application provides a non-contact stress state recognition method and system, which solves the problem that the contact state recognition method in the prior art brings discomfort to users, and realizes feature fusion of multi-dimensional facial data, thereby performing non-contact stress state recognition. Not only can key features be selectively selected in the convolution process to realize and distinguish the extraction of features in each dimension, but also in the process of multi-dimensional feature fusion, four-dimensional feature fusion can be performed, thereby effectively improving the accuracy of stress state level assessment.

[0066] The technical solution in the embodiment of the present invention is to solve the above technical problems, and the overall idea is as follows:

[0067] Existing stress state recognition methods are mostly contact-based state recognition, and data collection is mostly based on wearable devices, which cause physiological or psychological interference and discomfort to users, and cannot be promoted for use in special populations such as newborns and patients with large skin trauma.

[0068] At the same time, the existing technology recognizes stress states based on a single facial feature. Since it is easy to miss important information when using a single facial feature, the input data of the model processing process is highly one-sided, which reduces the scene adaptability of the model.

[0069] In order to solve the above problems, the present invention provides a non-contact stress state recognition method and system, the method comprising: obtaining a facial visible light video; based on the facial visible light video, obtaining first data to be processed; wherein the first data to be processed comprises: a facial motion unit intensity sequence, a gaze angle coordinate sequence, a head posture coordinate sequence and an rPPG signal; based on a preset feature network, performing convolution processing on the first data to be processed in spatial dimension and time dimension to obtain second data to be processed after convolution processing; based on a preset query vector Q, key K, value V, gradient d m and softmax function to obtain a cross-attention weight vector W′; based on the cross-attention weight vector W′, weighted fusion is performed on the second data to be processed to obtain a weighted fused feature descriptor Y′; dimension reduction is performed on the feature descriptor Y′, and stress state recognition is performed on the feature descriptor and HRV signal after dimension reduction based on a preset multi-classification activation function softmax to obtain a corresponding stress state level.

[0070] Based on the above processing, the present invention not only overcomes the inconvenience and high cost of traditional wearable devices, but also fuses the multi-dimensional facial data to perform non-contact stress state recognition. Not only can key features be selected in the convolution process, and features in each dimension can be extracted separately, but also in the multi-dimensional feature fusion process, feature fusion of four dimensions can be performed, and HRV features strongly related to the stress state can be used to effectively improve the accuracy of stress state level assessment.

[0071] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0072] See also Figure 1 , Figure 1 A flow chart of a non-contact stress state recognition method provided by the present invention, such as Figure 1 As shown, the method includes:

[0073] S1. Obtain facial visible light video.

[0074] S2. Based on the facial visible light video, obtain first data to be processed; wherein the first data to be processed includes: a facial motion unit intensity sequence, a gaze angle coordinate sequence, a head posture coordinate sequence and an rPPG signal.

[0075] S3, based on the preset feature network, the first data to be processed is convolved in the spatial dimension and the temporal dimension to obtain the second data to be processed after the convolution processing; wherein the second data to be processed includes: the spatiotemporal feature sequence ST of the facial motion unit intensity sequence A , the spatiotemporal feature sequence ST of the gaze angle coordinate sequence E , the spatiotemporal feature sequence ST of the head posture coordinate sequence H and rPPG signal spatiotemporal feature sequence ST R ; The feature network consists of a VGG16 convolutional neural network and a long short-term memory network.

[0076] S4, based on the preset query vector Q, key K, value V, gradient d m And softmax function to obtain the cross attention weight vector W′.

[0077] S5. Based on the cross-attention weight vector W′, weighted fusion is performed on the second data to be processed to obtain a weighted fused feature descriptor Y′.

[0078] S6. Perform dimensionality reduction processing on the feature descriptor Y′, and perform stress state recognition on the feature descriptor and HRV signal after dimensionality reduction processing based on a preset multi-classification activation function softmax to obtain a corresponding stress state level.

[0079] The HRV signal is obtained based on the rPPG signal. Specifically, the HRV signal represents the heart rate variability (HRV) index extracted from the rPPG signal, that is, the root mean square value RMSSD of the difference between adjacent NN intervals. In the actual working process, the change of facial blood concentration is measured from the rPPG signal to calculate BVP (Blood Volume Pulse), and the heart rate variability is calculated based on the peak value of BVP.

[0080] Based on the above processing, the present invention not only overcomes the inconvenience and high cost of traditional wearable devices, but also fuses multi-dimensional facial data to perform non-contact stress state recognition. Not only can key features be selectively obtained during the convolution process, features in various dimensions can be extracted, but also in the multi-dimensional feature fusion process, feature fusion in four dimensions can be performed, and HRV features that are strongly related to the stress state can be fused and used at the same time, thereby effectively improving the accuracy of stress state level assessment.

[0081] With respect to step S1, a visible light video of the user's face may be obtained through a video acquisition device such as a visible light camera.

[0082] For example, for a user who is driving, the visible light camera of the vehicle-mounted robot can be used to collect the visible light video of the user. Before collecting, the distance between the user and the visible light camera and the position of the visible light camera can be adjusted according to the actual situation, so that the visible light face of the user is fully displayed on the screen during the collection process, ensuring the collection quality of the visible light video. In addition, the duration of the visible light video can be one minute, which is not specifically limited here.

[0083] In some embodiments, step S2 comprises the following steps:

[0084] S201. Acquire each image frame in the facial visible light video as a first image to be processed.

[0085] S202: Convert the RGB values ​​of the pixels in the first image to be processed into a color representation in a YCbCr space as a second image to be processed.

[0086] S203. Based on the second image to be processed and the openface open source toolkit, a facial motion unit intensity sequence, a gaze angle coordinate sequence, and a head posture coordinate sequence are obtained.

[0087] For step S202, the technical solution provided by the present invention converts the RGB value of the pixel in the RGB picture in the visible light video (i.e., the first image to be processed in the present invention) into the color representation of the YCbCr space, that is, mapping the three channels of R, G, and B to the three channels of Y, Cb, and Cr.

[0088] Among them, G represents the value of the Green component in the RGB space, B represents the value of the Blue component in the RGB space, R represents the value of the Red component in the RGB space, Cr and Cb represent the concentration offset of red and blue respectively, and the brightness is Y.

[0089] Specifically, the conversion formula is as follows: Y = 0.299R + 0.578G + 0.114B

[0090] When Y>18 or Y<60:

[0091] Cr=0.5R+0.419G+0.081B;

[0092] Cb=0.169R+0.331G+0.5B.

[0093] When 18≤Y≤60:

[0094]

[0095]

[0096] Among them, a1>0, b1<0; a2>0, b2<0. a1, a2, b1, b2 represent function parameters, which are obtained by fitting specific data.

[0097] Based on the above processing, after the RGB value of the pixel is converted into the color representation of the YCbCr space, some redundant invalid information can be discarded, thereby reducing the bandwidth in the image processing transmission process, thereby effectively improving the efficiency of image processing and reducing the time required for stress state recognition.

[0098] For step S203, for the second image to be processed, OpenFace uses conditional localization neural field (CLNF) to describe and track facial feature points. The two main components of CLNF are: point distribution model (PDM) to capture landmark shape changes; patch expert to capture the appearance changes of each landmark.

[0099] On this basis, CLNF internally uses a 3D representation of facial landmarks and projects them onto the image using orthographic camera projection, thereby accurately estimating head pose and gaze angle.

[0100] Specifically, the gaze angle coordinate sequence is:

[0101] EG=(EG 1 ,EG 2 ,…,EG t ,…,EG T );

[0102]

[0103] Among them, EG represents the gaze angle coordinate sequence, EG t represents the gaze angle vector in the tth frame, and each component is the x, y, and z components of the left eye gaze direction vector (normalized) The x, y, and z components of the right eye gaze direction vector (normalized) The average value of the arc of the two gaze angle directions, is the average value of the arc when the eyes look left and right and deviate from the front. It is the average value of the arc that the eyes deviate from the front when looking up and down (the above coordinates are all in the world coordinate system with the camera as the origin).

[0104] The head posture coordinate sequence is:

[0105] HP=(HP 1 ,HP 2 ,…,HP t ,…,HP T );

[0106]

[0107] Among them, HP represents the head posture coordinate sequence, HP t Represents the gaze angle vector in the tth frame, and each component is the position (x, y, z) coordinate of the head relative to the camera. The rotation angle of the head in three dimensions in space (Yaw, Pitch, Roll) coordinate yaw t ,pitch t ,roll t (The above coordinates are all in the world coordinate system with the camera as the origin).

[0108] In one implementation, facial motion units are extracted using a new network model of a deep convolutional neural network proposed in the 2015 Facial Expression Recognition and Analysis Challenge to identify AUs. The network model consists of three convolutional layers and a maximum pooling layer, and the final fully connected layer provides classification output.

[0109] Specifically, the facial motion unit intensity sequence is:

[0110] AU=(AU1 ,AU 2 ,…,AU t ,…,AU T ).

[0111] The facial motion unit intensity vector in frame t:

[0112]

[0113] Among them, AU represents the facial motion unit intensity sequence, AU T represents the facial motion unit intensity vector in the tth frame, They represent the intensity of the 46 facial motion units in the t-th frame respectively.

[0114] In some embodiments, due to exercise or emotional stimulation, the blood flow in a person's blood vessels may change, which may manifest as extremely subtle color changes on the skin surface, and thus may be captured using photoplethysmography (rPPG).

[0115] Step S2 also includes the following steps:

[0116] S204: Divide the face region of interest in the first image to be processed based on the flooding method.

[0117] S205: extracting rPPG signals in the face region of interest based on a planar orthogonal skin algorithm.

[0118] The flooding method is used to divide the region of interest (ROI) of the face, and then the plane orthogonal skin algorithm (Plane Orthogonal-to-Skin, POS) is used to extract the rPPG signal, and the rPPG signal can be used for timing analysis and frequency domain analysis.

[0119] Specifically, let the color signal of the t-th frame image be C t ∈R 3 , represents the mean value of all pixels in the RGB three channels in the current ROI. Let the total skin color signal of the visible light video be T∈R 3×L , L represents the total number of frames of the video. Among them, Represents the mean value of all pixels in the RGB channels in the ROI of the second frame image.

[0120] The extraction process of rPPG signal is as follows: First, the color signal C is normalized by dividing the RGB value of each frame by the corresponding mean value of all frames, that is, Among them, C n is the normalized skin color signal, μ (C) is the row mean of C.

[0121] Then, C n Projection into two orthogonal projection directions, the projection direction is denoted as P, where The normalized signal C is converted by P n Projection, get the projection vector Wherein, SG represents a projection vector, SG1 represents a projection in the orthogonal projection direction, which can be understood as a motion interference signal; SG2 represents another projection in the orthogonal projection direction, which can be understood as a pulse signal;

[0122] Finally, the above projection vector is processed by standard deviation term to obtain the pulse signal as rPPG signal:

[0123]

[0124] Wherein, H represents the rPPG signal, σ(SG1) and σ(SG2) represent the standard deviations of SG1 and SG2, respectively.

[0125] In one implementation, after the rPPG signal is obtained, the rPPG signal is filtered and then analyzed in the frequency domain to extract an important parameter of HRV (Heart Rate Variety) - the square root of the mean sum of squared differences (RMSSD) between adjacent NN intervals, thereby characterizing the degree to which the heart rate rhythm deviates from the normal value, which can be used to assess the level of stress.

[0126] The calculation formula is: Where IBI is the time period between consecutive heartbeats, and N represents the number of frames.

[0127] In some implementations, step S3 includes the following:

[0128] S301. Perform convolution processing on the first data to be processed in the spatial dimension based on the preset VGG16 network, and perform convolution processing on the first data to be processed in the time dimension based on the preset long short-term memory network, and perform feature fusion on the convolution-processed data to obtain second data to be processed.

[0129] The convolution processing on the first to-be-processed data in the spatial dimension based on the preset VGG16 network includes the following steps:

[0130] Step a: performing convolution processing on the first data to be processed based on the convolution layer in the preset VGG16 network, obtaining the hierarchical features in each pooling layer in the VGG16 network, and setting a weight value for each hierarchical feature.

[0131] Step b: performing feature fusion processing on each hierarchical feature of the first data to be processed based on the weight value, and inputting the features after feature fusion processing into the fully connected layer in the preset VGG16 network.

[0132] In one implementation, the four temporal feature vectors of facial motion unit, gaze angle, head posture, and rPPG signal are represented as T A , T E , T H , T R , the corresponding initial state is T A0 , T E0 , T H0 , T R0 ; The four space vectors are represented as S A , S E , S H , S R , the corresponding initial state is S A0 , S E0 , S H0 , S R0 .

[0133] In some embodiments, the feature network is composed of a VGG16 convolutional neural network and a long short-term memory network. Convolution is performed in space and time respectively, and finally fused into a spatiotemporal feature sequence as ST A , ST E , ST H , ST R , the sequence length is N (according to the parameter setting of the convolutional neural network).

[0134] In one implementation, the convolution processing on the first to-be-processed data in the time dimension is performed based on a preset long short-term memory network, which can be expressed as:

[0135] T A =f(T A0 ), T E =f(T E0 ),

[0136] T H =f(T H0 ), T R =f(T R0 )

[0137] Where f(x) represents the temporal convolution function.

[0138] For the VGG16 network, the technical solution provided by the present invention is optimized to solve the problem of key feature loss in the layer-by-layer pooling process. Specifically, for the feature vector S i(where i = A, E, H, R), after the convolution operation, after each pooling layer, the hierarchical features S in each pooling layer are obtained in turn ij (where j = 1, 2, 3, 4, 5), and assign weights w in turn k (k=1,2,3,4,5, the specific value is determined according to the experimental results), and the features after layered feature fusion are input to the fully connected layer, and finally the features are fused with the data after the above convolution processing in the time dimension. The specific process can be expressed as:

[0139] S A =w1*S A1 ⊙w2*S A2 ⊙w3*S A3 ⊙w4*S A4 ⊙w5*S A5

[0140] S E =w1*S E1 ⊙w2*S E2 ⊙w3*S E3 ⊙w4*S E4 ⊙w5*S E5

[0141] S H =w1*S H1 ⊙w2*S H2 ⊙w3*S H3 ⊙w4*S H4 ⊙w5*S H5

[0142] S R =w1*S R1 ⊙w2*S R2 ⊙w3*S R3 ⊙w4*S R4 ⊙w5*S R5

[0143]

[0144]

[0145]

[0146]

[0147] Among them, ⊙ represents the feature fusion function, and normalization is performed. The specific processing process can be found in Figure 2 , Figure 2 A schematic diagram of a VGG16 network and a cross-attention module provided by the present invention.

[0148] like Figure 2 As shown in Figure 2, during the convolution process in the layered VGG16 network, the hierarchical features S are obtained in turn when passing through each pooling layer. ij , and assign corresponding weight value w to the hierarchical features of each pooling layer k Among them, the first three convolutional layers are the first pooling layer, the next two convolutional layers are the second pooling layer, the next three convolutional layers are the third pooling layer, the subsequent three convolutional layers are the fourth pooling layer, and the last three convolutional layers are the fifth pooling layer.

[0149] Depend on Figure 2 It can be seen that for the facial features of the above four dimensions (i.e., facial motion unit intensity sequence, gaze angle coordinate sequence, head posture coordinate sequence and rPPG signal), after feature fusion, the data after feature fusion is processed based on the cross-attention module, and the user's stress state is evaluated into four levels: no stress, mild stress, moderate stress and severe stress through the multi-classification activation function softmax.

[0150] In one implementation, in the criss-cross attention module, given a set of queries Q, keys K, values ​​V, and gradients d m (Set according to experimental results), the present invention uses the softmax function to obtain the cross-attention weight vector W′.

[0151] Specifically, take the fusion of facial motion unit intensity sequence and rPPG signal as an example:

[0152]

[0153]

[0154]

[0155]

[0156]

[0157]

[0158] Among them, Q AR Indicates Q A and Q R The combined query vector; Q A represents the query of facial motion units; Q R Indicates the query of rPPG signal; K AR K A and K R The key vector of the combination; V AR Indicates V A and V RThe value vector of the combination; w A ′ represents the feature weight value after cross-fusion of facial motion unit and rPPG signal; w E ′ represents the feature weight value after cross-fusion of head posture and rPPG signal; w H ′ represents the feature weight value after cross-fusion of rPPG signal and gaze angle; w R ″ represents the feature weight value after cross-fusion of rPPG signal and head posture; w R ″′ represents the feature weight value after cross-fusion of rPPG signal and gaze angle.

[0159] Similarly, based on the cross-attention weight vector W′, the second data to be processed are weightedly fused to obtain a weighted fused feature descriptor Y′.

[0160] Specifically: Y′=(Y A ′,Y E ′,Y H ′,Y R ′)

[0161] Among them, Y A ′=w A ′*ST A ,Y E ′=w E ′*ST E

[0162] Y H ′=w H ′*ST H ,Y R ′=(w R ′⊙w R ″⊙w R ″′)*ST R .

[0163] Among them, Y A ′ represents the facial motion unit feature after weighted fusion; Y E ′ represents the gaze angle feature after weighted fusion; Y H ′ represents the head posture feature after weighted fusion; Y R ′ represents the rPPG signal features after weighted fusion.

[0164] Then, the multi-dimensional spatiotemporal features of the weighted fusion are reduced in dimension, and the multi-classification activation function softmax is applied to determine whether the stress state is stressful or not. Moreover, under the premise of detecting stress, the HRV parameter RMSSD extracted from the rPPG signal is used to evaluate the three levels of mild stress, moderate stress, and severe stress, such as Figure 3 ,4 shown.

[0165] In addition, for the cross attention module, the technical solution provided by the present invention can also perform adversarial adjustment of feature weights for the following three problems:

[0166] 1. Occlusion problem

[0167] For data that blocks the forehead or cheek, the feature weight of the rPPG signal can be reduced; for data that blocks the eyes, the feature weight of the gaze angle can be reduced; for other occluded data, the feature weight of the AU can be reduced.

[0168] 2. Posture Problems

[0169] After extracting the head posture, for data that deviates more from the correct orientation, the feature weight of the rPPG signal is reduced when it deviates horizontally, and the feature weight of the AU is reduced when it deviates vertically.

[0170] 3. Skin color problem

[0171] For users with dark skin, the extraction accuracy of rPPG information is affected, and the feature weight of the rPPG signal can be reduced.

[0172] In the actual work process, after obtaining the data after the conversion of AU, head posture and skin color, the data after the conversion of AU, head posture and skin color is classified and processed. The classification results include: normal class, occlusion problem class, posture problem class, and skin color problem class.

[0173] For normal data, the non-contact stress state recognition method is used. For occlusion problem data, the feature weight value w′ is A Reduce a and set the feature weight w″′ R Increase a. (where a is a constant, and a∈(0,1), the specific value can be determined based on experimental data)

[0174] Based on the above processing, the present invention provides a multi-dimensional feature fusion method based on hierarchical features and cross-attention fusion model. For single-dimensional features, a hierarchical feature fusion VGG16 model is proposed, which can retain key features during the convolution process and extract features in each dimension separately; in the multi-dimensional feature fusion process, the cross-attention module is used to quantify its contribution in stress state recognition, and the feature fusion of four dimensions is performed to achieve a more accurate stress state level assessment.

[0175] In one implementation, see Figure 4 , Figure 4 Flow chart of another non-contact stress state identification method provided by the present invention. Figure 4As shown, by slicing the video (i.e., the visible light video provided by the present invention), an image (i.e., the first image to be processed provided by the present invention) is obtained. Then, the image is processed to obtain an rPPG signal. At the same time, the image is color converted, and feature sequences such as AU, eye gaze and head posture are obtained based on the new image after color conversion. Then, the feature descriptors of the above four dimensions are input into the hierarchical VGG16 convolutional network and the LSTM network for processing to determine whether the user is in a stress state. If so, the level of the stress state is evaluated in combination with the HRV signal. Among them, the levels of stress state include: mild stress, moderate stress and severe stress. It is worth noting that the HRV signal is used as a standard signal for stress state detection. The present invention divides the stress state level by the HRV signal, which not only reduces the processing difficulty of the stress state level division, but also effectively improves the accuracy of the stress state level division.

[0176] The present invention also provides a non-contact stress state recognition system, see Figure 5 , Figure 5 The architecture diagram of a non-contact stress state recognition system provided by the present invention is as follows: Figure 5 As shown, the system includes:

[0177] The first acquisition module 501 is used to acquire a facial visible light video;

[0178] A second acquisition module 502 is used to acquire first data to be processed based on the facial visible light video; wherein the first data to be processed includes: a facial motion unit intensity sequence, a gaze angle coordinate sequence, a head posture coordinate sequence and an rPPG signal;

[0179] The convolution processing module 503 is used to perform convolution processing on the first data to be processed in spatial dimension and time dimension based on a preset feature network to obtain the second data to be processed after convolution processing; wherein the second data to be processed includes: the spatiotemporal feature sequence ST of the facial motion unit intensity sequence A , the spatiotemporal feature sequence ST of the gaze angle coordinate sequence E , the spatiotemporal feature sequence ST of the head posture coordinate sequence H and rPPG signal spatiotemporal feature sequence ST R ; The feature network consists of a VGG16 convolutional neural network and a long short-term memory network;

[0180] The third acquisition module 504 is used to obtain the value of the query vector Q, key K, value V, and gradient d based on the preset query vector Q, key K, value V, and gradient d m And softmax function to obtain the cross attention weight vector W′;

[0181] A weighted fusion module 505 is used to perform weighted fusion on the second to-be-processed data based on the cross attention weight vector W′ to obtain a weighted fused feature descriptor Y′;

[0182] The state recognition module 506 is used to perform dimension reduction processing on the feature descriptor Y′, and perform stress state recognition on the feature descriptor after dimension reduction processing based on a preset multi-classification activation function softmax to obtain a corresponding stress state level.

[0183] Optionally, the convolution processing module 503 includes:

[0184] A feature fusion submodule, configured to perform convolution processing on the first data to be processed in the spatial dimension based on a preset VGG16 network, and perform convolution processing on the first data to be processed in the temporal dimension based on a preset long short-term memory network, and perform feature fusion on the convolution-processed data to obtain second data to be processed;

[0185] The performing convolution processing on the first to-be-processed data in the spatial dimension based on the preset VGG16 network includes:

[0186] Based on the convolution layer in the preset VGG16 network, the first to-be-processed data is subjected to convolution processing, hierarchical features in each pooling layer in the VGG16 network are obtained, and a weight value is set for each hierarchical feature;

[0187] Based on the weight values, feature fusion processing is performed on each hierarchical feature of the first data to be processed, and the features after the feature fusion processing are input into the fully connected layer in the preset VGG16 network.

[0188] Optionally, the second acquisition module 502 includes:

[0189] A first acquisition submodule is used to acquire each image frame in the facial visible light video as a first image to be processed;

[0190] A first conversion submodule, used for converting the RGB values ​​of the pixels in the first image to be processed into a color representation in a YCbCr space as a second image to be processed;

[0191] The second acquisition submodule is used to acquire a facial motion unit intensity sequence, a gaze angle coordinate sequence and a head posture coordinate sequence based on the second image to be processed and the openface open source toolkit.

[0192] Optionally, the convolution processing module 503 further includes:

[0193] A first segmentation submodule is used for segmenting a human face region of interest in the first image to be processed based on a flooding method after taking each image frame in the facial visible light video as a first image to be processed;

[0194] The first extraction submodule is used to extract the rPPG signal in the face region of interest based on a planar orthogonal skin algorithm.

[0195] It is understandable that the non-contact stress state recognition system provided by the embodiment of the present invention corresponds to the above-mentioned non-contact stress state recognition method, and the explanations, examples, beneficial effects and other parts of its relevant contents can refer to the corresponding contents in the non-contact stress state recognition method, which will not be repeated here.

[0196] In one implementation, the non-contact stress state recognition system is constructed according to the idea of ​​layering and componentization, and can be specifically divided into two major components: hardware and software.

[0197] See also Figure 6 , Figure 6 The hardware architecture diagram of a non-contact stress state recognition system provided by the present invention is shown in FIG. Figure 6 As shown in the figure, the hardware part mainly consists of visual sensors (visible light sensors, depth sensors), display screens, buttons, storage hard disks, wireless modules, video modules, power supplies, and artificial intelligence mainboards. The visual sensor provides a visible light video acquisition interface, the artificial intelligence mainboard provides a video processing and model calculation interface, and finally the data is stored in the storage hard disk, and the relevant results are visualized by the display screen.

[0198] See also Figure 7 , Figure 7 A software architecture diagram of a non-contact stress state recognition system provided by the present invention, such as Figure 7 As shown in the figure, the software part is mainly composed of three main layers: hardware driver layer, middle layer and business layer, and the data transmission between each layer is realized through the EventBus data bus. The hardware driver layer is composed of display driver, sensor driver, network driver, interface driver and some basic components. This layer mainly provides communication between the host system and various hardware, and outputs the basic data stream required by the system; the middle layer is composed of face recognition, model calculation, neural network, file management, database, and image processing modules. This layer mainly performs data processing, generates data that meets the input of each model, and provides a data interface to the business layer; the business layer is composed of permission management, two-way interaction, data charts, and stress state recognition modules, which mainly calculate the output results of the model and visualize them in a graphical interface.

[0199] The non-contact stress state recognition system software can be divided into three layers: hardware driver layer, middle layer and business layer, which includes multiple modules. The middle layer receives the basic data generated by the hardware driver layer and generates data for use by the business layer.

[0200] The face recognition module inputs the visible light video data transmitted by the visual sensor, and outputs the video data with face recognition identification, which is transmitted to the image processing module; the image processing module outputs the video data after cropping, alignment, color conversion and other processing, which is transmitted to the calculation model. The model calculation output is the classification value of the stress state, which can be transmitted to the data chart module for visualization. The file management module mainly manages and calls various data of the system. The database module receives the output data of each module in this layer and outputs the data information called by the file management system. The two-way interaction module is mainly composed of the user interface, etc. The humanized user interface and operation method make the system easy to use. The stress state recognition module inputs the detection instruction clicked by the user. The stress state recognition module will call the corresponding model calculation module in the middle layer, output the corresponding stress state classification value, and display it to the user through the user interface. The data chart module inputs the stress state classification value within a period of time, and outputs the stress state change curve within a period of time, which can clearly obtain the changes in the user's stress state within a period of time.

[0201] The embodiment of the present invention further provides an electronic device, such as Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804.

[0202] Memory 803, used for storing computer programs;

[0203] The processor 801 is used to implement any of the above-mentioned non-contact stress state recognition methods when executing the program stored in the memory 803.

[0204] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0205] The communication interface is used for communication between the above electronic device and other devices.

[0206] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0207] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0208] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned non-contact stress state identification methods are implemented.

[0209] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer is enabled to execute any non-contact stress state identification method in the above embodiments.

[0210] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated.

[0211] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0212] In summary, compared with the prior art, the technical solution provided by the present invention has the following beneficial effects:

[0213] 1. In the non-contact stress state recognition method, the present invention proposes a full-process method based on the existing stress recognition method - a multi-dimensional feature fusion method based on hierarchical features and cross-attention fusion model. In the single-dimensional feature, a hierarchical feature fusion VGG16 model is proposed to retain key features during the convolution process, realize and distinguish the extraction of features in each dimension; in the multi-dimensional feature fusion process, the cross-attention module is used to quantify its contribution in stress state recognition, and the feature fusion of four dimensions is performed to achieve a more accurate stress state level assessment.

[0214] 2. In the non-contact stress state recognition system, through hardware modules such as visual sensors and displays and software modules such as intelligent computing models, the interaction between the hardware and software modules is designed, forming a complete link of acquisition-processing-computation-analysis-output from visible light sensors, image processing modules, intelligent computing models to visualization modules, thus realizing the systematization of interactive and visual non-contact stress state recognition.

[0215] 3. The technical solution provided by the present invention effectively utilizes multi-dimensional facial data and overcomes the inconvenience and high cost of traditional wearable devices. Specifically, after extracting rich facial features, rPPG signals are extracted, and a feature fusion method combining hierarchical features and cross-attention is proposed, which breaks through the one-sidedness of traditional single data stress state recognition, utilizes multi-dimensional facial features, and realizes multi-classification of stress states.

[0216] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0217] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A non-contact stress state recognition method, characterized in that: The method comprises: Obtain facial visible light video; Based on the facial visible light video, first data to be processed is obtained; wherein the first data to be processed includes: a facial motion unit intensity sequence, a gaze angle coordinate sequence, a head posture coordinate sequence and an rPPG signal; Based on the preset feature network, the first data to be processed is convolved in the spatial dimension and the temporal dimension to obtain the second data to be processed after the convolution process; wherein the second data to be processed includes: the spatiotemporal feature sequence ST of the facial motion unit intensity sequence A , the spatiotemporal feature sequence ST of the gaze angle coordinate sequence E , the spatiotemporal feature sequence ST of the head posture coordinate sequence H and rPPG signal spatiotemporal feature sequence ST R ; The feature network consists of a VGG16 convolutional neural network and a long short-term memory network; Based on the preset query vector Q, key K, value V, gradient d m And softmax function to obtain the cross attention weight vector W′; Based on the cross attention weight vector W′, weighted fusion is performed on the second to-be-processed data to obtain a weighted fused feature descriptor Y′; The feature descriptor Y′ is subjected to dimensionality reduction processing, and stress state identification is performed on the feature descriptor and HRV signal after dimensionality reduction processing based on a preset multi-classification activation function softmax to obtain a corresponding stress state level; wherein the HRV signal is obtained based on the rPPG signal.

2. The method according to claim 1, characterized in that The method of performing convolution processing of the spatial dimension and the temporal dimension on the first data to be processed based on the preset feature network to obtain the second data to be processed after the convolution processing includes: Performing convolution processing on the first data to be processed in the spatial dimension based on a preset VGG16 network, and performing convolution processing on the first data to be processed in the temporal dimension based on a preset long short-term memory network, and performing feature fusion on the convolution-processed data to obtain second data to be processed; The performing convolution processing on the first to-be-processed data in the spatial dimension based on the preset VGG16 network includes: Based on the convolution layer in the preset VGG16 network, the first to-be-processed data is subjected to convolution processing, hierarchical features in each pooling layer in the VGG16 network are obtained, and a weight value is set for each hierarchical feature; Based on the weight values, feature fusion processing is performed on each hierarchical feature of the first data to be processed, and the features after the feature fusion processing are input into the fully connected layer in the preset VGG16 network.

3. The method according to claim 1, characterized in that The step of obtaining first data to be processed based on the facial visible light video includes: Acquire each image frame in the facial visible light video as a first image to be processed; Convert the RGB values ​​of the pixels in the first image to be processed into a color representation in a YCbCr space as a second image to be processed; Based on the second image to be processed and the openface open source toolkit, a facial motion unit intensity sequence, a gaze angle coordinate sequence and a head posture coordinate sequence are obtained.

4. The method according to claim 3, characterized in that After obtaining each image frame in the facial visible light video as a first image to be processed, the method further includes: Dividing a face region of interest in the first image to be processed based on the flooding method; The rPPG signal in the face region of interest is extracted based on a planar orthogonal skin algorithm.

5. A non-contact stress state recognition system, characterized in that: The system comprises: A first acquisition module is used to acquire facial visible light video; A second acquisition module is used to acquire first data to be processed based on the facial visible light video; wherein the first data to be processed includes: a facial motion unit intensity sequence, a gaze angle coordinate sequence, a head posture coordinate sequence and an rPPG signal; A convolution processing module is used to perform convolution processing of the first data to be processed in the spatial dimension and the temporal dimension based on a preset feature network to obtain the second data to be processed after the convolution processing; wherein the second data to be processed includes: the spatiotemporal feature sequence ST of the facial motion unit intensity sequence A , the spatiotemporal feature sequence ST of the gaze angle coordinate sequence E , the spatiotemporal feature sequence ST of the head posture coordinate sequence H and rPPG signal spatiotemporal feature sequence ST R ; The feature network consists of a VGG16 convolutional neural network and a long short-term memory network; The third acquisition module is used to obtain the value of the query vector Q, key K, value V, and gradient d based on the preset query vector Q, key K, value V, and gradient d. m And softmax function to obtain the cross attention weight vector W′; A weighted fusion module, used for performing weighted fusion on the second to-be-processed data based on the cross attention weight vector W′ to obtain a weighted fused feature descriptor Y′; The state recognition module is used to perform dimensionality reduction processing on the feature descriptor Y′, and to identify the stress state of the feature descriptor and HRV signal after dimensionality reduction processing based on a preset multi-classification activation function softmax to obtain a corresponding stress state level; wherein the HRV signal is obtained based on the rPPG signal.

6. The system according to claim 5, characterized in that The convolution processing module comprises: A feature fusion submodule, configured to perform convolution processing on the first data to be processed in the spatial dimension based on a preset VGG16 network, and perform convolution processing on the first data to be processed in the temporal dimension based on a preset long short-term memory network, and perform feature fusion on the convolution-processed data to obtain second data to be processed; The performing convolution processing on the first to-be-processed data in the spatial dimension based on the preset VGG16 network includes: Based on the convolution layer in the preset VGG16 network, the first to-be-processed data is subjected to convolution processing, hierarchical features in each pooling layer in the VGG16 network are obtained, and a weight value is set for each hierarchical feature; Based on the weight values, feature fusion processing is performed on each hierarchical feature of the first data to be processed, and the features after the feature fusion processing are input into the fully connected layer in the preset VGG16 network.

7. The system according to claim 5, characterized in that The second acquisition module includes: A first acquisition submodule is used to acquire each image frame in the facial visible light video as a first image to be processed; A first conversion submodule, used for converting the RGB values ​​of the pixels in the first image to be processed into a color representation in a YCbCr space as a second image to be processed; The second acquisition submodule is used to acquire a facial motion unit intensity sequence, a gaze angle coordinate sequence and a head posture coordinate sequence based on the second image to be processed and the openface open source toolkit.

8. The system according to claim 7, characterized in that The convolution processing module also includes: A first segmentation submodule is used for segmenting a facial region of interest in the first image to be processed based on a flooding method after obtaining each image frame in the facial visible light video as a first image to be processed; The first extraction submodule is used to extract the rPPG signal in the face region of interest based on a planar orthogonal skin algorithm.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the steps of any method described in claims 1-4 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the methods of claims 1-4 are implemented.

Images