A remote examination abnormal state prediction method, system, device and storage medium

Abstract

The present application discloses a method, system, device and storage medium for predicting abnormal states in remote examinations. The method comprises: obtaining the video resources of examinees collected by the RGB camera and the near-infrared camera in the remote examination system respectively and decomposing them in chronological order to obtain RGB multi-frame images and near-infrared multi-frame images of examinees; inputting the pre-processed RGB multi-frame images and near-infrared multi-frame images of examinees respectively into the trained fine expression recognition model and the facial remote blood pressure recognition model, respectively obtaining the facial fine expression recognition results and blood pressure recognition results of examinees in the examination room; judging and warning the abnormal state of examinees according to the facial fine expression and blood pressure recognition results. The present invention can accurately predict in real time whether examinees have abnormal state tendencies in the examination room by identifying and analyzing the fine expressions and facial remote blood pressure signals of examinees in remote examinations.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a method, system, device and storage medium for predicting abnormal conditions in remote examinations. Background Art

[0002] With the development of educational information technology, the form of examinations has gradually evolved from traditional offline examinations to remote examinations. Paperless remote examinations can reduce the impact of environmental factors such as space and time for invigilation. Affected by the epidemic, remote examinations have flourished for a time. The traditional offline method of relying on examination staff to implement invigilation cannot meet the requirements of remote examinations. At present, the remote examination platforms used by colleges and universities have launched invigilation modes, such as using cameras to capture fear and background monitoring of the status of candidates' answers. Some courses with higher invigilation requirements adopt a dual-camera operation method: while turning on the camera on the computer screen, students have to set up their mobile phones on the side and rear, and use the mobile phone camera to shoot simultaneously. Although remote invigilation has various settings for student cheating, it is difficult for the remote invigilation system to achieve ideal results in terms of accuracy and operability. Summary of the invention

[0003] In response to at least one defect or improvement need in the prior art, the present invention provides a method, system, device and storage medium for predicting abnormal conditions in remote examinations. By identifying and analyzing the examinee's subtle facial expressions and facial remote blood pressure signals in the remote examination, it is possible to accurately predict in real time whether the examinee has a tendency to be in an abnormal state in the examination room.

[0004] To achieve the above object, according to a first aspect of the present invention, a method for predicting abnormal conditions in a remote examination is provided, comprising:

[0005] The video resources of the examinee collected by the RGB camera and the near-infrared camera in the remote examination system are respectively obtained, and the video resources of the examinee are divided into multiple frame images in chronological order to obtain the RGB multi-frame images and the near-infrared multi-frame images of the examinee;

[0006] Input the examinee's RGB multi-frame images and near-infrared multi-frame images into the trained fine expression recognition model to obtain the examinee's facial fine expression recognition results in the examination room; input the examinee's RGB multi-frame images and near-infrared multi-frame images into the trained facial remote blood pressure recognition model to obtain the examinee's blood pressure recognition results in the examination room;

[0007] The abnormal condition of the examinee is judged and warned based on the examinee's subtle facial expressions and blood pressure recognition results.

[0008] Furthermore, the fine expression recognition model includes: a feature extraction and fusion module, a convolutional spatial Transformer, a temporal Transformer and a fully connected layer, the convolutional spatial Transformer includes N spatial encoders, the temporal Transformer includes M temporal encoders, and the spatial encoder and the temporal encoder are both composed of a multi-head attention mechanism;

[0009] The feature extraction and fusion module is used to extract features from RGB and near-infrared images respectively and perform feature fusion and input them into the convolutional space Transformer. The convolutional space Transformer is used to extract the examinee's facial spatial features from the input fusion features and input them into the temporal Transformer. The temporal Transformer uses the examinee's facial spatial features of all frames as input to generate discriminative feature representations and input them into the fully connected layer. The fully connected layer is used to obtain facial fine expression recognition results.

[0010] Furthermore, the convolutional space Transformer is used to implement the steps:

[0011] Extract the examinee’s face spatial features from the input fusion features for feature mapping, and flatten the obtained feature mapping into a one-dimensional sequence M f ;

[0012] According to M f Obtain the input embedding representation, and calculate the query vector, keyword vector, and value vector in each spatial encoder through the embedding representation of the previous layer;

[0013] According to the query vector and keyword vector, the self-attention coefficients of N spatial encoders are calculated. The weighted sum of the value vector is calculated using the self-attention coefficient of each attention head. The concatenation of the vectors of all attention heads is projected and operated through MLP to output the embedded representation.

[0014] The embedded representation Connected in the spatial level to generate feature map M r , calculate the feature embedding x′ of each frame t , x′ t =GMP(g(M r )), where g(·) represents the convolutional layer and GMP stands for global maximum pooling.

[0015] Furthermore, the temporal Transformer is used to implement the steps:

[0016] Obtain input embedding representation based on the examinee’s face spatial features;

[0017] The query vector, keyword vector, and value vector in each temporal encoder are calculated by the previous layer of embedding representation;

[0018] According to the query vector and keyword vector, the self-attention coefficient of each of the M temporal encoders is calculated. The weighted sum of the value vector is calculated using the self-attention coefficient of each attention head. The connection of the vectors of all attention heads is projected and operated through MLP to output the discriminative feature representation.

[0019] Furthermore, the facial remote blood pressure recognition model includes a feature extraction and fusion module, a blood pressure signal extraction module and a blood pressure signal processing module. The feature extraction and fusion module includes a module for extracting feature maps from the pre-processed RGB multi-frame images and near-infrared multi-frame images of the examinee and performing feature fusion and then inputting the features into the blood pressure signal extraction module. The blood pressure signal extraction module is used to extract facial color change features as blood pressure signals and input them into the blood pressure signal processing module; the blood pressure signal processing module is used to suppress interference caused by facial movement.

[0020] Furthermore, the feature extraction and fusion module includes a convolution layer, an average pooling layer and a feature fusion layer, the convolution layer is a 2D convolution layer or a 3D convolution layer, and the feature maps are extracted from the preprocessed RGB multi-frame images and near-infrared multi-frame images of the examinee through the convolution layer and then input into the average pooling layer, and the feature maps are input into the feature fusion layer after the average pooling operation is performed on the feature maps through the average pooling layer, and the feature fusion layer merges the feature maps into short-segment feature maps.

[0021] Furthermore, the blood pressure signal processing module includes a spatiotemporal attention module, an average pooling module and a 3D convolution module, the spatiotemporal attention module includes a spatiotemporal strip aggregation module and a spatiotemporal convolution block, and the blood pressure signal processing module is used to implement the steps:

[0022] The blood pressure signal is input into the spatiotemporal attention module to obtain the strip convergence of the input blood pressure signal in the horizontal, vertical and time dimensions in three-dimensional space;

[0023] The current position and neighboring features are adjusted through the spatiotemporal convolution block to obtain y v ,y h ,y K , and finally output Z, the calculation formula is as follows:

[0024]

[0025] Z=σ(f(Y,ω))

[0026] Where f(Y,ω) represents the spatiotemporal convolution block represented by parameter ω, σ represents the sigmoid function, and Y c,i,j,tIt means that the space-time strips of c channel converge along the horizontal, vertical and time directions. Indicates that the c channel converges along the horizontal strip. Indicates that the c channel converges along the vertical strip. represents the strip convergence of the c channel along the time direction, and Z represents the nonlinear normalization of the strip convergence;

[0027] The tensor Z generates a spatial information descriptor F through average pooling avg , the calculation formula is as follows:

[0028] F avg =σ(g avg (Z)

[0029] The spatial information descriptor F generated by average pooling avg Perform Hadamard product with tensor Z and finally output X′;

[0030] After average pooling, X′ is convolved with the 3D convolution module to output the predicted DBPG signal.

[0031] According to a second aspect of the present invention, there is also provided a remote examination abnormal state prediction system, comprising:

[0032] The acquisition module is used to obtain the candidate video resources collected by the RGB camera and the near-infrared camera in the remote examination system, and divide the candidate video resources into multiple frame images in chronological order to obtain the candidate RGB multi-frame images and near-infrared multi-frame images;

[0033] A recognition module is used to input the examinee's RGB multi-frame images and near-infrared multi-frame images into the trained fine expression recognition model to obtain the examinee's facial fine expression recognition results in the examination room, and input the examinee's RGB multi-frame images and near-infrared multi-frame images into the trained facial remote blood pressure recognition model to obtain the examinee's blood pressure recognition results in the examination room;

[0034] The judgment and warning module is used to judge and warn the abnormal status of the examinee based on the examinee's facial expressions and blood pressure recognition results.

[0035] According to the third aspect of the present invention, there is also provided an electronic device, comprising at least one processor and at least one storage module, wherein the storage module stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the above methods.

[0036] According to a fourth aspect of the present invention, there is also provided a storage medium storing a computer program executable by a processor, wherein when the computer program runs on the processor, the processor executes the steps of any one of the above methods.

[0037] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0038] (1) The present invention collects video resources of examinees collected by RGB cameras and near-infrared cameras in the remote examination system, recognizes and analyzes the examinees' subtle expressions and facial remote blood pressure signals in the remote examination, and can accurately predict in real time whether the examinees have abnormal state tendencies in the examination room.

[0039] (2) The examinee's fine expression recognition model provided by the present invention can extract more discriminative fine expression features from the spatial and temporal perspectives. This method can effectively handle the problems of non-frontal head posture, head movement, examination room lighting, etc. among examinees and achieve a robust effect.

[0040] (3) The facial remote blood pressure recognition model provided by the present invention can effectively reduce redundant information and enhance the correlation of long-distance videos. The spatiotemporal strip pooling designed in the spatiotemporal module can effectively reduce head motion noise. Finally, the designed loss function can make the model pay more attention to the DBPG (Deep Cross-Space Blood Pressure Signal) signal rather than interference information, which can effectively improve the accuracy of facial remote blood pressure recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a flow chart of a method for predicting abnormal conditions of a remote examination according to an embodiment of the present invention;

[0043] Figure 2 Schematic diagram of the principle of the method for predicting abnormal state of remote examination according to an embodiment of the present invention;

[0044] Figure 3 is a network diagram of a fine expression recognition model according to an embodiment of the present invention;

[0045] Figure 4 4 is a network diagram of a facial remote blood pressure recognition model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0047] The terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or modules is not limited to the listed steps or modules, but may optionally include steps or modules that are not listed, or may optionally include other steps or modules that are inherent to these processes, methods, products or devices.

[0048] like Figure 1 and Figure 2 As shown, the method for predicting abnormal state of remote examination according to the embodiment of the present invention comprises the steps of:

[0049] S101, respectively obtain the examinee video resources collected by the RGB camera and the near infrared (NIR) camera in the remote examination system, and divide the examinee video resources into multiple frame images in chronological order to obtain the examinee RGB multi-frame images and near infrared multi-frame images.

[0050] RGB cameras and near-infrared cameras are deployed in the examination room. The examinee is taking a remote examination using the remote examination equipment. In this scenario, the examinee's facial video sequence is collected using RGB and near-infrared binocular video acquisition instruments. The RGB and near-infrared examinee video images collected in this scenario provide an important data source for the fine expression recognition module and the facial remote blood pressure recognition module.

[0051] The candidate's video resources are divided into multiple frame images in chronological order to obtain the candidate's RGB multi-frame image sequence and near-infrared multi-frame image sequence.

[0052] Furthermore, before being input into step S102 for processing, the examinee's RGB multi-frame images and near-infrared multi-frame images are pre-processed respectively.

[0053] The preprocessed images are divided into training sets and test sets in proportion to train and test the facial fine expression recognition model and the facial remote blood pressure recognition model.

[0054] S102, input the examinee's RGB multi-frame images and near-infrared multi-frame images into the trained fine expression recognition model to obtain the examinee's facial fine expression recognition results in the examination room, input the examinee's RGB multi-frame images and near-infrared multi-frame images into the trained facial remote blood pressure recognition model to obtain the examinee's blood pressure recognition results in the examination room.

[0055] like Figure 3 As shown, in this embodiment, the fine expression recognition model includes a feature extraction and fusion module, a convolutional spatial transformer, a temporal transformer and a fully connected layer. The convolutional spatial transformer (FER-S-Former) includes feature mapping, spatial position embedding, and N spatial encoders; the temporal transformer (FER-T-Former) includes feature mapping, temporal position embedding, and M temporal encoders; the spatial encoder and the temporal encoder are both composed of a multi-head attention mechanism and are input into the feedforward network. The convolutional spatial transformer is used to extract the spatial features of the examinee's face by taking each frame of the examinee's image as input; the temporal transformer takes the spatial features of the examinee's face in all frames as input to generate discriminative feature representations; the fully connected layer is used to identify the final fine expression results.

[0056] The feature extraction and fusion process is as follows: the collected RGB candidate video image is R = [r 1 ,r 2 ,…,r t ], the collected NIR near infrared video images of candidates are N = [n 1 ,n 2 ,…,n t ], the t-frame images of the RGB and NIR near-infrared candidate videos are taken as input, and the features of facial expressions are extracted through convolution and pooling operations, and then the features are fused and expressed as A = [a 1 ,a 2 ,…,a n ].

[0057] The data processing process of the convolutional space Transformer is as follows:

[0058] (1) From a spatial perspective, the fine expression features of the examinee’s facial image in the examination room can be modeled as a series of vector representations: A = [a 1 ,a 2 ,…,a n ]

[0059] (2) Extract the candidate's facial spatial features and perform feature mapping. The convolutional spatial transformer (FER-S-Former) flattens the candidate's fine expression spatial features after feature fusion into a one-dimensional sequence Mf ∈R Q×C (Q = H' × W'), and further input to FER-S-Former.

[0060] (3) Compute the query, keyword, and value vectors in each spatial encoder. The input embedding representation can be calculated as where e p represents a learnable position embedding, represents the corresponding visual word embedding, Represents the input embedding of the spatial encoder. The query vector for the k-head attention of the l-th layer in each spatial encoder is calculated as The keyword vector of the k-head attention at layer l is The value vector of the k-head attention at layer l is where LN(·) represents layer normalization, k∈{1,...K} represents the number of attention heads, represents the potential dimension of each attention head, represents the query vector of the k-head attention at layer l, The parameter matrix representing the query vector of the k-head attention at layer l, represents the keyword vector of the k-head attention at layer l, The parameter matrix representing the keyword vector of the k-head attention at layer l, represents the value vector of the k-head attention of the l-th layer, Parameter matrix representing the value vector of the k-head attention at layer l.

[0061] (4) Self-attention weight For each query p, the dot product is calculated as Where SM represents the softmax activation function, Represents the self-attention weight of the k-head attention of the l-th layer. In order to calculate the embedding representation of the l-th layer First, use the self-attention coefficient of each attention head to calculate the weighted sum of the value vector, that is, represents the weighted sum of the value vectors of the k-head attentions at layer l, represents the self-attention weight coefficient corresponding to the value vector in each attention head of the lth layer; then the connection of the vectors of all attention heads is projected and operated through the MLP, which can be expressed as Where W o represents the learnable parameter matrix.

[0062] (5) Embedded representation of Q Connected in the spatial level to generate a more refined feature map M r, and the feature embedding x′ of each frame t can be calculated as x′ t =GMP(g(M r )), where g(·) represents the convolutional layer and GMP stands for global maximum pooling.

[0063] The data processing process of the temporal Transformer model is as follows:

[0064] (1) The output of the convolutional spatial Transformer is used as the input of the temporal Transformer model. Given an input X′∈R T×F , the input embedding representation of the temporal Transformer can be calculated as where e t′ ∈R F represents a learnable positional embedding representation that encodes temporal position, represents the input embedding of the temporal Transformer, x′ t ′ represents the feature embedding of each frame. Unlike the spatial encoder, a learnable vector x′ is added at the first position of the sequence 0 Token representing the classification (embedded representation of the class).

[0065] (2) For the temporal Transformer, the previous layer embedding representation The query vector for the k-head attention at the l-th layer in each temporal encoder is calculated as The keyword vector of the k-head attention at layer l is The value vector of the k-head attention at layer l is In the formula, represents the query vector of the k-head attention at layer l, The parameter matrix representing the query vector of the k-head attention at layer l, represents the keyword vector of the k-head attention at layer l, The parameter matrix representing the keyword vector of the k-head attention at layer l, represents the value vector of the k-head attention of the l-th layer, Parameter matrix representing the value vector of the k-head attention at layer l.

[0066] (3) The self-attention coefficients of the M temporal encoders are calculated based on the query vector and the keyword vector. The weighted sum of the value vector is calculated using the self-attention coefficient of each attention head. The concatenation of the vectors of all attention heads is projected and operated through the MLP to output the discriminative feature representation.

[0067] For each query vector the self-attention weight can be calculated as:

[0068]

[0069] In the formula, represents the self-attention weight of the k-head attention head in the l-th layer, and F′ represents the potential dimension of each attention head, that is, F′=F / K.

[0070] The embedding representation at layer l can be calculated as

[0071]

[0072] In the formula, represents the embedding representation at layer l, represents the self-attention weight coefficient corresponding to the value vector in each attention head of layer l, represents the self-attention weight coefficient corresponding to the learnable value vector, Represents the encoding at layer l.

[0073] The fully connected layer calculates the final recognition result. The final clip embedding is obtained from the last layer of the temporal transformer, the token class, and the final recognition result can be calculated as Among them, FC represents the fully connected layer network, J represents the category of expression recognition, Represents the class embedding representation of the last layer of the M temporal encoders. The categories of expression recognition are divided into three categories: positive, neutral, and negative.

[0074] like Figure 4 As shown, the facial remote blood pressure recognition model includes a feature extraction and fusion module, a blood pressure signal extraction module (Deep Cross-Space Blood Pressure Signal, DBPG) and a blood pressure signal processing module (DBPG signal processing module). The feature extraction and fusion module includes a module for extracting feature maps from the pre-processed RGB multi-frame images and near-infrared multi-frame images of the examinee and performing feature fusion and inputting them into the blood pressure signal extraction module. The DBPG signal extraction module is used to extract facial color changes from feature fusion as deep blood pressure signals and input them into the DBPG signal processing module. The DBPG signal processing module is used to avoid ignoring important local information and prevent interference from related areas under motion conditions. The DBPG signal extraction module consists of a spatiotemporal block (ST Block). The DBPG signal processing module includes a spatiotemporal attention module, an average pooling module and a 3D convolution module, and the spatiotemporal attention module includes a spatiotemporal strip aggregation module and a spatiotemporal convolution block.

[0075] The feature extraction and fusion module includes a convolution layer, an average pooling layer and a feature fusion layer. The convolution layer is a 2D convolution layer or a 3D convolution layer. The feature maps are extracted from the preprocessed RGB multi-frame images and near-infrared multi-frame images of the candidates through the convolution layer and then input into the average pooling layer. After the average pooling operation is performed on the feature maps, they are input into the feature fusion layer. The feature fusion layer merges the feature maps into short-segment feature maps.

[0076] The data processing steps of the feature extraction and fusion module are as follows:

[0077] Step 3.3.1: Separately transform the preprocessed candidate RGB video sequence V∈R ′×L×H×W , NIR near infrared video sequence N∈R C×L×H×W As input tensor, where C, L, H, W represent the number of channels, video length, height and width respectively;

[0078] Step 3.3.2: Extract feature maps. Extract feature maps X through 3DCNN with two layers of [3,3,3] convolution kernels conv3D ∈R C′×L×H×W , C′ represents the number of channels of the feature map. 3DCNN can enhance learning information in the time dimension. Feature maps can also be obtained through 2DCNN with two layers of [3,3] convolution kernels. 2DCNN focuses more on the video sequence V∈R B×C×H×W Extract facial features from , where B is the batch size, which is numerically equal to the video length;

[0079] Step 3.3.3: After average pooling, use the feature fusion function to merge the feature maps into spatiotemporal feature maps Where T is the number of short video sequences. The calculation formula is: where g avg (·) represents the average pooling function, The function of 3DCNN or 2DCNN is represented by the convolution filter parameter ω, and f represents the feature fusion function;

[0080] The data processing steps of the DBPG signal extraction module are as follows:

[0081] (1) Map the spatiotemporal features obtained in the face feature extraction module Input into DBPG signal extraction module;

[0082] (2) Use three spatiotemporal blocks (ST Block) to extract DBPG signals. ST Block consists of two sets of cascaded spatiotemporal convolutions (ST Conv) and an average pooling layer; ST Conv consists of two sets of cascaded 3D convolution filters with kernel sizes of [1,3,3] and [3,1,1] respectively. ST Conv can significantly reduce the number of parameters of the convolution filter. In addition, ST Conv can also encapsulate information related to objects, scenes, and motions in the video. The key formula for DBPG signal extraction is as follows:

[0083] I k =h(X+∑ k (1+a k )δ k ) (11)

[0084] Among them, I k represents the DBPG signal extracted from the kth spatiotemporal block, X represents the spatiotemporal feature map, k is the number of spatiotemporal blocks, and a k represents the scaling factor of the kth spatiotemporal block, δ k represents the displacement parameter of the kth spatiotemporal block, and h represents the signal feature extraction function.

[0085] The data processing steps of the DBPG signal processing module are as follows:

[0086] (1) Extract the DBPG signal I k ∈R C×T×H×W The tensor is input into the spatiotemporal attention module, which includes spatiotemporal strip convergence and spatiotemporal convolution (ST Conv). Spatiotemporal strip convergence is used to generate the convergence of DBPG signals in the horizontal, temporal and vertical directions in three-dimensional space. The convergence along the vertical strip generated by spatiotemporal strip convergence is Similarly, the strip convergence calculation along the horizontal direction and time direction is:

[0087] (2) Adjust the current position and adjacent features through ST Conv to obtain y v ∈R C×W ,y h ∈R C×H ,y K ∈R C×T , and finally output Z∈R C×T×H×W , the key equation of spatiotemporal strip pooling is as follows:

[0088]

[0089] Z=σ(f(Y,ω)) (13)

[0090] Where f(Y,ω) represents the ST Conv represented by the parameter ω, σ represents the sigmoid function, and Y c,i,j,t It means that the space-time strips of c channel converge along the horizontal, vertical and time directions. Indicates that the c channel converges along the horizontal strip. Indicates that the c channel converges along the vertical strip. represents the strip convergence of the c channel along the time direction, and Z represents the nonlinear normalization of the strip convergence.

[0091] (3) The tensor Z generates a spatial information descriptor F through average pooling avg , the calculation formula is as follows:

[0092] F avg =σ(g avg (Z)) (14)

[0093] (4) Spatial information descriptor F generated by average pooling avg Perform Hadamard product with tensor Z and finally output X′∈R C×T×H×W , the calculation formula is as follows:

[0094] X′=Z⊙F avg (15)

[0095] (5) X′ outputs the predicted DBPG signal after global average pooling and convolution with a 3D convolution module with [1,1,1].

[0096] In order to minimize the position error between the estimated signal peak and the true value signal peak, the negative Pearson correlation coefficient is used as the loss function, and the formula is as follows:

[0097]

[0098] Where L represents the length of the signal, x is the ground truth DBPG signal, and y is the predicted DBPG signal.

[0099] The loss of the facial remote blood pressure recognition model is divided into two parts. Loss is generated in the DBPG signal extraction module. pre , Loss pre The purpose is to pre-guide the network to learn DBPG signals instead of other physiological signals; Loss is generated in the DBPG signal processing module post Used to make the learned DBPG signal as accurate as possible, thereby improving the periodicity of the restored DBPG signal. post 、Loss pre All are calculated by formula (16). Through the two-part loss, the model can pay more attention to the DBPG signal, and the total loss function is L:

[0100] L=λ*Loss pre +(1-λ)*Loss post (17)

[0101] Where λ∈[0,1] is used to adjust the importance of the two parts of the loss.

[0102] S103, judging and issuing warnings on the abnormal state of the examinee based on the examinee's facial expressions and blood pressure recognition results.

[0103] According to the facial fine expression and facial remote blood pressure recognition results of the examinee at different times, feature fusion is performed to comprehensively judge whether it is greater than the threshold f. If it is greater than the threshold f, the examinee has abnormal behavior. If it is less than the threshold f, the examinee is in a normal state. The feature fusion rules are as follows:

[0104]

[0105] Based on the abnormal behavior judgment results, the examination staff is warned of the abnormal behavior of the candidates, and the examination staff can take corresponding intervention measures to prevent the occurrence of abnormal behavior.

[0106] A remote examination abnormal state prediction system according to an embodiment of the present invention includes:

[0107] The acquisition module is used to obtain the candidate video resources collected by the RGB camera and the near-infrared camera in the remote examination system, and divide the candidate video resources into multiple frame images in chronological order to obtain the candidate RGB multi-frame images and near-infrared multi-frame images;

[0108] A recognition module is used to input the examinee's RGB multi-frame images and near-infrared multi-frame images into the trained fine expression recognition model to obtain the examinee's facial fine expression recognition results in the examination room, and input the examinee's RGB multi-frame images and near-infrared multi-frame images into the trained facial remote blood pressure recognition model to obtain the examinee's blood pressure recognition results in the examination room;

[0109] The judgment and warning module is used to judge and warn the abnormal status of the examinee based on the examinee's facial expressions and blood pressure recognition results.

[0110] The implementation method of the system is the same as the above method and will not be repeated here.

[0111] This embodiment also provides an electronic device, which includes at least one processor and at least one memory, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the processor executes any step of the above-mentioned remote examination abnormal state prediction method. For specific steps, please refer to the method embodiment and will not be repeated here. In this embodiment, the types of processor and memory are not specifically limited. For example, the processor can be a microprocessor, a digital information processor, an on-chip programmable logic system, etc.; the memory can be a volatile memory, a non-volatile memory, or a combination thereof.

[0112] The present application also provides a storage medium storing a computer program executable by a processor, and when the computer program is executed on the processor, the processor executes any of the steps of the above-mentioned remote examination abnormal state prediction method. The computer-readable storage medium may include, but is not limited to, any type of disk, including a floppy disk, an optical disk, a DVD, a CD-ROM, a micro drive, and a magneto-optical disk, a ROM, a RAM, an EPROM, an EEPROM, a DRAM, a VRAM, a flash memory device, a magnetic card or an optical card, a nanosystem (including a molecular memory IC), or any type of medium or device suitable for storing instructions and / or data.

[0113] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0114] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0115] In the several embodiments provided in the present application, it should be understood that the disclosed system can be implemented in other ways. For example, the system embodiments described above are only schematic, such as the division of the modules, which is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, indirect coupling or communication connection of systems or modules, which can be electrical or other forms.

[0116] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0117] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0118] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, disk or optical disk and other media that can store program code.

[0119] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by entering a program to instruct the relevant hardware. The program may be stored in a computer-readable memory, and the memory may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0120] The above is only an exemplary embodiment of the present disclosure, and the scope of the present disclosure cannot be limited thereto. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the specification and practicing the disclosure here, those skilled in the art will easily think of the implementation scheme of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the technical field not recorded in the present disclosure. The description and examples are regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

[0121] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for predicting abnormal conditions in remote examinations, characterized in that: include: The video resources of the examinee collected by the RGB camera and the near-infrared camera in the remote examination system are respectively obtained, and the video resources of the examinee are divided into multiple frame images in chronological order to obtain the RGB multi-frame images and the near-infrared multi-frame images of the examinee; Input the examinee's RGB multi-frame images and near-infrared multi-frame images into the trained fine expression recognition model to obtain the examinee's facial fine expression recognition results in the examination room; input the examinee's RGB multi-frame images and near-infrared multi-frame images into the trained facial remote blood pressure recognition model to obtain the examinee's blood pressure recognition results in the examination room; According to the examinee's facial expression and blood pressure recognition results, the examinee's abnormal state is judged and warned; The fine expression recognition model includes: a feature extraction and fusion module, a convolutional spatial transformer, a temporal transformer and a fully connected layer, wherein the convolutional spatial transformer includes N spatial encoders, the temporal transformer includes M temporal encoders, and the spatial encoder and the temporal encoder are both composed of a multi-head attention mechanism; The feature extraction and fusion module is used to extract features from RGB and near-infrared images respectively and perform feature fusion and input them into the convolutional space Transformer. The convolutional space Transformer is used to extract the face spatial features of the examinee from the input fusion features and input them into the temporal Transformer. The temporal Transformer uses the face spatial features of the examinee of all frames as input to generate discriminant feature representation and input them into the fully connected layer. The fully connected layer is used to obtain the facial fine expression recognition results. The convolutional spatial Transformer is used to implement the steps Extract the examinee's facial spatial features from the input fusion features for feature mapping, and flatten the obtained feature mapping into a one-dimensional sequence ;according to Get the input embedding representation, calculate the query vector, keyword vector and value vector in each spatial encoder through the embedding representation of the previous layer; calculate the self-attention coefficient of each N spatial encoder according to the query vector and keyword vector, use the self-attention coefficient of each attention head to calculate the weighted sum of the value vector, then project the connection of the vectors of all attention heads and pass the MLP operation to output the embedding representation ; embedding representation Connected in the spatial level to generate feature maps , calculate the feature embedding of each frame , ,in represents the convolutional layer, GMP stands for global maximum pooling; The facial remote blood pressure recognition model includes a feature extraction and fusion module, a blood pressure signal extraction module and a blood pressure signal processing module. The feature extraction and fusion module includes a module for extracting feature maps from the pre-processed RGB multi-frame images and near-infrared multi-frame images of the examinee and performing feature fusion before inputting the feature maps into the blood pressure signal extraction module. The blood pressure signal extraction module is used to extract facial color change features as blood pressure signals and input them into the blood pressure signal processing module. The blood pressure signal processing module is used to suppress interference caused by facial movement. The blood pressure signal processing module includes a spatiotemporal attention module, an average pooling module and a 3D convolution module. The spatiotemporal attention module includes a spatiotemporal strip aggregation module and a spatiotemporal convolution block. The blood pressure signal processing module is used to implement the steps: The blood pressure signal is input into the spatiotemporal attention module to obtain the strip convergence of the input blood pressure signal in the horizontal, vertical and time dimensions in three-dimensional space; The current position and neighboring features are adjusted through the spatiotemporal convolution block to obtain , , , and finally output , the calculation formula is as follows: in, Indicates the use of parameters Represented by the spatiotemporal convolutional block, represents the sigmoid function, express c The channels converge along the space-time strips in the horizontal, vertical and time directions. express c The channels converge along horizontal strips. express c The channels converge along vertical strips, express c The channels converge in strips along the time direction, represents the nonlinear normalization of the band convergence; Tensor Z generates spatial information descriptor through average pooling , the calculation formula is as follows: Spatial information descriptor generated by average pooling Perform Hadamard product with tensor Z and finally output ; After average pooling, it is convolved with the 3D convolution module to output the predicted DBPG signal.

2. The method for predicting abnormal conditions in a remote examination as claimed in claim 1, characterized in that: The temporal Transformer is used to implement the following steps: Obtain input embedding representation based on the examinee’s face spatial features; The query vector, keyword vector, and value vector in each temporal encoder are calculated by the previous layer of embedding representation; According to the query vector and keyword vector, the self-attention coefficient of each of the M temporal encoders is calculated. The weighted sum of the value vector is calculated using the self-attention coefficient of each attention head. The connection of the vectors of all attention heads is projected and operated through MLP to output the discriminative feature representation.

3. The method for predicting abnormal conditions in a remote examination as claimed in claim 1, characterized in that: The feature extraction and fusion module includes a convolution layer, an average pooling layer and a feature fusion layer. The convolution layer is a 2D convolution layer or a 3D convolution layer. The feature maps are extracted from the preprocessed RGB multi-frame images and near-infrared multi-frame images of the examinee through the convolution layer and then input into the average pooling layer. After the average pooling operation is performed on the feature maps, the average pooling layer inputs them into the feature fusion layer. The feature fusion layer merges the feature maps into short-segment feature maps.

4. A remote examination abnormal state prediction system, characterized in that: include: The acquisition module is used to obtain the candidate video resources collected by the RGB camera and the near-infrared camera in the remote examination system, and divide the candidate video resources into multiple frame images in chronological order to obtain the candidate RGB multi-frame images and near-infrared multi-frame images; A recognition module is used to input the examinee's RGB multi-frame images and near-infrared multi-frame images into the trained fine expression recognition model to obtain the examinee's facial fine expression recognition results in the examination room, and input the examinee's RGB multi-frame images and near-infrared multi-frame images into the trained facial remote blood pressure recognition model to obtain the examinee's blood pressure recognition results in the examination room; The judgment and warning module is used to judge and warn the abnormal status of the examinee based on the examinee's facial expression and blood pressure recognition results; The fine expression recognition model includes: a feature extraction and fusion module, a convolutional spatial transformer, a temporal transformer and a fully connected layer, wherein the convolutional spatial transformer includes N spatial encoders, the temporal transformer includes M temporal encoders, and the spatial encoder and the temporal encoder are both composed of a multi-head attention mechanism; The feature extraction and fusion module is used to extract features from RGB and near-infrared images respectively and perform feature fusion and input them into the convolutional space Transformer. The convolutional space Transformer is used to extract the face spatial features of the examinee from the input fusion features and input them into the temporal Transformer. The temporal Transformer uses the face spatial features of the examinee of all frames as input to generate discriminant feature representation and input them into the fully connected layer. The fully connected layer is used to obtain the facial fine expression recognition results. The convolutional spatial transformer is used to implement the steps of: extracting the examinee's face spatial features from the input fusion features for feature mapping, and flattening the acquired feature mapping into a one-dimensional sequence ;according to Get the input embedding representation, calculate the query vector, keyword vector and value vector in each spatial encoder through the embedding representation of the previous layer; calculate the self-attention coefficient of each N spatial encoder according to the query vector and keyword vector, use the self-attention coefficient of each attention head to calculate the weighted sum of the value vector, then project the connection of the vectors of all attention heads and pass the MLP operation to output the embedding representation ; embedding representation Connected in the spatial level to generate feature maps , calculate the feature embedding of each frame , ,in represents the convolutional layer, GMP stands for global maximum pooling; The facial remote blood pressure recognition model includes a feature extraction and fusion module, a blood pressure signal extraction module and a blood pressure signal processing module. The feature extraction and fusion module includes a module for extracting feature maps from the pre-processed RGB multi-frame images and near-infrared multi-frame images of the examinee and performing feature fusion before inputting the feature maps into the blood pressure signal extraction module. The blood pressure signal extraction module is used to extract facial color change features as blood pressure signals and input them into the blood pressure signal processing module. The blood pressure signal processing module is used to suppress interference caused by facial movement. The blood pressure signal processing module includes a spatiotemporal attention module, an average pooling module and a 3D convolution module. The spatiotemporal attention module includes a spatiotemporal strip aggregation module and a spatiotemporal convolution block. The blood pressure signal processing module is used to implement the steps: The blood pressure signal is input into the spatiotemporal attention module to obtain the strip convergence of the input blood pressure signal in the horizontal, vertical and time dimensions in three-dimensional space; The current position and neighboring features are adjusted through the spatiotemporal convolution block to obtain , , , and finally output , the calculation formula is as follows: in, Indicates the use of parameters Represented by the spatiotemporal convolutional block, represents the sigmoid function, express c The channels converge along the space-time strips in the horizontal, vertical and time directions. express c The channels converge along horizontal strips. express c The channels converge along vertical strips, express c The channels converge in strips along the time direction, represents the nonlinear normalization of the band convergence; Tensor Z generates spatial information descriptor through average pooling , the calculation formula is as follows: Spatial information descriptor generated by average pooling Perform Hadamard product with tensor Z and finally output ; After average pooling, it is convolved with the 3D convolution module to output the predicted DBPG signal.

5. An electronic device, characterized in that: The method comprises at least one processor and at least one storage module, wherein the storage module stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 3.

6. A storage medium, characterized in that: The device stores a computer program, and when the computer program is run on a processor, the processor is enabled to execute the steps of the method according to any one of claims 1 to 3.

Images