Examination room abnormal action analysis method and system based on feature fusion

Abstract

The invention discloses an examination room abnormal action analysis method and system based on feature fusion, and the method comprises the steps: obtaining an examination room monitoring video to be analyzed; inputting an examination room monitoring video to be analyzed into the trained examination room abnormal action analysis model to obtain an examination room abnormal action analysis result; wherein the trained examination room abnormal action analysis model is used for extracting primary characteristics of an examination room monitoring video to be analyzed, extracting environment characteristics and action characteristics from the primary characteristics respectively, fusing the environment characteristics and the action characteristics to obtain fused characteristics, and extracting the fused characteristics to obtain final characteristics; and classifying and identifying the final features to obtain an examination room abnormal action analysis result. And better effects are achieved in the aspects of identification accuracy and robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of motion recognition, and in particular to a method and system for analyzing abnormal motions in an examination room based on feature fusion. Background Art

[0002] Action recognition is an important research field in computer vision. It focuses on enabling computers to understand actions in videos or image sequences through algorithms. This involves identifying, classifying, and locating human behavior in videos. Action recognition technology has a wide range of applications in many fields, such as intelligent monitoring systems, human-computer interaction, virtual reality, sports analysis, medical health, etc.

[0003] Before the rise of deep learning, action recognition mainly relied on hand-crafted features to describe actions in videos. Hand-crafted features refer to a series of feature descriptors carefully designed by researchers based on task requirements and prior knowledge. These feature descriptors can capture key information in videos, such as the appearance, shape, texture of objects, and their motion characteristics. In recent years, deep learning methods have made significant progress in the field of video action recognition, improving the performance and practicality of the system.

[0004] A few years ago, a deep learning model called SlowFast network was proposed in the field of action recognition, which combines long-term and short-term features in videos. By introducing a multi-path mechanism to enhance the model's temporal perception, SlowFast network can effectively capture action features of different time scales in videos, thus achieving remarkable results in the field of action recognition.

[0005] On the other hand, the Transformer model has been applied to video processing tasks such as frame synthesis, action recognition, and video retrieval due to its advantages in sequence data processing. These tasks usually require processing the spatiotemporal dimension information of the video, and the Transformer model can adapt well to such tasks.

[0006] Although the action recognition method based on deep learning has achieved remarkable results, it still has some shortcomings: the model mainly performs dependency calculations on the spatial and temporal dimensions of the video data respectively, and both time and space are calculated by sampling discrete frames, while the action is performed continuously. The above approach has limited ability to extract continuous information of the action and ignores global information; video data sets are difficult to annotate, and insufficient data volume will lead to poor recognition effect of the model and poor generalization ability. Summary of the invention

[0007] Although previous deep learning-based action recognition methods have achieved remarkable results, there are still some shortcomings, such as the limited ability of the model to extract continuous information about actions, insufficient video data volume leading to poor actual recognition results of the model, etc. In order to address the shortcomings of the prior art, the present invention provides an abnormal action analysis method and system for examination rooms based on feature fusion; the action recognition model of the present invention is also based on a deep learning method, which has achieved better results in both recognition accuracy and robustness.

[0008] On the one hand, a method for analyzing abnormal movements in the examination room based on feature fusion is provided, including:

[0009] Obtain the examination room surveillance video to be analyzed;

[0010] The examination room surveillance video to be analyzed is input into the trained examination room abnormal motion analysis model to obtain the examination room abnormal motion analysis results; wherein the trained examination room abnormal motion analysis model is used to extract primary features from the examination room surveillance video to be analyzed, extract environmental features and motion features from the primary features respectively, fuse the environmental features and motion features to obtain fused features, extract features from the fused features to obtain final features, classify and identify the final features to obtain the examination room abnormal motion analysis results.

[0011] On the other hand, a feature fusion-based abnormal motion analysis system for the examination room is provided, including:

[0012] An acquisition module is configured to: acquire the examination room surveillance video to be analyzed;

[0013] The analysis module is configured as follows: inputting the examination room surveillance video to be analyzed into the trained examination room abnormal action analysis model to obtain the examination room abnormal action analysis results; wherein the trained examination room abnormal action analysis model is used to extract primary features from the examination room surveillance video to be analyzed, extract environmental features and action features from the primary features respectively, fuse the environmental features and action features to obtain fused features, extract features from the fused features to obtain final features, classify and identify the final features to obtain the examination room abnormal action analysis results.

[0014] The above technical solution has the following advantages or beneficial effects:

[0015] 1. This invention innovatively designs a new feature extraction neural network ConFrames, and establishes two channels to collect continuous features and discrete features respectively. Continuous features are used to represent action features, and discrete features are used to represent environmental features. Then, they are fused through the feature fusion module, and then the fused features are passed through Moformer again to obtain richer video spatiotemporal feature information.

[0016] 2. This paper innovatively designs a lightweight transformer structure Moformer, which has three variants, namely Moformer_E, MoAformer_A and Moformer, which are used for environmental feature extraction, action feature extraction and feature further processing respectively. They all consist of two repeated structures, a corresponding block and a convolutional feedforward network, but they differ in details.

[0017] 3. The present invention innovatively designs a feature fusion module FuseFrames to fuse environmental features into action features. Since the feature dimensions of the two branches are different, the FuseFrames module first uses convolution to align the dimensions, and then uses the channel attention mechanism to help the model better capture key information, thereby enhancing the feature expression capability and improving the model robustness.

[0018] 4. The present invention integrates context awareness into the self-attention mechanism. A dual-branch structure is adopted, and context-aware weights are fused in the local branch to aggregate high-frequency local information. For the global branch, ordinary self-attention is used, but K and V are downsampled to reduce FLOPs, which helps the model capture low-frequency global information. Then, the outputs of the local branch and the global branch are fused.

[0019] 5. The present invention uses a pre-training-fine-tuning paradigm, first using a public dataset model for training, then using an internal dataset for training, and fine-tuning the parameters in the network that work better, thereby achieving good recognition performance and reducing the difficulty of video dataset labeling and the impact of insufficient data on the model.

[0020] 6. The present invention uses a weighted cross entropy loss function instead of the ordinary cross entropy loss function, changes the weight of the loss function according to the frequency of actions in the examination room, and improves the accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0022] Figure 1 is a flow chart of the method of embodiment 1;

[0023] Figure 2 The overall architecture of the abnormal motion analysis model in the examination room of the first embodiment;

[0024] Figure 3 The internal structure diagram of Moformer_E of Example 1;

[0025] Figure 4It is a structural diagram of MoEBlock of Moformer_E of Example 1;

[0026] Figure 5 It is a structural diagram of the ConvFFN of Moformer_E of Example 1;

[0027] Figure 6 The internal structure diagram of Moformer_A of the first embodiment;

[0028] Figure 7 It is a structural diagram of MoABlock of Moformer_A of Example 1;

[0029] Figure 8 It is a structural diagram of ConvFFN of Moformer_A of Example 1;

[0030] Fig. 9 This is a structural diagram of a feature fusion module of Embodiment 1;

[0031] Fig.10 It is a structural diagram of Moformer of Example 1;

[0032] Fig.11 This is a structural diagram of MoBlock in Example 1. DETAILED DESCRIPTION

[0033] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0034] In terms of action recognition, the present invention has many advantages. Compared with the previous action recognition model slowfast, the slowfast model captures semantic information in the video through a slow channel, while the fast channel captures motion details, but it involves multiple hyperparameters (such as frame rate, channel number ratio, etc.), which need to be carefully adjusted to achieve optimal performance, and the fast channel and slow channel lack global information. In recent years, the self-attention mechanism has performed well in tasks such as video classification, target detection, and behavior recognition. The self-attention mechanism can effectively capture the long-range dependencies between video frames, both in time and space, help the model understand dynamic changes and event relationships, and allow the model to adaptively focus on important areas or frames in the video without the need for a fixed convolution window, so that different scenes and the diversity of actions can be better handled. Although self-attention can capture global information, it may ignore some local details, which may cause information loss in some tasks. How to combine the advantages of the slowfast model and the self-attention mechanism is a problem to be solved, and the present invention has made certain answers to this problem. The present invention designs a transformer structure Moformer to replace the channel in slowfast, and then changes the sampling method to better adapt to the examination environment, and uses the self-attention mechanism to process the action features. Subsequently, the present invention designs a feature fusion module FuseFrames to effectively fuse the two features. The fused features are once again passed through the Moformer structure to more fully extract the features.

[0035] The quality and diversity of the data set directly affect the learning effect and generalization ability of the model. High-quality data sets can help the model capture more features and patterns, thereby improving performance. However, this may be difficult to achieve in this task. Because in the examination room monitoring scenario, the movement amplitude is small, there are many people, and the frequency of abnormal movements is relatively small, so it is extremely difficult to mark, and the examination room environment is relatively homogeneous, so it is difficult to ensure the diversity of the data set. The pre-training-fine-tuning paradigm is an effective model training strategy, especially in video understanding and other deep learning tasks. Train the model on a large-scale data set to learn common features and knowledge to improve the performance and generalization ability of the model. Then further train the pre-trained model on the examination room monitoring data set of the present invention, and adjust the model parameters to better adapt to the needs of specific tasks. The data volume required at this stage is relatively small, and better results can be achieved with limited data.

[0036] Embodiment 1

[0037] like Figure 1 As shown, this embodiment provides a method for analyzing abnormal movements in an examination room based on feature fusion, including:

[0038] S101: Obtaining the examination room surveillance video to be analyzed;

[0039] S102: Input the examination room surveillance video to be analyzed into the trained examination room abnormal motion analysis model to obtain the examination room abnormal motion analysis results; wherein the trained examination room abnormal motion analysis model is used to extract primary features from the examination room surveillance video to be analyzed, extract environmental features and motion features from the primary features respectively, fuse the environmental features and motion features to obtain fused features, extract features from the fused features to obtain final features, classify and identify the final features to obtain the examination room abnormal motion analysis results.

[0040] Furthermore, the S101: obtains the examination room surveillance video to be analyzed, and collects it through a camera.

[0041] Furthermore, the training process of the trained abnormal motion analysis model in the examination room includes:

[0042] Constructing a training set, wherein the training set is a test room surveillance video with known abnormal action labels;

[0043] The training set is input into the abnormal motion analysis model for the examination room, and the model is trained. When the total loss function value of the model no longer decreases, or the number of iterations exceeds the set number, the training is stopped to obtain the trained abnormal motion analysis model for the examination room.

[0044] It should be understood that weighted cross entropy is an improved version of the cross entropy loss function, which is mainly used to solve the problem of class imbalance. In the environment of examination room monitoring, the number of samples of various action categories in the data set is often unbalanced. If not handled, the model may tend to predict those categories with a large number of samples, such as students doing questions, resulting in poor prediction performance for minority categories, such as picking up things, drinking water, etc. Therefore, the weighted cross entropy loss function is used in the prediction stage. Here, the formula for weighted cross entropy can be written as:

[0045] L w (p,q)= -∑ i w i q i logp i (3-8)

[0046] Here w i is the weight of the i-th category, which is determined based on the category frequency. The present invention is set according to the inverse of the category frequency, and the formula can be expressed as:

[0047]

[0048] Where N is the total number of samples, n i is the number of samples in the i-th class.

[0049] Since it is difficult to annotate video datasets, the cost of labeled exam audio datasets is very high, and the oral proficiency of candidates varies greatly, it is difficult to train a model with good recognition effect simply using the exam dataset. In order to overcome this problem, the present invention adopts the pre-training-fine-tuning paradigm, and the specific steps are as follows:

[0050] (1) The ConFrames model is pre-trained using the public dataset AVA (V2.2), with a total video data length of approximately 100 hours.

[0051] (2) Use the manually labeled internal examination room monitoring dataset to fine-tune some parameters of the ConFrames model. The total length of video data is about 1 hour.

[0052] Regarding the selection of parameters that need to be fine-tuned, the present invention fine-tunes the parameters of the channel self-attention calculation module of the feature fusion module. This is because in the examination scene, the proportion of the environment and the character's actions is relatively fixed. Therefore, the present invention mainly fine-tunes the parameters of this module to achieve better recognition effects on the internal data set.

[0053] Furthermore, if Figure 2 As shown, the trained abnormal motion analysis model for the examination room includes:

[0054] A backbone module, the input end of which is used to input the examination room surveillance video to be analyzed;

[0055] The output end of the backbone module is connected to the input end of the first branch and the input end of the second branch respectively, the first branch comprises: a first downsampling layer and a Moformer_E module connected in sequence, the input end of the first downsampling layer is the input end of the first branch; the output end of the Moformer E module is the output end of the first branch;

[0056] The second branch includes: a Moformer_A module, an input end of the Moformer_A module is an input end of the second branch, and an output end of the Moformer_A module is an output end of the second branch;

[0057] The output end of the first branch and the output end of the second branch are both connected to the input end of the feature fusion module, the output end of the feature fusion module is connected to the input end of the Moformer module, and the output end of the Moformer module outputs the abnormal action classification and recognition result.

[0058] It should be understood that the structure of the action recognition model ConFrames proposed in the present invention is as follows Figure 2As shown in the figure, the input of the model is the location labels of students and teachers and the clipped abnormal action video frames. First, the input video frame image passes through a convolution trunk as the first layer of its input processing. The trunk consists of three convolution layers, and the step sizes of the convolution layers are 2, 2, and 1 respectively. Then it is divided into two groups and sent to two channels, which are divided into environmental channels and action channels. The environmental channel first uses the downsampling module to reduce the number of images, and then enters the Moformer_E module to extract environmental features. At the same time, the action channel uses the Moformer_A module to extract action features. Then, the extracted action features and environmental features are fused, and after fusion, they are input into the Moformer module again for further feature extraction, and finally the output results are used for classification and recognition.

[0059] Furthermore, the backbone module includes: a first convolutional layer, a second convolutional layer and a third convolutional layer connected in sequence; the backbone module is used to extract primary features from the examination room surveillance video to be analyzed.

[0060] Further, such as Figure 3 As shown, the Moformer_E module includes:

[0061] The first MoEBlock module, the first ConvFFN module, the second MoEBlock module and the second ConvFFN module are connected in sequence.

[0062] Furthermore, the Moformer_E module is used to extract environmental features from primary features.

[0063] Furthermore, if Figure 4 As shown, the internal structures of the first MoEBlock module and the second MoEBlock module are consistent, and the first MoEBlock module includes:

[0064] The first layer normalization module, the input end of the first layer normalization module is the input end of the first MoEBlock module;

[0065] The output end of the first layer normalization module is connected to the input end of the first local branch and the input end of the second global branch respectively;

[0066] The first local branch includes: a first fully connected layer, the input end of the first fully connected layer is connected to the output end of the first layer normalization module, the output end of the first fully connected layer is respectively connected to the input end of the first depth-separable convolution layer, the input end of the second depth-separable convolution layer and the input end of the third depth-separable convolution layer; the output end of the second depth-separable convolution layer and the output end of the third depth-separable convolution layer are connected to the input end of the first multiplier; the output end of the first multiplier is connected to the input end of the second fully connected layer, the output end of the second fully connected layer is connected to the input end of the first activation function layer, the output end of the first depth-separable convolution layer and the output end of the first activation function layer are both connected to the input end of the second multiplier, and the output end of the second multiplier is the output end of the first local branch;

[0067] The second global branch includes: a first self-attention mechanism layer, wherein the input end of the first self-attention mechanism layer is connected to the output end of the first layer normalization module, the output end of the first self-attention mechanism layer is connected to the input end of the third fully connected layer, and the output end of the third fully connected layer is respectively connected to the input end of the second downsampling layer, the input end of the third downsampling layer and the input end of the third multiplier; the output end of the second downsampling layer is connected to the input end of the fourth multiplier; the output end of the third downsampling layer is connected to the input end of the third multiplier; the output end of the third multiplier is connected to the input end of the second activation function layer; the output end of the second activation function layer is connected to the input end of the fourth multiplier, and the output end of the fourth multiplier is the output end of the second global branch;

[0068] The output end of the first local branch and the output end of the second global branch are both connected to the input end of the first connection unit, and the output end of the first connection unit is connected to the input end of the fourth fully connected layer; the output end of the fourth fully connected layer is connected to the input end of the first adder, and the input end of the first adder is also connected to the input end of the first layer normalization module; the output end of the first adder is the output end of the first MoEBlock module.

[0069] Furthermore, the self-attention input transformation layer of the second global branch is expressed as follows:

[0070]

[0071] Where X is the vector representation of the input, W q is the query weight matrix, Q gl is the generated query vector, W k is the key weight matrix, K gl is the generated key vector, W v is the value weight matrix, V gl is the resulting matrix of value vectors.

[0072] First, K and V are downsampled, and then the standard attention calculation process is performed on Q, K, and V to extract low-frequency global information. The extracted information is as follows:

[0073] X gl =Softmax(Q gl Pool(K gl )) · Pool(V gl ) (3-2)

[0074] Among them, Pool represents the downsampling operation.

[0075] The pattern of the second global branch effectively reduces the number of FLOPs required for attention and also obtains a global receptive field. However, although it can effectively capture low-frequency global information, it is insufficient in processing high-frequency local information.

[0076] Furthermore, the formula expression of the first local branch is as follows:

[0077] In the first local branch, a linear transformation is first applied to obtain Q, K, and V:

[0078] Q, K, V = FC(X0) (3-3)

[0079] Among them, X0 represents the initial input of the first local branch, and FC represents the fully connected layer;

[0080] After performing the linear transformation, a local feature aggregation process with shared weights is first performed on V. Then, based on the processed V and Q, K, context-aware local enhancement is performed. For V, a depthwise separable convolution (DWconv) is used to aggregate local information. The weights of DWconv are globally shared.

[0081] After integrating the local information of V with the shared weights, the joint Q and K are combined to generate context-aware weights. Two DWconv are used to aggregate the local information of Q and K respectively.

[0082] Then, the Hadamard product of Q and K is calculated and a series of transformations are performed on the result to obtain context-aware weights. Finally, the resulting weights are used to enhance local features. The calculation process is as follows:

[0083]

[0084] Among them, Softmax means that softmax is used as the activation function, so that each element of the output is between [0,1].

[0085] Next, the first connection unit fuses the output of the first local branch and the output of the first global branch. The first connection unit connects the two outputs in the channel dimension, and then the fourth fully connected layer is used to process the connected channel-sized data.

[0086] The use of the self-attention mechanism may reduce the efficiency of the model, so a lightweight transformer model is needed to improve the computational efficiency while maintaining the recognition effect. Therefore, the present invention adopts a dual-branch structure, which is divided into a first local branch and a first global branch. Context-aware weights are fused in the first local branch to aggregate high-frequency local information. For the first global branch, ordinary self-attention is used, but K and V are downsampled to reduce FLOPs, which helps the model capture low-frequency global information. Then, the outputs of the first local branch and the first global branch are fused.

[0087] Environmental features reflect the spatial features of the examination room. Extracting environmental features does not require time information, so there is no need to use all the input video frames, and downsampling can be performed first. In the past, some ideas were to select a frame every few frames. However, in the scenario of examination room monitoring, the environment is almost unchanged throughout the entire video, so you can even take only one frame, so that you can use it as an image for feature extraction. The feature extraction of images has better performance than that of videos. Therefore, the present invention designs a lightweight transformer structure for environmental feature extraction, named Moformer_E. The structure of Moformer_E is as follows Figure 3 As shown in Figure 1. Moformer_E consists of two repeated structures, each of which consists of a MoEBlock block and a convolutional feedforward network (ConvFFN). The structures of the MoEBlock block and the convolutional feedforward network (ConvFFN) are introduced below. First, the MoEBlock block has a structure as shown in Figure 1. Figure 4 As shown, each block consists of a local branch and a global branch, the left branch in the figure is the local branch, and the right branch is the global branch. The present invention adds a local branch, which enables the model of the present invention to achieve high performance. It includes some standard attention operations.

[0088] Furthermore, if Figure 5 As shown, the internal structures of the first ConvFFN module and the second ConvFFN module are consistent, and the first ConvFFN module includes:

[0089] A second normalization module, a fifth fully connected layer, a third activation function layer, a fourth depthwise separable convolutional layer, a sixth fully connected layer, and a second adder connected in sequence;

[0090] The input end of the second-layer normalization module is the input end of the first ConvFFN module, the input end of the second-layer normalization module is connected to the input end of the fifth depth-wise separable convolutional layer, the output end of the fifth depth-wise separable convolutional layer is connected to the input end of the third-layer normalization module, the output end of the third-layer normalization module is connected to the input end of the seventh fully connected layer, the output end of the seventh fully connected layer is connected to the input end of the second adder, and the output end of the second adder is the output end of the first ConvFFN module.

[0091] ConvFFN is a module that combines the features of convolutional layers and feed-forward neural networks to enhance the model's ability to capture local and global features simultaneously. ConvFFN uses deep convolution (DWconv) after GELU activation, which enables ConvFFN to aggregate local information. With DWconv, downsampling is performed directly in ConvFFN.

[0092] Furthermore, the working process of the first ConvFFN module includes:

[0093] The input data first passes through batch normalization and a fully connected layer, and then uses the GELU activation function for nonlinear transformation; then, after GELU activation, a depthwise separable convolution (DWconv) is used to make ConvFFN aggregate local information; again, it passes through a fully connected layer (FC Layer) for linear transformation;

[0094] The other line represents the skip connection, which is used to solve the gradient vanishing problem in deep neural networks. In the skip connection, feature extraction is first performed through a depthwise separable convolution (DW convolution). Then, it is regularized by batch normalization (BN). Then, the data enters a fully connected layer (FC Layer) for linear transformation. The DW convolution and fully connected layers are used to downsample and increase the dimension of the input, respectively. Finally, the results of the sixth and seventh fully connected layers are added to obtain the output of the first ConvFFN module.

[0095] Further, such as Figure 6 As shown, the Moformer_A module includes:

[0096] The first MoABlock module, the third ConvFFN module, the second MoABlock module and the fourth ConvFFN module are connected in sequence.

[0097] Motion features reflect the changes in the movements of characters in the video. Compared with environmental features that only sample a small number of frames, motion features retain continuous input frames, which can maximize the retention of action information. In the scene of examination room monitoring, the movements of characters have the following characteristics: 1. The amplitude of the movement is small; 2. The duration of the movement varies; 3. The size of the characters making the movement is relatively small. Therefore, the effect of extracting features using traditional methods is generally poor. The present invention innovatively designs a novel motion feature extraction module, named Moformer_A, with the following structure: Figure 6 As shown in the figure, Moformer_A also consists of two repeated structures, each of which consists of a MoABlock block and a convolutional feed-forward network (ConvFFN). Next, we will introduce the MoABlock block and ConvFFN.

[0098] The structure of MoABlock is as follows Figure 7 As shown in the figure, compared with MoEBlock, MoABlock only retains the local branch and discards the global branch. This is because the action features are more focused on local features and do not need to be combined with global information. In addition, more video frames are required to extract action features. Reducing one branch can also improve operating efficiency.

[0099] Furthermore, if Figure 7 As shown, the internal structures of the first MoABlock module and the second MoABlock module are consistent, and the first MoABlock module includes:

[0100] An eighth fully connected layer, wherein the input end of the eighth fully connected layer is the input end of the first MoABlock module; the output end of the eighth fully connected layer is respectively connected to the input end of the first three-dimensional convolutional layer, the input end of the second three-dimensional convolutional layer, and the input end of the third three-dimensional convolutional layer;

[0101] The output of the second three-dimensional convolutional layer and the output of the third three-dimensional convolutional layer are both connected to the input of the fifth multiplier, the output of the fifth multiplier is connected to the input of the ninth fully connected layer, and the output of the ninth fully connected layer is connected to the input of the Swish layer. Swish is an activation function in deep learning, which is essentially the product of the input x and its sigmoid value, where sigmoid (x) is a common S-type function that can map the input value to between 0 and 1. The output of the Swish layer is connected to the input of the tenth fully connected layer, and the output of the tenth fully connected layer is connected to the input of the fourth activation function layer;

[0102] The output end of the first three-dimensional convolutional layer and the output end of the fourth activation function layer are both connected to the input end of the sixth multiplier, the output end of the sixth multiplier is connected to the input end of the eleventh fully connected layer, the output end of the eleventh fully connected layer is connected to the input end of the third adder, and the output end of the third adder is the output end of the first MoABlock module.

[0103] Furthermore, the first MoABlock module comprises:

[0104] The input data is also linearly transformed to obtain Q, K and V. For Q and K, first a 3D convolution layer is used for feature extraction; then, the data enters a fully connected layer for linear transformation; then, the Swish activation function is used for nonlinear transformation; it passes through a fully connected layer for linear transformation again, and the softmax function is used to convert the result into a probability distribution; and V will pass through a 3D convolution module and multiply it to obtain a new feature representation; finally, a fully connected layer is used to process the data of the connected channel size to obtain the final output.

[0105] Furthermore, the calculation process of the first MoABlock module is as follows:

[0106]

[0107] Where d is the dimension of query (Q), key (K), and value (V). When d is large, the calculated attention score may become too large, resulting in numerical instability. To solve this problem, the present invention divides the attention score by sqrt(d), that is, scales the attention score. Compared with the previous method, this method introduces stronger nonlinearity. Specifically, in the previous process of generating context-aware weights for local self-attention, the only nonlinear operator was Softmax. However, in this module, in addition to Softmax, Swish is also performed. Stronger nonlinearity results in higher quality context-aware weights.

[0108] Then the structure of ConvFFN is different from that of the environment extraction module, such as Figure 8 As shown in the figure, firstly, the depthwise separable convolution is changed to 3D convolution, and secondly, all operations in the skip connection are deleted to speed up the inference.

[0109] Furthermore, if Figure 8 As shown, the internal structures of the third ConvFFN module and the fourth ConvFFN module are consistent, and the third ConvFFN module includes:

[0110] A fourth normalization module, a twelfth fully connected layer, a fifth activation function layer, a fourth three-dimensional convolutional layer, a thirteenth fully connected layer and a fourth adder connected in sequence;

[0111] The input end of the fourth adder is also connected to the input end of the fourth layer normalization module;

[0112] The input of the fourth layer normalization module is the input of the third ConvFFN module;

[0113] The output terminal of the fourth adder is the output terminal of the third ConvFFN module.

[0114] Furthermore, the third ConvFFN module is used for inputting data to first pass through batch normalization and a fully connected layer, and then use a GELU activation function for nonlinear transformation; then, after GELU activation, a three-dimensional convolution (3dconv) is used to make ConvFFN aggregate spatiotemporal information; and again, a linear transformation is performed through a fully connected layer (FC Layer);

[0115] The other line represents the skip connection. The skip connection of the third ConvFFN module does not require additional operations. It directly adds the input data and the result of the thirteenth fully connected layer to obtain the output of the third ConvFFN module.

[0116] Furthermore, if Fig. 9 As shown, the feature fusion module includes:

[0117] A fourth convolutional layer, wherein an input value of the fourth convolutional layer is an environmental feature;

[0118] The output of the fourth convolutional layer is connected to the input of the splicing unit, and the input of the splicing unit also inputs the action feature;

[0119] The output of the concatenation unit is connected to the input of the global average pooling layer;

[0120] The output of the global average pooling layer is connected to the input of the fourteenth fully connected layer;

[0121] An output terminal of the fourteenth fully connected layer is connected to an input terminal of the seventh multiplier;

[0122] The output end of the seventh multiplier is the output end of the feature fusion module.

[0123] Furthermore, the feature fusion module obtains a dimension of and a dimension of The output of ; a 1×1 convolutional layer is used to expand the number of channels with dimension 1 to Then the outputs of the two adjusted branches are concatenated in the horizontal dimension;

[0124] Next, the SE module is applied to adjust the fused features. The SE module consists of a squeeze operation layer, a fully connected network, and a feature mapping layer connected in sequence.

[0125] First, the Squeeze operation is to perform global average pooling on the feature map, compressing the three-dimensional spatial information into a one-dimensional vector. The Squeeze operation compresses all the spatial information in the feature map into a channel-level descriptor:

[0126] Z i =f gap (x i ) (3-5)

[0127] Among them, x i is the feature map of the i-th channel, Z i is the global information of the channel, f gap Represents a global average pooling operation.

[0128] The weight of each channel is generated through a fully connected network. This fully connected network contains a dimensionality reduction layer and a dimensionality increase layer. The former is used to reduce the number of parameters, and the latter is restored to the original number of channels. In this way, the model can learn the interdependence between different channels and generate a weight vector s based on this. Each element in the vector corresponds to the importance score of a channel:

[0129] s=W2(ReLU(W1(Z)) (3-6)

[0130] The ReLU activation function is used, and W1 and W2 are the weight matrices of the fully connected layer.

[0131] Finally, the generated weight vector s is applied to the original feature map by element-by-element multiplication:

[0132] x i ′ = s i · x i (3-7)

[0133] Furthermore, if Fig.10 As shown, the Moformer module includes:

[0134] The first MoBlock module, the fifth ConvFFN module, the second MoBlock module and the sixth ConvFFN module are connected in sequence.

[0135] Furthermore, if Fig.11As shown, the internal structures of the first MoBlock module and the second MoBlock module are consistent, and the first MoBlock module includes:

[0136] The fifth layer normalization module, the input end of the fifth layer normalization module is the input end of the first MoBlock module;

[0137] The output end of the fifth layer normalization module is connected to the input end of the second local branch and the input end of the second global branch respectively;

[0138] The second local branch includes: a fifteenth fully connected layer, the input end of the fifteenth fully connected layer is connected to the output end of the fifth normalization module, the output end of the fifteenth fully connected layer is respectively connected to the input end of the sixth depth-separable convolutional layer, the input end of the seventh depth-separable convolutional layer and the input end of the eighth depth-separable convolutional layer; the output end of the seventh depth-separable convolutional layer and the output end of the eighth depth-separable convolutional layer are connected to the input end of the eighth multiplier; the output end of the eighth multiplier is connected to the input end of the sixteenth fully connected layer, the output end of the sixteenth fully connected layer is connected to the input end of the Swish layer, the output end of the Swish layer is connected to the input end of the seventeenth fully connected layer, the output end of the seventeenth fully connected layer is connected to the input end of the sixth activation function layer, the output end of the sixth depth-separable convolutional layer and the output end of the sixth activation function layer are both connected to the input end of the ninth multiplier, and the output end of the ninth multiplier is the output end of the second local branch;

[0139] The second global branch includes: a second self-attention mechanism layer, the input end of the second self-attention mechanism layer is connected to the output end of the fifth normalization module, the output end of the second self-attention mechanism layer is connected to the input end of the eighteenth fully connected layer, the output end of the eighteenth fully connected layer is respectively connected to the input end of the fourth downsampling layer, the input end of the fifth downsampling layer and the input end of the tenth multiplier; the output end of the fourth downsampling layer is connected to the input end of the eleventh multiplier; the output end of the fifth downsampling layer is connected to the input end of the tenth multiplier; the output end of the tenth multiplier is connected to the input end of the seventh activation function layer; the output end of the seventh activation function layer is connected to the input end of the eleventh multiplier, and the output end of the eleventh multiplier is the output end of the second global branch;

[0140] The output end of the second local branch and the output end of the second global branch are both connected to the input end of the second connection unit, and the output end of the second connection unit is connected to the input end of the nineteenth fully connected layer; the output end of the nineteenth fully connected layer is connected to the input end of the fifth adder, and the input end of the fifth adder is also connected to the input end of the fifth layer normalization module; the output end of the fifth adder is the output end of the first MoBlock module.

[0141] Furthermore, the calculation process of the first MoBlock module is as follows:

[0142]

[0143] Among them, X lo is the output of the second local branch, X gl is the output of the second global branch. The calculation of the second global branch is exactly the same as that of the first global branch. out It is the final output of the MoBlock module.

[0144] Furthermore, the internal structures of the fifth ConvFFN module and the sixth ConvFFN module are consistent with the internal structure of the third ConvFFN module.

[0145] After feature fusion is completed, further processing is required to enhance the expressiveness of the features. The fused features are input into Moformer. Moformer has many similarities with the previous Moformer_A and Moformer_E, and only differs in details. It is also composed of two repeated structures, such as Fig.10 shown.

[0146] The structure of ConvFFN is exactly the same as that of the action extraction module. Compared with MoEBlock, MoBlock also changes the depthwise separable convolution to 3D convolution. At the same time, in the local branch, referring to the structure of MoABlock, a Swish activation function and a fully connected layer are added to obtain stronger nonlinearity, such as Fig.11 shown.

[0147] Moformer combines some features of Moformer_E and MoAfomer_A. It can combine global and local perception as well as time perception, further improving the expressiveness of features.

[0148] Embodiment 2

[0149] This embodiment provides an abnormal motion analysis system for an examination room based on feature fusion, including:

[0150] An acquisition module is configured to: acquire the examination room surveillance video to be analyzed;

[0151] The analysis module is configured as follows: inputting the examination room surveillance video to be analyzed into the trained examination room abnormal action analysis model to obtain the examination room abnormal action analysis results; wherein the trained examination room abnormal action analysis model is used to extract primary features from the examination room surveillance video to be analyzed, extract environmental features and action features from the primary features respectively, fuse the environmental features and action features to obtain fused features, extract features from the fused features to obtain final features, classify and identify the final features to obtain the examination room abnormal action analysis results.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. The abnormal motion analysis method in the examination room based on feature fusion is characterized by: include: Obtain the examination room surveillance video to be analyzed; The examination room surveillance video to be analyzed is input into the trained examination room abnormal motion analysis model to obtain the examination room abnormal motion analysis results; wherein the trained examination room abnormal motion analysis model is used to extract primary features from the examination room surveillance video to be analyzed, extract environmental features and motion features from the primary features respectively, fuse the environmental features and motion features to obtain fused features, extract features from the fused features to obtain final features, classify and identify the final features to obtain the examination room abnormal motion analysis results.

2. The method for analyzing abnormal movements in an examination room based on feature fusion as claimed in claim 1, characterized in that: The trained abnormal motion analysis model for the examination room includes: A backbone module, the input end of which is used to input the examination room surveillance video to be analyzed; The output end of the backbone module is connected to the input end of the first branch and the input end of the second branch respectively, the first branch comprises: a first downsampling layer and a Moformer_E module connected in sequence, the input end of the first downsampling layer is the input end of the first branch; the output end of the Moformer_E module is the output end of the first branch; The second branch includes: a Moformer_A module, an input end of the Moformer_A module is an input end of the second branch, and an output end of the Moformer_A module is an output end of the second branch; The output end of the first branch and the output end of the second branch are both connected to the input end of the feature fusion module, the output end of the feature fusion module is connected to the input end of the Moformer module, and the output end of the Moformer module outputs the abnormal action classification and recognition result.

3. The method for analyzing abnormal movements in an examination room based on feature fusion as claimed in claim 2, characterized in that: The Moformer_E module includes: A first MoEBlock module, a first ConvFFN module, a second MoEBlock module, and a second ConvFFN module connected in sequence; The internal structures of the first MoEBlock module and the second MoEBlock module are consistent. The first MoEBlock module includes: a first-layer normalization module, the input end of the first-layer normalization module is the input end of the first MoEBlock module; the output end of the first-layer normalization module is respectively connected to the input end of the first local branch and the input end of the second global branch; The first local branch includes: a first fully connected layer, the input end of the first fully connected layer is connected to the output end of the first layer normalization module, the output end of the first fully connected layer is respectively connected to the input end of the first depth-separable convolution layer, the input end of the second depth-separable convolution layer and the input end of the third depth-separable convolution layer; the output end of the second depth-separable convolution layer and the output end of the third depth-separable convolution layer are connected to the input end of the first multiplier; the output end of the first multiplier is connected to the input end of the second fully connected layer, the output end of the second fully connected layer is connected to the input end of the first activation function layer, the output end of the first depth-separable convolution layer and the output end of the first activation function layer are both connected to the input end of the second multiplier, and the output end of the second multiplier is the output end of the first local branch; The second global branch includes: a first self-attention mechanism layer, wherein the input end of the first self-attention mechanism layer is connected to the output end of the first layer normalization module, the output end of the first self-attention mechanism layer is connected to the input end of the third fully connected layer, and the output end of the third fully connected layer is respectively connected to the input end of the second downsampling layer, the input end of the third downsampling layer and the input end of the third multiplier; the output end of the second downsampling layer is connected to the input end of the fourth multiplier; the output end of the third downsampling layer is connected to the input end of the third multiplier; the output end of the third multiplier is connected to the input end of the second activation function layer; the output end of the second activation function layer is connected to the input end of the fourth multiplier, and the output end of the fourth multiplier is the output end of the second global branch; The output end of the first local branch and the output end of the second global branch are both connected to the input end of the first connection unit, and the output end of the first connection unit is connected to the input end of the fourth fully connected layer; the output end of the fourth fully connected layer is connected to the input end of the first adder, and the input end of the first adder is also connected to the input end of the first layer normalization module; the output end of the first adder is the output end of the first MoEBlock module.

4. The method for analyzing abnormal movements in an examination room based on feature fusion as claimed in claim 3, characterized in that: The internal structures of the first ConvFFN module and the second ConvFFN module are consistent. The first ConvFFN module includes: A second normalization module, a fifth fully connected layer, a third activation function layer, a fourth depthwise separable convolutional layer, a sixth fully connected layer, and a second adder connected in sequence; The input end of the second-layer normalization module is the input end of the first ConvFFN module, the input end of the second-layer normalization module is connected to the input end of the fifth depth-separable convolutional layer, the output end of the fifth depth-separable convolutional layer is connected to the input end of the third-layer normalization module, the output end of the third-layer normalization module is connected to the input end of the seventh fully connected layer, the output end of the seventh fully connected layer is connected to the input end of the second adder, and the output end of the second adder is the output end of the first ConvFFN module; The working process of the first ConvFFN module includes: the input data first passes through batch normalization and a fully connected layer, and then uses the GELU activation function for nonlinear transformation; then, after the GELU activation, a depthwise separable convolution is used to make the ConvFFN aggregate local information; again, it passes through a fully connected layer for linear transformation; The other line represents the skip connection, which is used to solve the gradient vanishing problem in deep neural networks. In the skip connection, feature extraction is first performed through a depthwise separable convolution. Then, it is regularized through batch normalization. Then, the data enters a fully connected layer for linear transformation. The depthwise separable convolution and the fully connected layer are used to downsample and increase the dimension of the input, respectively. Finally, the results of the sixth and seventh fully connected layers are added together to obtain the output of the first ConvFFN module.

5. The method for analyzing abnormal movements in an examination room based on feature fusion as claimed in claim 2, characterized in that: The Moformer_A module includes: A first MoABlock module, a third ConvFFN module, a second MoABlock module and a fourth ConvFFN module connected in sequence; The internal structures of the first MoABlock module and the second MoABlock module are consistent. The first MoABlock module includes: An eighth fully connected layer, wherein the input end of the eighth fully connected layer is the input end of the first MoABlock module; the output end of the eighth fully connected layer is respectively connected to the input end of the first three-dimensional convolutional layer, the input end of the second three-dimensional convolutional layer, and the input end of the third three-dimensional convolutional layer; The output end of the second three-dimensional convolutional layer and the output end of the third three-dimensional convolutional layer are both connected to the input end of the fifth multiplier, the output end of the fifth multiplier is connected to the input end of the ninth fully connected layer, the output end of the ninth fully connected layer is connected to the input end of the Swish layer, the output end of the Swish layer is connected to the input end of the tenth fully connected layer, and the output end of the tenth fully connected layer is connected to the input end of the fourth activation function layer; The output end of the first three-dimensional convolutional layer and the output end of the fourth activation function layer are both connected to the input end of the sixth multiplier, the output end of the sixth multiplier is connected to the input end of the eleventh fully connected layer, the output end of the eleventh fully connected layer is connected to the input end of the third adder, and the output end of the third adder is the output end of the first MoABlock module; The first MoABlock module includes: the input data is also linearly transformed to obtain Q, K and V. For Q and K, first, a 3D convolution layer is used for feature extraction; then, the data enters a fully connected layer for linear transformation; then, the Swish activation function is used for nonlinear transformation; it is linearly transformed again through a fully connected layer, and the result is converted into a probability distribution using a softmax function; and V will pass through a 3D convolution module and then be multiplied with it to obtain a new feature representation; finally, a fully connected layer is used to process the data of the connected channel size to obtain the final output.

6. The method for analyzing abnormal movements in an examination room based on feature fusion as claimed in claim 5, characterized in that: The internal structures of the third ConvFFN module and the fourth ConvFFN module are consistent. The third ConvFFN module includes: A fourth normalization module, a twelfth fully connected layer, a fifth activation function layer, a fourth three-dimensional convolutional layer, a thirteenth fully connected layer and a fourth adder connected in sequence; The input end of the fourth adder is also connected to the input end of the fourth layer normalization module; The input of the fourth layer normalization module is the input of the third ConvFFN module; The output end of the fourth adder is the output end of the third ConvFFN module; The third ConvFFN module is used for inputting data, which first passes through batch normalization and a fully connected layer, and then uses a GELU activation function for nonlinear transformation; then, after GELU activation, a three-dimensional convolution is used to enable ConvFFN to aggregate spatiotemporal information; again, a linear transformation is performed through a fully connected layer FC Layer; another line represents a skip connection, and the skip connection of the third ConvFFN module does not require additional operations, and directly adds the input data and the result of the thirteenth fully connected layer to obtain the output of the third ConvFFN module.

7. The method for analyzing abnormal movements in an examination room based on feature fusion as claimed in claim 2, characterized in that feature fusion After the feature extraction of the two branches, the module obtains a dimension of and a dimension of Output: Use a 1×1 convolutional layer to expand the number of channels of dimension 1 to Then the outputs of the two adjusted branches are concatenated in the horizontal dimension; Next, the SE module is applied to adjust the fused features. The SE module includes a squeeze operation layer, a fully connected network, and a feature mapping layer connected in sequence. First, the Squeeze operation is to perform global average pooling on the feature map, compressing the three-dimensional spatial information into a one-dimensional vector; the Squeeze operation compresses all the spatial information in the feature map into a channel-level descriptor: Z i =f gap (x i ) (3-5) Among them, x i is the feature map of the i-th channel, Z i is the global information of the channel, f gap represents the global average pooling operation; The weight of each channel is generated through a fully connected network to generate a weight vector s, in which each element corresponds to the importance score of a channel: s=W2(ReLU(W1(Z)) (3-6) Using the ReLU activation function, W1 and W2 are the weight matrices of the fully connected layer; Finally, the generated weight vector s is applied to the original feature map by element-by-element multiplication: x i ′ =s i ·x i (3-7)。 8. The method for analyzing abnormal movements in an examination room based on feature fusion as claimed in claim 2, characterized in that: The Moformer module includes: a first MoBlock module, a fifth ConvFFN module, a second MoBlock module and a sixth ConvFFN module connected in sequence; The internal structures of the first MoBlock module and the second MoBlock module are consistent. The first MoBlock module includes: The fifth layer normalization module, the input end of the fifth layer normalization module is the input end of the first MoBlock module; The output end of the fifth layer normalization module is connected to the input end of the second local branch and the input end of the second global branch respectively; The second local branch includes: a fifteenth fully connected layer, the input end of the fifteenth fully connected layer is connected to the output end of the fifth normalization module, the output end of the fifteenth fully connected layer is respectively connected to the input end of the sixth depth-separable convolutional layer, the input end of the seventh depth-separable convolutional layer and the input end of the eighth depth-separable convolutional layer; the output end of the seventh depth-separable convolutional layer and the output end of the eighth depth-separable convolutional layer are connected to the input end of the eighth multiplier; the output end of the eighth multiplier is connected to the input end of the sixteenth fully connected layer, the output end of the sixteenth fully connected layer is connected to the input end of the Swish layer, the output end of the Swish layer is connected to the input end of the seventeenth fully connected layer, the output end of the seventeenth fully connected layer is connected to the input end of the sixth activation function layer, the output end of the sixth depth-separable convolutional layer and the output end of the sixth activation function layer are both connected to the input end of the ninth multiplier, and the output end of the ninth multiplier is the output end of the second local branch; The second global branch includes: a second self-attention mechanism layer, the input end of the second self-attention mechanism layer is connected to the output end of the fifth normalization module, the output end of the second self-attention mechanism layer is connected to the input end of the eighteenth fully connected layer, the output end of the eighteenth fully connected layer is respectively connected to the input end of the fourth downsampling layer, the input end of the fifth downsampling layer and the input end of the tenth multiplier; the output end of the fourth downsampling layer is connected to the input end of the eleventh multiplier; the output end of the fifth downsampling layer is connected to the input end of the tenth multiplier; the output end of the tenth multiplier is connected to the input end of the seventh activation function layer; the output end of the seventh activation function layer is connected to the input end of the eleventh multiplier, and the output end of the eleventh multiplier is the output end of the second global branch; The output end of the second local branch and the output end of the second global branch are both connected to the input end of the second connection unit, and the output end of the second connection unit is connected to the input end of the nineteenth fully connected layer; the output end of the nineteenth fully connected layer is connected to the input end of the fifth adder, and the input end of the fifth adder is also connected to the input end of the fifth layer normalization module; the output end of the fifth adder is the output end of the first MoBlock module.

9. The method for analyzing abnormal movements in an examination room based on feature fusion as claimed in claim 8, characterized in that: The Moformer module includes: A first MoBlock module, a fifth ConvFFN module, a second MoBlock module, and a sixth ConvFFN module connected in sequence; The calculation process of the first MoBlock module is as follows: Among them, X lo is the output of the second local branch, X gl is the output of the second global branch. The calculation of the second global branch is exactly the same as that of the first global branch. out It is the final output of the MoBlock module.

10. The abnormal motion analysis system in the examination room based on feature fusion is characterized by: include: An acquisition module is configured to: acquire the examination room surveillance video to be analyzed; The analysis module is configured as follows: inputting the examination room surveillance video to be analyzed into the trained examination room abnormal action analysis model to obtain the examination room abnormal action analysis results; wherein the trained examination room abnormal action analysis model is used to extract primary features from the examination room surveillance video to be analyzed, extract environmental features and action features from the primary features respectively, fuse the environmental features and action features to obtain fused features, extract features from the fused features to obtain final features, classify and identify the final features to obtain the examination room abnormal action analysis results.

Images