Video stream dimension lossless deep learning feature enhancement algorithm

Abstract

The invention belongs to the technical field of computer vision, and particularly relates to a video stream dimension lossless deep learning feature enhancement algorithm, which comprises the following steps of: firstly, connecting a feature tensor of a video clip of an input model and an expanded and grouped 3D (three-dimensional) convolution with a 3D space-time attention enhancement module: firstly, carrying out average pooling and maximum pooling on a channel layer; performing non-linear processing on the pooled features through an activation function, and multiplying the pooled features with an input feature tensor; and the space layer and the time layer are subjected to related processing according to the operation. According to the algorithm, a space layer and a channel layer of 3D convolution are expanded, grouping is carried out on a time layer, and the sequence of time dimensions in 3D convolution is not changed, so that original semantic information of a video is not damaged, however, the relevance between front and back frames is enhanced, meanwhile, a 3D space attention mechanism is increased by combining an expanded and grouped 3D convolution structure, and therefore, the video quality is improved. And the understanding capability of the computer on environment information, detail features and semantic logic in the video information is further improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a deep learning feature enhancement algorithm for lossless video stream dimension. Background Art

[0002] Extracting feature information from video streams is a challenging problem because it involves complex spatiotemporal feature processing and classification problems. Traditional 2D image processing methods have strong image feature processing capabilities, but weak understanding of spatiotemporal sequence features, weak matching capabilities for behavioral relationships between people and objects, and are easily disturbed by negative samples and background environments, with high requirements for data volume and quality. Although 3D convolutional neural networks can process spatiotemporal data, they still have certain limitations when processing behaviors with complex backgrounds and diversity, and are easily disturbed by similar action behaviors.

[0003] like Figure 1 As shown in the figure, when the existing 3D convolution-based deep learning technology extracts key features of video streams, it often uses temporal displacement technology to increase the semantic correlation between frames or video clips. This technology can transfer some channels along the time dimension, and flexibly insert them into the model to promote information exchange between adjacent frames without increasing computing power and parameters. But the most critical problem is that the order of channels between moving frames will destroy the coherence of the video, especially when the more feature information between moving frames, the more destructive it is to the original image. At present, two methods are generally used to weaken this effect: 1. Selectively move channels to reduce the number of changes in channel features; 2. Rearrange channels in branches and connect them to the main road in the form of residual connections to avoid destroying the integrity of the original video. However, both methods only indirectly solve or reduce the impact of this problem, and do not fundamentally solve the problem. Summary of the invention

[0004] 1. Technical problems to be solved

[0005] The purpose of the present invention is to solve the problem that 3D convolutional neural networks in the prior art still have certain limitations when processing behaviors with complex backgrounds and diversity, and to propose a deep learning feature enhancement algorithm that is lossless in video stream dimension.

[0006] 2. Technical solution

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A video stream dimension lossless deep learning feature enhancement algorithm, comprising the following steps:

[0009] S1: Use the in-vehicle camera to capture the driver area image and transmit the RGB image to the vehicle controller;

[0010] S2: First, the in-vehicle controller performs a scaling operation on the driver area image and inputs it frame by frame into the 3D neural network of the controller for feature analysis;

[0011] S3: When the video stream is input into the neural network model, first use 1*1 convolution to expand the channel dimension in the feature matrix of each frame of the original video stream by E times;

[0012] S4: Then, for the video stream data, the channel dimension of each segment of the video is grouped and split, and the splitting multiple is G times;

[0013] S5: The expansion multiple and the grouping multiple will have two forms: when E>G, the number of original video frames is compressed, so as to ignore the non-critical frames that have little or no impact on the classification of behavior states, making it easier for the model to observe the motion information of the behavior; when E<G, the number of original frames is expanded, and the model is easier to extract the environmental detail features in the video, which is equivalent to making the model capture the spatial information in the image in a video with a slower time flow;

[0014] S6: Copy the feature matrix of the original video into two identical feature tensors, respectively concatenate and construct a two-branch model with the two expansion and grouping forms of E<G and E>G, and then connect it with the 3D attention mechanism to increase the semantic relevance of the video stream channel layer, time series layer and spatial layer;

[0015] S7: In the 3D attention mechanism, first perform average pooling and max pooling on the channel layer, and the pooled features are then non-linearly processed through an activation function and multiplied by the input feature tensor to ensure that the tensor dimension information remains unchanged. Then, perform the relevant processing on the spatial layer and the time series layer in the above way. Finally, add the feature matrices output by the two branches bit by bit;

[0016] S8: The feature matrix after bit-by-bit addition is connected to the cross-entropy loss function. Through the batch gradient descent algorithm, after model training, the behavior category with the highest predicted proportion is obtained by the Softmax operator, and the model prediction result is output accordingly.

[0017] Preferably, in the S1, the in-vehicle controller includes an initialization module, an inference model, a data calculation module and a decision fusion module.

[0018] Preferably, in the S4, E is not equal to G, because when E=G, it is equivalent to expanding the channel dimension of each frame, and the purpose of interacting the feature information of the front and rear frames is not achieved.

[0019] Preferably, in each branch of the S6, after the expansion, grouping and rearrangement of the features are completed, it is then connected to the 3D attention mechanism.

[0020] 3. Beneficial effects

[0021] Compared with the prior art, the advantages of the present invention are:

[0022] (1) In the present invention, the algorithm expands the spatial layer and channel layer of the 3D convolution and groups them in the temporal layer. The order of the temporal dimension in the 3D convolution is not changed, so the original semantic information of the video is not destroyed. However, the correlation between the previous and next frames is enhanced. At the same time, the expanded and grouped 3D convolution structure is combined with the 3D spatial attention mechanism, which further enhances the computer's ability to understand the environmental information, detail features and semantic logic in the video information.

[0023] (2) In the present invention, the test was completed in the public AVA dataset of behavior recognition videos, and the average accuracy rate reached 92.5%, and the recall rate reached 94.6%. The model significantly improved the accuracy of behavior recognition in video streams. It can recognize large-target human motion behaviors, object motion, and small-target behaviors, that is, the technology improves the accuracy of behavior recognition in video streams, thereby verifying the correctness of video stream feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of the timing shift technology of the system;

[0025] Figure 2 Schematic diagram of the expansion group channel rearrangement proposed by the present invention, in which (a) E>G; (b) E <G;

[0026] Figure 3 This is a diagram showing the serial structure of the three-dimensional residual attention mechanism and the expansion grouping module proposed in the present invention;

[0027] Figure 4 Schematic diagram of the experimental hardware platform proposed in the present invention, wherein (a) is a hardware scheme diagram; (b) is a physical diagram of the hardware. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0029] Embodiment 1:

[0030] Reference Figure 2-3 , a deep learning feature enhancement algorithm for video stream dimension lossless, including the following steps:

[0031] S1: Use the in-vehicle camera to capture the driver area image and transmit the RGB image to the vehicle controller;

[0032] S2: The controller first scales the driver area image and inputs it into the controller's 3D neural network frame by frame for feature analysis;

[0033] S3: When the video stream is input into the neural network model, first use a 1*1 convolution to expand the channel dimension in the feature matrix of each frame of the original video stream by E times;

[0034] S4: Then, for the video stream data, the channel dimension of each video segment is grouped and split, and the splitting multiple is G times, where E is not equal to G. Because when E = G, it is equivalent to expanding the channels of each frame, and the purpose of feature information interaction between front and back frames is not achieved;

[0035] S5: The expansion multiple and the grouping multiple will have two forms: when E > G, the number of original video frames is compressed, so as to ignore the non-critical frames that have little or no impact on the classification of behavior states, making it easier for the model to observe the motion information of the behavior; when E < G, the number of original frames is expanded, and the model is easier to extract the environmental detail features in the video, which is equivalent to making the model capture the spatial information in the image in a video with a slower time flow;

[0036] S6: Copy the feature matrix of the original video into two identical feature tensors, and respectively construct a two-branch model in series with the two expansion and grouping forms of E < G and E > G. In each branch, after the expansion, grouping and rearrangement of the features are completed, then connect with the 3D attention mechanism to increase the semantic relevance of the video stream channel layer, time series layer and spatial layer;

[0037] S7: In the 3D attention mechanism, first perform average pooling and max pooling on the channel layer, and the pooled features are then non-linearly processed through an activation function and multiplied by the input feature tensor to ensure that the tensor dimension information remains unchanged. Then, perform the relevant processing on the spatial layer and the time series layer in the same way as above. Finally, add the feature matrices output by the two branches bit by bit;

[0038] S8: The feature matrix after bit-by-bit addition is connected to the cross-entropy loss function. Through the batch gradient descent algorithm, after model training, the behavior category with the highest predicted proportion is obtained by the Softmax operator, and the model prediction result is output accordingly.

[0039] In the present invention, in the algorithm, the spatial layer and the channel layer of the 3D convolution are expanded, and grouped in the time layer, without changing the order of the time dimension in the 3D convolution. Therefore, the original semantic information of the video is not damaged, but the front-back relevance between front and back frames is enhanced. At the same time, combined with the 3D convolution structure of expansion and grouping, the 3D spatial attention mechanism is added, further improving the computer's understanding ability of the environmental information, detail features and semantic logic in the video information.

[0040] Embodiment 2:

[0041] In this embodiment, the method is applied to the field of driver abnormal behavior recognition, and effectively monitors driver behaviors such as smoking, making phone calls, and fatigue (including: yawning with hands covering the mouth). The main implementation methods are as follows:

[0042] In this embodiment, refer to Figure 4 The main equipment includes controller, driver monitoring camera, related wiring harness and input and output devices such as mouse, keyboard and screen. The controller is mainly responsible for running models and related algorithms, processing signal input and output, and generating and storing operation logs; the camera is used to collect in-vehicle video data in real time and monitor driver behavior. The camera is installed in the rearview mirror, A-pillar, or steering wheel fixed area.

[0043] In this embodiment, the test case inspection method is used to verify the functional completeness of the model. The inspection cases include but are not limited to:

[0044]

[0045]

[0046] In this embodiment, after actual testing, the accuracy of fatigue monitoring reached 0.992, the accuracy of visual distraction reached 0.989, and the accuracy of smoking and phone call distraction reached 0.972.

[0047] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A video stream dimension lossless deep learning feature enhancement algorithm, characterized in that: It includes the following steps: S1: Use an in-vehicle camera to capture images of the driver area and transmit the RGB images to the vehicle-mounted controller; S2: In the vehicle-mounted controller, first perform a scaling operation on the driver area image and input it frame by frame into the controller's 3D neural network for feature analysis; S3: When the video stream is input into the neural network model, first use a 1*1 convolution to expand the channel dimension in the feature matrix of each frame of the original video stream by E times; S4: Then, for the video stream data, group and split the channel dimension of each video segment, and the splitting multiple is G times; S5: The expansion multiple and the grouping multiple will have two forms: when E > G, the number of original video frames is compressed, so as to ignore the non-critical frames that have little or no impact on the classification of behavior states, making it easier for the model to observe the motion information of the behavior; when E < G, the original number of frames is expanded, and the model is easier to extract the environmental detail features in the video. It is equivalent to letting the model capture the spatial information in the image in a video with a slower time flow; S6: Copy the feature matrix of the original video into two identical feature tensors, respectively concatenate and construct a two-branch model with the two expansion and grouping forms of E < G and E > G, and then connect it with the 3D attention mechanism to increase the semantic relevance of the video stream channel layer, time series layer and spatial layer; S7: In the 3D attention mechanism, first perform average pooling and max pooling on the channel layer. The pooled features are then non-linearly processed through an activation function and multiplied by the input feature tensor to ensure that the tensor dimension information remains unchanged. Then, perform the relevant processing on the spatial layer and the time series layer in the above way. Finally, add the feature matrices output by the two branches bit by bit; S8: The feature matrix after bit-by-bit addition is connected to the cross-entropy loss function. Through the batch gradient descent algorithm, after model training, the behavior category with the highest predicted proportion is obtained by the Softmax operator, and the model prediction result is output accordingly.

2. The video stream dimension lossless deep learning feature enhancement algorithm according to claim 1, characterized in that: In S1, the vehicle-mounted controller includes an initialization module, an inference model, a data calculation module and a decision fusion module.

3. The video stream dimension lossless deep learning feature enhancement algorithm according to claim 1, characterized in that: In S4, E is not equal to G because when E = G, it is equivalent to expanding the channel dimension of each frame and does not achieve the purpose of feature information interaction between front and rear frames.

4. The video stream dimension lossless deep learning feature enhancement algorithm according to claim 1, characterized in that: In S6, in each branch, after completing the expansion, grouping and rearrangement of features, it is then connected to the 3D attention mechanism.

Images