Abnormal driving behavior recognition method and related device based on video stream neural network

Abstract

A method and related device for identifying abnormal driving behavior based on a video stream neural network includes: extracting a first video frame sequence and a second video frame sequence from an input video stream at different sampling rates; extracting convolutional features from the first and second video frame sequences; fusing the features extracted from the first and second video frame sequences at different spatiotemporal scales; and classifying the fusion results, outputting classification information for the input video stream. This technical solution avoids feature loss, improves the accuracy of abnormal driving behavior, and reduces the imaging environment requirements of the identification method.

Description

Technical field

[0001] The present invention relates to the technical field of computer vision and pattern recognition, and in particular to a method and related device for identifying abnormal driving behavior based on a video stream neural network. [Background Technology]

[0002] The identification of abnormal driving behavior plays an important role in intelligent transportation systems. It can detect abnormal driving behavior in a timely manner, automatically identify, promptly warn, and reduce the accident rate. It can be widely used in scenarios such as intelligent transportation, traffic control, and smart cities. Its various potential applications from safety to traffic control have attracted widespread research attention from the public in recent years.

[0003] Currently, the commonly used method for identifying abnormal driving behavior is a convolutional neural network recognition method based on the dual-stream method. This method uses two network branches. One branch is a convolutional neural network that obtains a single-frame video image of the input video stream, which is mainly used to obtain the spatial features of objects in the image; the other branch is an optical flow convolutional neural network that obtains a dense optical flow sequence of the input video stream, which is mainly used to identify the temporal features of the input video stream; then the temporal features and spatial features of the two branches are fused to identify abnormal driving behavior.

[0004] When applied to specific scenarios, this method presents the following problems: First, when a person's position changes significantly, locating the person using only a single frame image results in large positioning errors. Second, when converting the input video stream into a dense optical flow sequence, the driver's driving behavior is easily obscured, making it impossible to capture the details of the driver's driving behavior. Furthermore, during driving, the vehicle is constantly shaking and the scene outside the window is rapidly changing. The generated dense optical flow map obscures the driver's subtle movements, resulting in the driver's driving behavior not being reflected in the optical flow map. Third, only performing feature fusion once can easily lead to a large loss of feature information. [Summary of the invention]

[0005] In response to the shortcomings of the existing technology, the purpose of this application is to provide a method and related device for identifying abnormal driving behavior based on video stream neural network, so as to avoid the loss of different depth features and improve the accuracy of abnormal driving behavior; at the same time, reduce the requirements of the identification method on the imaging environment.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] In a first aspect, a method for identifying abnormal driving behavior based on a video stream neural network is provided, comprising:

[0008] Extracting a first video frame sequence and a second video frame sequence respectively at different sampling rates from the acquired input video stream;

[0009] Extracting convolution features from the first video frame sequence and the second video frame sequence respectively;

[0010] fusing the features extracted from the first video frame sequence and the second video frame sequence respectively for different spatiotemporal scales;

[0011] The fusion result is classified, and classification information of the input video stream is output.

[0012] The present invention is further configured such that: the number of frames in the first video frame sequence is smaller than the number of frames in the second video frame sequence.

[0013] The present invention is further configured as follows: before extracting convolution features from the first video frame sequence and the second video frame sequence respectively, the specific steps include:

[0014] Each frame in the first video frame sequence and the second video frame sequence is processed into an image with consistent width and height.

[0015] The present invention is further configured as follows: extracting convolution features from the first video frame sequence and the second video frame sequence respectively includes:

[0016] Obtaining the first video frame sequence and the second video frame sequence and inputting them into a first convolution branch and a second convolution branch respectively;

[0017] In the first convolution branch, a 3D convolution block and a plurality of first convolution residual blocks are sequentially used to perform convolution operations to extract spatial features;

[0018] In the second convolution branch, a 3D convolution block and a plurality of second convolution residual blocks are sequentially used to perform convolution operations to extract temporal features;

[0019] Acquire spatial features and temporal features, and fuse the temporal features into the spatial features of the same scale to obtain fused spatiotemporal features;

[0020] The fused spatiotemporal features are obtained as features for subsequent convolution operations of the first convolution branch.

[0021] The present invention is further configured such that: the frame rate of the first convolution branch is greater than the frame rate of the second convolution branch.

[0022] The present invention is further configured as follows: after obtaining the fused spatiotemporal features as features for subsequent convolution operations of the first convolution branch, the specific steps include:

[0023] Obtain the spatiotemporal features output from the last first convolution residual block, and obtain the temporal features output from the last second convolution residual block, and fuse the spatiotemporal features and the temporal features to serve as features input to the classification module.

[0024] The present invention is further configured as follows: in the first convolution branch, a 3D convolution block and a plurality of first convolution residual blocks are sequentially used to perform convolution operations to extract spatial features, including:

[0025] The 3D convolution block includes a convolution layer and a maximum pooling layer, wherein the convolution layer adopts a 1*7*7 convolution kernel to reduce the dimension of the convolution feature so as to submit it to the maximum pooling layer for downsampling.

[0026] The present invention is further configured as follows: the first convolution residual block and the second convolution residual block are both based on 3D-ResNet, and the original 3D convolution kernel 3*3*3 is replaced by separable convolution. Separate convolution is to split the 3D convolution kernel into 1*3*3 and 3*1*1 convolution kernels.

[0027] The present invention is further configured such that: both the first convolution residual block and the second convolution residual block repeat a convolution operation with a step size of 1 multiple times, until the last repetition, a convolution operation with a step size of 2 is configured to reduce the size of the image.

[0028] The present invention is further configured to: after classifying the fusion result and outputting the classification information of the input video stream, include:

[0029] The classification information of the input video stream is obtained and submitted to the loss optimization module for calculation and optimization of the classification loss.

[0030] The present invention is further configured to: obtain a data set formed based on the input video stream, and the data set includes three parts: a training set, a verification set and a test set. The training set is used to train the network model, the verification set is used to verify the performance of the network model, and the test set is used to evaluate the generalization ability of the network model.

[0031] In a second aspect, a device for identifying abnormal driving behavior based on a video stream neural network is provided, comprising:

[0032] A video stream sampling module is used to extract a first video frame sequence and a second video frame sequence from the acquired input video stream at different sampling rates;

[0033] a feature extraction module, configured to extract convolution features from the first video frame sequence and the second video frame sequence respectively;

[0034] A feature fusion module, configured to fuse features extracted from the first video frame sequence and the second video frame sequence respectively at different spatiotemporal scales;

[0035] The classification module is used to classify the fusion result and output classification information of the input video stream.

[0036] The present invention is further configured to include a loss optimization module for obtaining classification information of the input video stream and submitting the classification information to the loss optimization module for calculation and optimization of classification loss.

[0037] The present invention is further configured as follows: it includes a processing chip and a storage medium, one or more programs are stored in the storage medium and configured to be executed by the processing chip, and the one or more programs are used to drive the processing chip structure to execute the abnormal driving behavior identification method as described in any one of claims 1-11.

[0038] In a third aspect, a readable computer medium is provided, comprising one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the abnormal driving behavior identification method according to any one of claims 1-11.

[0039] Compared with the prior art, the present invention has the following advantages:

[0040] 1. This application samples different numbers of frames of video from the input video stream at different sampling rates and inputs them into the first convolution branch and the second convolution branch respectively. This avoids determining the position of a person in the entire input video stream based on the position of the person in a single frame of video, making the positioning of the person more accurate.

[0041] 2. This application utilizes 3D convolutional neural networks for both the first and second convolutional branches, which can identify the temporal and spatial features of the input video stream, avoiding the loss of temporal features when converting the input video stream into an optical flow sequence. The first convolutional branch uses a slower frame rate to obtain spatial features from the input video stream. Simultaneously, the optical flow sequence neural network branch is replaced with a 3D convolutional neural network branch with a higher frame rate to obtain the temporal behavior features of the input video stream. This eliminates the need for dense optical flow sequences, avoids the loss of temporal features, improves the accuracy of abnormal driving behavior, and reduces the requirements of the recognition method for the imaging environment.

[0042] 3. This application performs multi-scale fusion of the features of the first and second convolution branches, avoiding the loss of features at different depths. Networks of different depths all fuse the spatiotemporal features of the previous scale. Finally, the features of the last scale of the two branches are fused again, so that the subsequent convolution process of the first convolution branch also takes into account the temporal features of the previous scale, making the features extracted by the convolution operation more accurate and reliable.

[0043] 4. By using residual connections in the network model, a larger receptive field is achieved, which allows for training deeper networks and extracting more accurate feature details from the input video stream; at the same time, it reduces the loss in the convolution process without increasing the number of parameters required for convolution.

Brief Description of the Drawings

[0044] Figure 1 is a flowchart of the steps of this embodiment;

[0045] Figure 2 is a flow chart of the identification method of this embodiment;

[0046] Figure 3 : is a network structure diagram of the residual block of this embodiment;

[0047] Figure 4 This is a structural diagram of the fused convolution block of this embodiment.

[0048] Figure numerals: 110, video stream sampling module; 120, feature extraction module; 130, feature fusion module; 140, classification module; 150, loss optimization module; 151, loss calculation module; 152, optimizer. [Specific implementation method]

[0049] The following describes embodiments of the present invention in detail, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0050] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0051] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0052] As will be understood by those skilled in the art, the terms "terminal" and "terminal device" as used herein include both devices having a wireless signal receiver, i.e., devices having only a wireless signal receiver without transmitting capability, and devices having receiving and transmitting hardware capable of two-way communication over a two-way communication link. Such devices may include: cellular or other communication devices having a single-line display, a multi-line display, or a cellular or other communication device without a multi-line display; a Personal Communication System (PCS) which may combine voice, data processing, fax, and / or data communication capabilities; a Personal Digital Assistant (PDA) which may include a radio frequency receiver, a pager, Internet / Intranet access, a web browser, a notepad, a calendar, and / or a GPS (Global Positioning System) receiver; and conventional laptop and / or palmtop computers or other devices having and / or including a radio frequency receiver. As used herein, the terms "terminal" or "terminal device" may be portable, transportable, installed in a vehicle (air, sea, and / or land), or adapted and / or configured to operate locally, and / or in a distributed manner, at any other location on Earth and / or in space. The terms "terminal" or "terminal device" as used herein may also refer to a communication terminal, an Internet terminal, or a music / video playback terminal, such as a PDA, an MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, a set-top box, or other device.

[0053] like Figure 1 and Figure 2 As shown, the present invention discloses a method for identifying abnormal driving behavior based on a video stream neural network, which specifically includes the following steps:

[0054] S1. Extracting a first video frame sequence and a second video frame sequence from an acquired input video stream at different sampling rates;

[0055] S2, extracting convolution features from the first video frame sequence and the second video frame sequence respectively;

[0056] S3, for different spatiotemporal scales, fusing the features extracted from the first video frame sequence and the second video frame sequence respectively;

[0057] S4. Classify the fusion result and output classification information of the input video stream.

[0058] Optionally, in S1, extracting a first video frame sequence and a second video frame sequence from the acquired input video stream at different sampling rates, specifically comprising:

[0059] Extracting a video stream from a video device or a video stream already extracted by others, organizing it into smaller video clips, and saving them in chronological order as an input video stream; classifying abnormal driving behaviors in the video clips; reading the input video stream and submitting it to the video stream sampling module 110 for sampling at different sampling rates to obtain video frame sequences with different frame numbers;

[0060] A first video frame sequence and a second video frame sequence are obtained, each frame in the first video frame sequence and the second video frame sequence is processed into an image with consistent width and height, and input into a dual-branch 3D convolutional network model; wherein the number of frames in the first video frame sequence is less than the number of frames in the second video frame sequence.

[0061] Specifically, when the input video stream is set to 64 frames, the input video stream is sampled at a sampling rate of 8 Hz to obtain a first video frame sequence after sampling, and the number of frames of the first video frame sequence is 8 frames; then, the input video stream is sampled at a sampling rate of 2 Hz to obtain a second video frame sequence after sampling, and the number of frames of the second video frame sequence is 32 frames. Both the first video frame sequence and the second video frame sequence are preprocessed, that is, the video clips are cut into frames using the computer vision library OpenCV, and each image frame in the first video frame sequence and the second video frame sequence is deformed into an image with a width and height of 224*224.

[0062] Optionally, in S2, convolution features are extracted from the first video frame sequence and the second video frame sequence respectively, and the specific steps include:

[0063] Obtaining a first video frame sequence and a second video frame sequence, and inputting the sequences into a first convolution branch and a second convolution branch, respectively; wherein the frame rate of the first convolution branch is greater than the frame rate of the second convolution branch;

[0064] In the first convolution branch, a 3D convolution block and a plurality of first convolution residual blocks are sequentially used to perform convolution operations to extract spatial features, and the spatial features of the first video frame sequence are extracted in the first convolution branch.

[0065] Specifically, the first video frame sequence is input into the first convolution branch, and the first convolution branch includes a 3D convolution block, a first convolution residual block 1, a first convolution residual block 2 and a first convolution residual block 3 in sequence. A fusion convolution block is provided between the 3D convolution block and the first convolution residual block of the above-mentioned first convolution branch, and between the first convolution residual block and the first convolution residual block, so that the convolution features in the first convolution branch have both temporal and spatial dimensional features, thereby improving the recognition performance; wherein, the first video frame sequence is first input into the 3D convolution block, and the 3D convolution block includes a convolution layer and a maximum pooling layer, and the parameters of the convolution layer are The parameters are set to keral:[1,7,7], stride:[1,2,2] and padding:[0,3,3], that is, a convolution layer with a convolution kernel of 1*7*7 is used to obtain a larger receptive field, ensuring the completeness of local information as much as possible; at the same time, the convolution features processed by the convolution layer are submitted to the maximum pooling layer for downsampling, where the parameters of the maximum pooling layer are set to keral:[1,3,3], stride:[1,2,2] and padding:[0,1,1] to reduce the dimension of the convolution features, simplify the network complexity and improve the robustness of the features.

[0066] In the second convolution branch, 3D convolution blocks and multiple second convolution residual blocks are used in sequence to perform convolution operations to extract temporal features;

[0067] Specifically, the second video frame sequence is input into the second convolution branch, and the second convolution branch includes a 3D convolution block, a second convolution residual block 1, a second convolution residual block 2, and a second convolution residual block 3 in sequence; wherein, the second video frame sequence is first input into the 3D convolution block, and the 3D convolution block includes a convolution layer and a maximum pooling layer. The convolution parameters of the convolution layer are set to keral: [1, 7, 7], stride: [1, 2, 2], and padding: [0, 3, 3], that is, a larger receptive field is obtained through a convolution layer with a convolution kernel of 1*7*7, so as to ensure the completeness of local information as much as possible; at the same time, the convolution features processed by the convolution layer are submitted to the maximum pooling layer for downsampling, wherein the downsampling parameters of the maximum pooling layer are set to keral: [1, 3, 3], stride: [1, 2, 2], and padding: [0, 1, 1], which reduces the dimensionality of the convolution features, simplifies the network complexity, and improves the robustness of the features.

[0068] Optionally, in S3, features extracted from the first video frame sequence and the second video frame sequence are fused respectively for different spatiotemporal scales, and the specific steps include:

[0069] Acquire spatial features and temporal features, and fuse the temporal features into the spatial features of the same scale; obtain the fused spatiotemporal features as the features of subsequent convolution operations of the first convolution branch.

[0070] Specifically, first, the spatial features of the first video frame are extracted from the 3D convolution block in the first convolution branch, and the temporal features of the second video frame are extracted from the 3D convolution block in the second convolution branch. At this time, the temporal features and spatial features are input into the fusion convolution block between the 3D convolution block and the first convolution residual block 1 for fusion, thereby obtaining spatiotemporal features with both spatial and temporal dimensions.

[0071] Then, the spatiotemporal features in the first convolution branch are input into the subsequent first convolution residual block 1 to continue the convolution operation; at the same time, the spatial features in the second convolution branch are further input into the second convolution residual block 1 for convolution operation. The above convolution residual blocks are all based on 3D-ResNet, in which the convolution features are circulated 3 times in the first convolution residual block 1 and the second convolution residual block 1, respectively, the convolution features are circulated 6 times in the first convolution residual block 2 and the second convolution residual block 2, respectively, and the convolution features are circulated 3 times in the first convolution residual block 3 and the second convolution residual block 3.

[0072] For example, if the spatiotemporal feature is subjected to three convolution operations in the first convolution residual block 1, as an example:

[0073] The first convolutional residual block 1 is based on 3D-ResNet and uses separable convolution to replace the original 3D convolution kernel 3*3*3. Separable convolution splits the 3D convolution kernel into 1*3*3 and 3*1*1 convolution kernels, and connects the convolution kernels of 1*3*3 and 3*1*1 in series as the first residual branch of the first residual block 1; at the same time, the convolution kernel of 1*1*1 is used as the second residual branch of the first residual block 1, and the convolution result of the second residual branch is used as the equivalent mapping of the convolution result of the first residual branch.

[0074] First, if Figure 3 and Figure 4As shown, the spatiotemporal features are input into the first residual branch and the second residual branch respectively. The convolution parameters of the spatiotemporal features in the first residual branch are as follows: the first cycle: convolution 1 (kernel: 1*1*1, step size: 1*1*1), convolution 2 (kernel: 1*3*3, step size: 1*1*1), convolution 3 (kernel: 3*1*1, step size: 1*1*1); the second cycle: convolution 1 (kernel: 1*1*1, step size: 1*1*1), convolution 2 (kernel: 1*3*3, step size: 1*1*1), convolution 3 (kernel: 3*1*1, step size: 1*1*1); the third cycle: convolution 1 (kernel: 1*1*1, step size: 1*1* 1), convolution 2 (kernel: 1*3*3, step size: 1*1*1), convolution 3 (kernel: 3*1*1, step size: 1*2*2). When the last cycle of the first residual branch is completed, the step size of the convolution operation is changed to 2, so that the image size output by the first residual branch is halved. After a 224*224 picture is input into the network, the feature size extracted by the convolution module is reduced successively. The feature sizes extracted by the 3D convolution block, the first convolution residual block 1, the first convolution residual block 2 and the first convolution residual block 3 are 112*112, 56*56, 28*28 and 14*14 respectively, forming features of various scales in the first convolution branch.

[0075] The convolution parameters of the spatiotemporal features in the above-mentioned second residual branch are as follows: the first cycle: convolution (kernel: 1*1*1, step size: 1*1*1); the second cycle: convolution (kernel: 1*1*1, step size: 1*1*1); the third cycle: convolution (kernel: 1*1*1, step size: 1*2*2). In the last cycle of the second residual branch, the step size of the convolution operation is changed to 2, so that the image size output by the second residual branch is halved. After a 224*224 picture is input into the network, the feature size extracted by the convolution module is reduced successively. The feature sizes extracted by the 3D convolution block, the second convolution residual block 1, the second convolution residual block 2, and the second convolution residual block 3 are 112*112, 56*56, 28*28, and 14*14, respectively, to form features of various scales in the second convolution branch.

[0076] After the first residual branch and the second residual branch complete a cycle, the convolution result of the first residual branch is stacked with the convolution result of the second residual branch to determine whether the convolution result of the first residual branch is ideal. If the loss is larger than that of the second residual branch, the convolution weight of the first residual branch can be set to 0 to reduce the loss in multiple convolution feature extractions, which is conducive to optimizing feature extraction and increasing the depth to improve the accuracy of feature extraction. The above-mentioned first residual block 1 uses skip connections to alleviate the gradient vanishing problem caused by increasing the depth in deep neural networks.

[0077] Optionally, in S4, the fusion result is classified and classification information of the input video stream is output, including:

[0078] Obtain the spatiotemporal features output from the last first convolution residual block, and obtain the temporal features output from the last second convolution residual block, fuse the spatiotemporal features and the temporal features as features input to the classification module 140, classify the features in the classification module 140, and output classification information of the input video stream.

[0079] Optionally, in S4, after classifying the fusion result and outputting classification information of the input video stream, the following steps are included:

[0080] The classification information of the input video stream is obtained and submitted to the loss optimization module 150 for calculation and optimization of the classification loss. The loss optimization module 150 includes a loss calculation module 151 and an optimizer 152.

[0081] Specifically, video streams are extracted from video devices or video streams that have been extracted by others, organized into smaller video clips, and the classification of abnormal driving behaviors in the video clips is labeled to obtain a data set formed based on the input video stream.

[0082] The dataset consists of three parts: a training set, a validation set, and a test set. The training set is used to train the network model, the validation set is used to verify the performance of the trained model, and the test set is used to evaluate the generalization ability of the network model. The training set and validation set are respectively input into the above-mentioned model to perform dual-branch 3D convolutional network model training. During the training process, features are extracted and multi-scale features are fused. Finally, spatial features and temporal features are fused on the channel to obtain a feature vector. The features are input into the classification module 140. The classification module 140 can be a Softmax classifier, which outputs the classification score of the feature. At the same time, the classification score is combined with the classification label of the input video stream to calculate the classification loss. The images in the training set and validation set are iteratively trained using the SGD optimizer until the value of the multi-task fusion loss function L continues to converge and stabilize, thereby obtaining a trained dual-branch 3D convolutional network model. The test set is then input into the dual-branch 3D convolutional network model to obtain the final driving behavior recognition result.

[0083] like Figure 2 As shown, an embodiment of the present invention provides an abnormal driving behavior recognition device based on a video stream neural network, comprising:

[0084] The video stream sampling module 110 is configured to extract a first video frame sequence and a second video frame sequence from the acquired input video stream at different sampling rates;

[0085] A feature extraction module 120 is configured to extract convolution features from the first video frame sequence and the second video frame sequence respectively;

[0086] A feature fusion module 130 is configured to fuse features extracted from the first video frame sequence and the second video frame sequence respectively at different spatiotemporal scales;

[0087] The classification module 140 is used to classify the fusion result and output classification information of the input video stream.

[0088] It also includes a loss optimization module 150 for obtaining classification information of the input video stream and submitting it to the loss optimization module for calculation and optimization of classification loss.

[0089] Among them, the loss optimization module 150 includes a loss calculation module 151 and an optimizer 152. The loss calculation module 151 is used to calculate the feature loss of classification information, and the optimizer 152 is used to iteratively train the images in the training set and the validation set until the value of the multi-task fusion loss function L continues to converge and become stable.

[0090] The embodiment of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of any one of the telecommunications billing methods described above when executed by a processor. The storage medium includes but is not limited to any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROM (Read-Only Memory), RAM (Random Access Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory, magnetic cards, or optical cards. That is, the storage medium includes any medium that can be used by a device (e.g., a computer) to store or transmit information in a readable form. It can be a read-only memory, a disk, or an optical disk, etc.

[0091] In summary, the abnormal driving behavior identification method and related device based on video stream neural network provided by this application have the following beneficial effects:

[0092] When a car is driving at a low speed, the driver tends to relax his vigilance and make many large-arc movements. In the convolutional neural network branch of a single-frame image, only a single-frame image is extracted from the input video stream, and there is a large error in the positioning of the human body. In this application, the input video stream is sampled at different sampling rates to obtain a first video frame sequence and a second video frame sequence of multiple frames, respectively, thereby improving the accuracy of positioning the behavior of the characters in the image; at the same time, the multi-scale fusion in the convolution process enables the spatial features extracted by the first convolution branch to be integrated with the temporal features, making the feature extraction of the first convolution branch more reliable and reducing the loss of different depth features.

[0093] There is no need to use dense optical flow sequences to avoid masking the driver's subtle abnormal driving behaviors. At the same time, during the driving process, the vehicle is constantly shaking, and the scene outside the window is changing rapidly. If a dense optical flow map is generated, it will mask the driver's subtle movements, resulting in the inability to reflect the driver's driving behavior in the optical flow map. Both branches use 3D convolutional neural networks, which not only overcomes the problems of converting the input video stream into an optical flow sequence, but also increases the depth of the network without introducing a large number of network parameters, and has a good effect on human behavior recognition in frame images.

[0094] The model uses residual connections to facilitate in-depth feature extraction, making the extracted features more detailed. At the same time, it reduces the model's feature loss in feature extraction and avoids the loss of input video stream features. This allows it to identify more subtle abnormal driving behaviors, provide timely warnings of abnormal driving behaviors, and reduce the accident rate.

[0095] Those skilled in the art will appreciate that the steps, measures, and schemes in the various operations, methods, and processes discussed in the present invention may be interchanged, modified, combined, or deleted. Furthermore, other steps, measures, and schemes in the various operations, methods, and processes discussed in the present invention may also be interchanged, modified, rearranged, decomposed, combined, or deleted. Furthermore, steps, measures, and schemes in the prior art that are similar to those disclosed in the present invention may also be interchanged, modified, rearranged, decomposed, combined, or deleted.

[0096] The above descriptions are only partial embodiments of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and modifications without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for identifying abnormal driving behavior based on video stream neural network, characterized in that: include: Extracting a first video frame sequence and a second video frame sequence respectively at different sampling rates from the acquired input video stream; Extracting convolution features from the first video frame sequence and the second video frame sequence respectively; the number of frames in the first video frame sequence is smaller than the number of frames in the second video frame sequence; fusing the features extracted from the first video frame sequence and the second video frame sequence respectively for different spatiotemporal scales; Classifying the fusion result and outputting classification information of the input video stream; Before extracting convolution features from the first video frame sequence and the second video frame sequence respectively, the specific steps include: Processing each frame in the first video frame sequence and the second video frame sequence into an image with consistent width and height; The extracting convolution features of the first video frame sequence and the second video frame sequence respectively includes: Obtaining the first video frame sequence and the second video frame sequence and inputting them into a first convolution branch and a second convolution branch respectively; In the first convolution branch, a 3D convolution block and a plurality of first convolution residual blocks are sequentially used to perform convolution operations to extract spatial features; In the second convolution branch, a 3D convolution block and a plurality of second convolution residual blocks are sequentially used to perform convolution operations to extract temporal features; Acquire spatial features and temporal features, and fuse the temporal features into the spatial features of the same scale to obtain fused spatiotemporal features; The fused spatiotemporal features are obtained as features for subsequent convolution operations of the first convolution branch.

2. The abnormal driving behavior identification method according to claim 1, characterized in that: The frame rate of the first convolution branch is greater than the frame rate of the second convolution branch.

3. The abnormal driving behavior identification method according to claim 2, characterized in that: After obtaining the fused spatiotemporal features as features for subsequent convolution operations of the first convolution branch, the specific steps include: Obtain the spatiotemporal features output from the last first convolution residual block, and obtain the temporal features output from the last second convolution residual block, and fuse the spatiotemporal features and the temporal features to serve as features input to the classification module.

4. The abnormal driving behavior identification method according to claim 3, characterized in that: In the first convolution branch, a 3D convolution block and a plurality of first convolution residual blocks are sequentially used to perform convolution operations to extract spatial features, including: The 3D convolution block includes a convolution layer and a maximum pooling layer, wherein the convolution layer adopts 1 7 The convolution kernel of 7 is used to reduce the dimension of the convolution feature and submit it to the maximum pooling layer for downsampling.

5. The abnormal driving behavior identification method according to claim 4, characterized in that: The first convolution residual block and the second convolution residual block are both based on 3D-ResNet, and separate convolution is used to replace the original 3D convolution kernel 3. 3 3. Separate convolution is to split the 3D convolution kernel into 1 3 3 and 3 1 The convolution kernel is 1.

6. The abnormal driving behavior identification method according to claim 5, characterized in that: Both the first convolution residual block and the second convolution residual block repeat a convolution operation with a step size of 1 multiple times, until the last repetition, a convolution operation with a step size of 2 is configured to reduce the size of the image.

7. The abnormal driving behavior identification method according to claim 1, characterized in that: After classifying the fusion result and outputting the classification information of the input video stream, the method further includes: The classification information of the input video stream is obtained and submitted to the loss optimization module for calculation and optimization of the classification loss.

8. The abnormal driving behavior identification method according to claim 7, characterized in that: A data set formed based on the input video stream is obtained, and the data set includes three parts: a training set, a validation set, and a test set. The training set is used to train the network model, the validation set is used to verify the performance of the network model, and the test set is used to evaluate the generalization ability of the network model.

9. A device for identifying abnormal driving behavior based on a video stream neural network, characterized in that: include: A video stream sampling module is used to extract a first video frame sequence and a second video frame sequence from the acquired input video stream at different sampling rates; a feature extraction module, configured to extract convolution features from the first video frame sequence and the second video frame sequence respectively; the number of frames in the first video frame sequence is smaller than the number of frames in the second video frame sequence; a feature fusion module, configured to fuse features extracted from the first video frame sequence and the second video frame sequence respectively at different spatiotemporal scales; A classification module, configured to classify the fusion result and output classification information of the input video stream; a frame processing module, configured to process each frame in the first video frame sequence and the second video frame sequence into an image with consistent width and height; The feature extraction module is further configured to: Obtaining the first video frame sequence and the second video frame sequence and inputting them into a first convolution branch and a second convolution branch respectively; In the first convolution branch, a 3D convolution block and a plurality of first convolution residual blocks are sequentially used to perform convolution operations to extract spatial features; In the second convolution branch, a 3D convolution block and a plurality of second convolution residual blocks are sequentially used to perform convolution operations to extract temporal features; Acquire spatial features and temporal features, and fuse the temporal features into the spatial features of the same scale to obtain fused spatiotemporal features; The fused spatiotemporal features are obtained as features for subsequent convolution operations of the first convolution branch.

10. The abnormal driving behavior recognition device according to claim 9, characterized in that: It also includes a loss optimization module for obtaining classification information of the input video stream and submitting it to the loss optimization module for calculation and optimization of classification loss.

11. A device for identifying abnormal driving behavior based on video stream neural network, characterized in that: It includes a processing chip and a storage medium, one or more programs are stored in the storage medium and are configured to be executed by the processing chip, and the one or more programs are used to drive the processing chip to execute the abnormal driving behavior identification method as described in any one of claims 1-8.

12. A computer-readable medium, characterized in that The one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the abnormal driving behavior identification method according to any one of claims 1-8.

Images