Driver Behavior Monitoring Method Based on Online Video Understanding Network

Abstract

The present invention provides a driver behavior monitoring method based on an online video understanding network, including: acquiring online video data; inputting the current frame image data into the feature extraction network in the online video understanding network to obtain the feature extraction result of the current frame image data; covering the feature extraction result of the current frame image data with a set part of the feature extraction results of historical frame image data to obtain the combined feature extraction result of the current frame image data; inputting the combined feature extraction result into the target prediction network in the online video understanding network to obtain the event monitoring task score of the current frame image data; and obtaining the driver behavior monitoring result according to the event monitoring task score. The present invention combines the features of each frame with the features of all historical frames without consuming too much computing resources, and performs target prediction based on the combined features, thereby realizing lightweight, high-efficiency, fast iteration, and good continuity of driver behavior monitoring.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning, and in particular to a driver behavior monitoring method based on an online video understanding network. Background Art

[0002] Identifying the driver's behavior inside the cockpit is crucial for ensuring the driver's safety. The algorithm needs to monitor and identify the driver's behavior in real time, and give timely alarms to reduce the driver's unsafe driving behaviors, such as smoking, making phone calls, closing eyes due to fatigue, yawning, camera occlusion, distraction, direct sunlight, whether the camera angle is correct, and whether the driver is within the camera range.

[0003] To achieve the purpose of monitoring the driver's behavior inside the cabin, the prior art provides some algorithms that utilize facial key points, smoking detection / classification, or mobile phone detection / classification, and combine post-processing logic to judge and identify behaviors.

[0004] However, the prior art has problems such as complex post-processing logic adjustment and high computational resource requirements, making it difficult to deploy the driver behavior monitoring model on the vehicle mobile device side, and unable to balance the accuracy rate and processing speed of the monitoring results.

[0005] Therefore, how to provide a lightweight driver behavior monitoring method to achieve vehicle-mounted mobile device monitoring with low computational resource requirements, fast processing, and high accuracy has become an urgent problem to be solved in the industry. Summary of the Invention

[0006] The present invention provides a driver behavior monitoring method based on an online video understanding network to solve the defect in the prior art that it is difficult to deploy the driver behavior monitoring model on the vehicle mobile device side and unable to balance the accuracy rate and processing speed of the monitoring results, and to achieve driver behavior monitoring with low computational resource requirements, fast processing, and high accuracy.

[0007] The present invention provides a driver behavior monitoring method based on an online video understanding network, including:

[0008] Obtaining online video data; the online video data includes image data of at least two consecutive frames;

[0009] Inputting the current frame image data into the feature extraction network in the online video understanding network to obtain the feature extraction result of the current frame image data;

[0010] Covering a set part of the feature extraction result of the historical frame image data with the feature extraction result of the current frame image data to obtain the combined feature extraction result of the current frame image data;

[0011] Input the combined feature extraction result into the target prediction network in the online video understanding network to obtain the event monitoring task score of the current frame image data;

[0012] Based on the event monitoring task score, obtain the driver behavior monitoring result;

[0013] The online video understanding network is trained through online video samples.

[0014] According to a driver behavior monitoring method based on an online video understanding network provided by the present invention, the step of obtaining the combined feature extraction result of the current frame image data by covering a set part of the feature extraction result of the historical frame image data with the feature extraction result of the current frame of the image data includes:

[0015] If it is determined that the serial number i of the current frame is 1, then obtain the feature extraction result of the reference image through the feature extraction network as the historical feature set; the reference image is a completely black image with the same size as the image data;

[0016] If it is determined that the serial number i of the current frame is not 1, then obtain the feature extraction result of the (i - 1)-th frame image data through the feature extraction network as the historical feature set;

[0017] Cover the feature extraction result of the i-th frame of the image data with a set part of the historical feature set to obtain the combined feature extraction result of the i-th frame of the image data.

[0018] According to a driver behavior monitoring method based on an online video understanding network provided by the present invention, the step of covering the feature extraction result of the i-th frame of the image data with a set part of the historical feature set to obtain the combined feature extraction result of the i-th frame of the image data includes:

[0019] Cover the feature extraction result of the i-th frame of the image data with the first combined historical feature set to obtain the combined feature extraction result of the i-th frame of the image data;

[0020] The first combined historical feature includes the j-th sub-feature in the historical feature set; the j-th sub-feature is the result output by the j-th feature extraction layer of the feature extraction network; where j is an element in the combined serial number set, and the combined serial number set is a proper subset of the feature extraction layer serial number set.

[0021] According to a driver behavior monitoring method based on an online video understanding network provided by the present invention, the step of covering the feature extraction result of the i-th frame of the image data with a set part of the historical feature set to obtain the combined feature extraction result of the i-th frame of the image data includes:

[0022] Cover the feature extraction result of the i-th frame of the image data with the second combined historical feature set to obtain the combined feature extraction result of the i-th frame of the image data;

[0023] The second combined historical feature includes a set part of sub-features in the historical feature set; the sub-feature is the result output by a single feature extraction layer in the feature extraction network.

[0024] According to a driver behavior monitoring method based on an online video understanding network provided by the present invention, the event monitoring tasks include a driver smoking monitoring task, a driver handheld phone call monitoring task, a driver distracted attention monitoring task, a driver fatigue monitoring task, a direct sunlight monitoring task, a video acquisition device occlusion monitoring task, and a video acquisition device angle anomaly monitoring task.

[0025] According to a driver behavior monitoring method based on an online video understanding network provided by the present invention, the step of obtaining the driver behavior monitoring result according to the event monitoring task score includes:

[0026] Judge the event monitoring task score. If the event monitoring task scores of a set number of consecutive frames are all greater than the threshold, then regard the event occurrence as the driver behavior monitoring result; otherwise, regard the event non-occurrence as the driver behavior monitoring result.

[0027] The present invention also provides a driver behavior monitoring system based on an online video understanding network, including:

[0028] A data acquisition module for acquiring online video data; the online video data includes image data of at least two consecutive frames;

[0029] A feature extraction module for inputting the current frame image data into a feature extraction network in the online video understanding network to obtain the feature extraction result of the current frame image data;

[0030] A feature coverage module for covering the feature extraction result of the current frame of the image data with a set part of the feature extraction result of the historical frame image data to obtain the combined feature extraction result of the current frame image data;

[0031] A target prediction module for inputting the combined feature extraction result into a target prediction network in the online video understanding network to obtain the event monitoring task score of the current frame image data;

[0032] A monitoring result module for obtaining the driver behavior monitoring result according to the event monitoring task score;

[0033] The online video understanding network is trained through online video samples.

[0034] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of any one of the above-mentioned driver behavior monitoring methods based on an online video understanding network are implemented.

[0035] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of any one of the above-mentioned driver behavior monitoring methods based on an online video understanding network are implemented.

[0036] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the steps of any one of the above-mentioned driver behavior monitoring methods based on an online video understanding network are implemented.

[0037] The driver behavior monitoring method based on an online video understanding network provided by the present invention combines the features of each frame with the features of all historical frames by covering a part of the historical frame feature extraction result onto the feature extraction result of the current frame, without consuming excessive computing resources, and performs target prediction based on the combined features, thereby realizing lightweight, high-efficiency, fast iteration, and good continuity of in-cabin driver behavior monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0039] Figure 1 is a schematic flowchart of the driver behavior monitoring method based on an online video understanding network provided by the present invention;

[0040] Figure 2 is a schematic structural diagram of the online video understanding network provided by an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of feature combination based on channels provided by an embodiment of the present invention;

[0042] Figure 4 is a schematic structural diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0043] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention in conjunction with the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts fall within the protection scope of the present invention.

[0044] The following will describe Figures 1-3 the driver behavior monitoring method based on the online video understanding network of the present invention.

[0045] As Figure 1 shown, an embodiment of the present invention provides a driver behavior monitoring method based on an online video understanding network, including

[0046] Step 101, obtaining online video data; the online video data includes image data of at least two consecutive frames;

[0047] Step 103, inputting the current frame image data into the feature extraction network in the online video understanding network to obtain the feature extraction result of the current frame image data;

[0048] Step 105, covering the feature extraction result of the image data of the current frame with a set part of the feature extraction results of the historical frame image data to obtain the combined feature extraction result of the current frame image data;

[0049] Step 107, inputting the combined feature extraction result into the target prediction network in the online video understanding network to obtain the event monitoring task score of the current frame image data;

[0050] Step 109, obtaining the driver behavior monitoring result according to the event monitoring task score;

[0051] Figure 2 shows a schematic structural diagram of the online video understanding network provided in this embodiment. The online video understanding network is trained through online video samples.

[0052] The execution subject of this embodiment can be a monitoring program running in a vehicle-side mobile device; the online video data obtained in step 101 refers to the in-cabin driver video obtained in real time by the vehicle mobile terminal.

[0053] The fact that the online video data in this embodiment includes image data of at least two consecutive frames can be understood as that the method of this embodiment needs to be based on at least two frames of data, namely the current frame and the previous frame of the in-cabin driver video obtained in real time by the vehicle mobile terminal, for driver behavior monitoring based on feature combination.

[0054] In actual use, on the premise that the video acquisition device has been turned on, since the combined feature extraction result of the image data of the i-th frame includes a part of the image data feature extraction result of the (i - 1)-th frame, and the image data feature extraction result of the (i - 1)-th frame is also a combined feature extraction result, that is, the image data feature extraction result of the (i - 1)-th frame also includes a part of the image data feature extraction result of the (i - 2)-th frame, and so on. It can be known that in this embodiment, the event monitoring task score obtained for each frame of image data is based on the combination of the features of this frame of image and all the frames before this frame, and is obtained by inputting into the target prediction network.

[0055] Through this setting, the target prediction result obtained by combining historical frame data can significantly reduce the misjudgment caused by adverse reasons such as the noise of single-frame image data.

[0056] At the same time, the proportion of the feature extraction result of the historical frame farther away from the current frame in the combined feature extraction result of the current frame is smaller, which dilutes the features of the earlier frames and can avoid the adverse impact of historical frames on the determination result of the current frame.

[0057] It is worth mentioning that although the feature extraction based on the fusion of adjacent frames can also achieve a similar effect of reducing misjudgment, the adjacent frame fusion scheme (that is, directly fusing the historical frame features and the current frame features) has the following two disadvantages compared with this embodiment:

[0058] 1. The feature dilution effect of the earlier frames is relatively weaker, which may cause an adverse impact of historical frames on the target prediction result of the current frame;

[0059] 2. The amount of calculation required for the fusion process of historical frame features and current frame features is much larger than the method of this embodiment (that is, covering a set part of the feature extraction result of historical frame image data to the feature extraction result of the image data of the current frame to obtain the combined feature extraction result of the current frame image data), so that the adjacent frame fusion scheme is more difficult to deploy on in-vehicle mobile devices and it is more difficult to balance the accuracy and processing speed of the monitoring result.

[0060] The beneficial effects of this embodiment are as follows:

[0061] In this embodiment, by covering a part of the historical frame feature extraction result to the feature extraction result of the current frame, on the premise of not consuming too much computing resources, the features of each frame are combined with the features of all historical frames, and target prediction is performed based on the combined features, thus realizing lightweight, high-efficiency, fast iteration, and good continuity of in-cabin driver behavior monitoring.

[0062] According to the above embodiment, in this embodiment:

[0063] The step of obtaining the combined feature extraction result of the current frame image data by covering the set part of the feature extraction result of the historical frame image data with the feature extraction result of the current frame includes:

[0064] If it is determined that the sequence number i of the current frame is 1, the feature extraction result of the reference image is obtained through the feature extraction network as the historical feature set; the reference image is a completely black image with the same size as the image data;

[0065] If it is determined that the sequence number i of the current frame is not 1, the feature extraction result of the (i - 1)-th frame image data is obtained through the feature extraction network as the historical feature set;

[0066] The set part of the historical feature set is used to cover the feature extraction result of the i-th frame of the image data to obtain the combined feature extraction result of the i-th frame of the image data.

[0067] The above provides the feature combination steps for the first frame.

[0068] In this embodiment, the combination of features can be achieved in the following two ways.

[0069] In the first way, the step of obtaining the combined feature extraction result of the i-th frame of the image data by covering the set part of the historical feature set with the feature extraction result of the i-th frame of the image data includes:

[0070] The first combined historical feature set is used to cover the feature extraction result of the i-th frame of the image data to obtain the combined feature extraction result of the i-th frame of the image data;

[0071] The first combined historical feature includes the j-th sub-feature in the historical feature set; the j-th sub-feature is the result output by the j-th feature extraction layer of the feature extraction network; where j is an element in the combination sequence number set, and the combination sequence number set is a proper subset of the feature extraction layer sequence number set.

[0072] The first way can be simply understood as:

[0073] First, according to the number and connection relationship of the feature extraction layers in the feature extraction network, select some of the feature extraction layers with serial numbers as the combined basic layers, and use the serial number set of the combined basic layers as the combination sequence number set;

[0074] Subsequently, the feature extraction result of the (i - 1)-th frame in the combined basic layer is covered into the feature extraction result of the i-th frame corresponding to the part of the combined basic layer.

[0075] That is to say, the first way selects a part of the feature extraction layers for feature combination.

[0076] In a preferred solution of this embodiment, the selection of the feature extraction layer should consider both deep features and shallow features simultaneously to achieve a better feature combination effect, which is beneficial to improving the accuracy of the final target prediction. For example, for a feature extraction network including 12 feature extraction layers, select the 8 feature extraction layers from the 3rd to the 10th as the combined basic layers, and the combined serial number set includes 8 elements from 3 to 10.

[0077] As Figure 3 shown, in the second method, the step of obtaining the combined feature extraction result of the i-th frame of the image data by covering the feature extraction result of the i-th frame of the image data with a set part of the historical feature set includes:

[0078] Cover the feature extraction result of the i-th frame of the image data with the second combined historical feature set to obtain the combined feature extraction result of the i-th frame of the image data;

[0079] The second combined historical feature includes a set part of the sub-features in the historical feature set; the sub-feature is the result output by a single feature extraction layer in the feature extraction network.

[0080] The second method can be simply understood as:

[0081] First, in the feature extraction result of the (i - 1)-th frame, for the output feature of each feature extraction layer, select a part as the combined basic feature;

[0082] Subsequently, for the output feature of each feature extraction layer in the feature extraction result of the i-th frame, cover it with the selected part of the output feature of the corresponding feature extraction layer of the (i - 1)-th frame.

[0083] That is to say, in the second method, a part of the output feature of each feature extraction layer is selected for feature combination.

[0084] In a preferred solution of this embodiment, the selection of the output feature of each feature extraction layer can be based on channels, and a part of the channels of each feature extraction layer, such as the features output by 1 / 4 of the channels, is selected for feature combination.

[0085] The above two methods can be used separately or in combination.

[0086] The beneficial effects of this embodiment are as follows:

[0087] (1) By using the deep learning algorithm, a large amount of real alarm data returned on a large scale can be widely utilized, and the generalization ability for corner cases is better;

[0088] (2) It reduces the complexity of post-processing logic adjustment. The single model conforms to the design of high cohesion and has fast iteration;

[0089] (3) The computational cost occupied by the online video understanding model is very low, similar to that of an ordinary 2D classification network, so it is very suitable for deployment on the device side.

[0090] According to any of the above embodiments, in this embodiment:

[0091] The event monitoring tasks include driver smoking monitoring task, driver handheld phone calling monitoring task, driver distraction monitoring task, driver fatigue monitoring task, direct sunlight monitoring task, video acquisition device occlusion monitoring task, and video acquisition device angle anomaly monitoring task.

[0092] The step of obtaining the driver behavior monitoring result according to the event monitoring task score includes:

[0093] Judge the event monitoring task score. If the event monitoring task scores of a set number of consecutive frames are all greater than the threshold, then take the event occurrence as the driver behavior monitoring result; otherwise, take the event non-occurrence as the driver behavior monitoring result.

[0094] In this embodiment, multiple event monitoring tasks are respectively predicted by multiple branches of the target prediction network; the driver behavior monitoring result is also obtained based on each event monitoring task respectively.

[0095] For example, if the event monitoring task scores of 20 consecutive frames are all greater than the threshold, then take the event occurrence as the driver behavior monitoring result. Using the method of this embodiment to monitor the online video data, the driver smoking monitoring task scores of the 1st to 20th frames are all greater than the threshold, and the driver distraction monitoring task scores are all greater than the threshold, while the scores of the remaining monitoring tasks are not all greater than the threshold. Then at the 20th frame, the output driver behavior monitoring result is that the driver smoking event occurs, the driver distraction event occurs, and the remaining events do not occur.

[0096] The beneficial effect of this embodiment is that:

[0097] Through multi-task branches, multi-task prediction under a single model is realized, effectively reducing the computational resources and time consumed in the feature extraction process, and achieving highly cohesive driver behavior monitoring.

[0098] Next, the driver behavior monitoring device based on the online video understanding network provided by the present invention will be described. The driver behavior monitoring device based on the online video understanding network described below can be correspondingly referred to the driver behavior monitoring method based on the online video understanding network described above.

[0099] An embodiment of the present invention provides a driver behavior monitoring system based on an online video understanding network, which is characterized by including:

[0100] A data acquisition module, configured to acquire online video data; the online video data includes image data of at least two consecutive frames.

[0101] A feature extraction module, configured to input the current frame image data into a feature extraction network in an online video understanding network to obtain a feature extraction result of the current frame image data.

[0102] A feature coverage module, configured to cover a set part of the feature extraction result of the current frame's image data with the feature extraction result of the historical frame image data to obtain a combined feature extraction result of the current frame image data.

[0103] A target prediction module, configured to input the combined feature extraction result into a target prediction network in the online video understanding network to obtain an event monitoring task score of the current frame image data.

[0104] A monitoring result module, configured to obtain a driver behavior monitoring result according to the event monitoring task score.

[0105] The online video understanding network is trained through online video samples.

[0106] Further, the feature coverage module includes:

[0107] An initial frame sub-module, configured to determine that the serial number i of the current frame is 1, and then obtain a feature extraction result of a reference image through the feature extraction network as a historical feature set; the reference image is a completely black image with the same size as the image data.

[0108] A non-initial frame sub-module, configured to determine that the serial number i of the current frame is not 1, and then obtain a feature extraction result of the (i - 1)-th frame image data through the feature extraction network as a historical feature set.

[0109] A frame feature coverage sub-module, configured to cover a set part of the feature extraction result of the i-th frame's image data with the historical feature set to obtain a combined feature extraction result of the i-th frame's image data.

[0110] The frame feature coverage sub-module includes a first combination unit or a second combination unit.

[0111] The first combination unit is configured to cover the feature extraction result of the i-th frame's image data with a first combined historical feature set to obtain a combined feature extraction result of the i-th frame's image data; the first combined historical feature includes the j-th sub-feature in the historical feature set; the j-th sub-feature is the result output by the j-th feature extraction layer of the feature extraction network; where j is an element in a combination serial number set, and the combination serial number set is a proper subset of the feature extraction layer serial number set.

[0112] A second combination unit is configured to cover the feature extraction result of the i-th frame of the image data with a second combination of historical feature sets to obtain a combined feature extraction result of the i-th frame of the image data; the second combination of historical features includes a set part of sub-features in the historical feature set; the sub-features are the results output by a single feature extraction layer in the feature extraction network.

[0113] The event monitoring tasks include a driver smoking monitoring task, a driver handheld phone call monitoring task, a driver distracted attention monitoring task, a driver fatigue monitoring task, a direct sunlight monitoring task, a video acquisition device occlusion monitoring task, and a video acquisition device abnormal angle monitoring task.

[0114] The monitoring result module includes:

[0115] A judgment sub-module is configured to judge the scores of the event monitoring tasks. If the scores of the event monitoring tasks for a set number of consecutive frames are all greater than a threshold, an event occurrence is used as the driver behavior monitoring result; otherwise, an event non-occurrence is used as the driver behavior monitoring result.

[0116] The beneficial effects of this embodiment are as follows:

[0117] By covering a part of the historical frame feature extraction result to the current frame feature extraction result, without consuming excessive computing resources, the features of each frame are combined with the features of all historical frames, and target prediction is performed based on the combined features, thereby realizing lightweight, high-efficiency, fast iteration, and good continuity in-cabin driver behavior monitoring.

[0118] Figure 4 Illustrates a schematic physical structure diagram of an electronic device, such as Figure 4As shown in the figure, the electronic device may include: a processor 410, a communications interface 420, a memory 430, and a communication bus 440. Among them, the processor 410, the communications interface 420, and the memory 430 complete communication with each other through the communication bus 440. The processor 410 may call the logical instructions in the memory 430 to execute a driver behavior monitoring method based on an online video understanding network. The method includes: obtaining online video data; the online video data includes image data of at least two consecutive frames; inputting the current frame image data into a feature extraction network in the online video understanding network to obtain a feature extraction result of the current frame image data; covering the feature extraction result of the image data of the current frame with a set part of the feature extraction results of the historical frame image data to obtain a combined feature extraction result of the current frame image data; inputting the combined feature extraction result into an object prediction network in the online video understanding network to obtain an event monitoring task score of the current frame image data; obtaining a driver behavior monitoring result according to the event monitoring task score; the online video understanding network is obtained by training with online video samples.

[0119] In addition, when the logical instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0120] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the driver behavior monitoring method based on the online video understanding network provided by the above-mentioned various methods. The method includes: obtaining online video data; the online video data includes image data of at least two consecutive frames; inputting the current frame image data into the feature extraction network in the online video understanding network to obtain the feature extraction result of the current frame image data; covering the feature extraction result of the image data of the current frame with a set part of the feature extraction results of the historical frame image data to obtain the combined feature extraction result of the current frame image data; inputting the combined feature extraction result into the target prediction network in the online video understanding network to obtain the event monitoring task score of the current frame image data; obtaining the driver behavior monitoring result according to the event monitoring task score; the online video understanding network is trained through online video samples.

[0121] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the driver behavior monitoring method based on the online video understanding network provided by the above-mentioned various methods. The method includes: obtaining online video data; the online video data includes image data of at least two consecutive frames; inputting the current frame image data into the feature extraction network in the online video understanding network to obtain the feature extraction result of the current frame image data; covering the feature extraction result of the image data of the current frame with a set part of the feature extraction results of the historical frame image data to obtain the combined feature extraction result of the current frame image data; inputting the combined feature extraction result into the target prediction network in the online video understanding network to obtain the event monitoring task score of the current frame image data; obtaining the driver behavior monitoring result according to the event monitoring task score; the online video understanding network is trained through online video samples.

[0122] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0123] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A driver behavior monitoring method based on an online video understanding network, characterized in that, Including: Obtain online video data; The online video data includes image data of at least two consecutive frames; Input the current frame image data into the feature extraction network in the online video understanding network to obtain the feature extraction result of the current frame image data; Cover the feature extraction result of the image data of the current frame with a set part of the feature extraction results of the historical frame image data to obtain the combined feature extraction result of the current frame image data; Input the combined feature extraction result into the target prediction network in the online video understanding network to obtain the event monitoring task score of the current frame image data; Obtain the driver behavior monitoring result according to the event monitoring task score; The online video understanding network is trained through online video samples; The step of covering the feature extraction result of the image data of the current frame with a set part of the feature extraction results of the historical frame image data to obtain the combined feature extraction result of the current frame image data includes: If it is determined that the serial number i of the current frame is 1, obtain the feature extraction result of the reference image through the feature extraction network as the historical feature set; the reference image is a completely black image with the same size as the image data; If it is determined that the serial number i of the current frame is not 1, obtain the feature extraction result of the (i - 1)-th frame image data through the feature extraction network as the historical feature set; Cover the feature extraction result of the image data of the i-th frame with a set part of the historical feature set to obtain the combined feature extraction result of the image data of the i-th frame; The step of covering the feature extraction result of the image data of the i-th frame with a set part of the historical feature set to obtain the combined feature extraction result of the image data of the i-th frame includes: Cover the feature extraction result of the image data of the i-th frame with the second combined historical feature set to obtain the combined feature extraction result of the image data of the i-th frame; The second combined historical feature includes a set part of the sub-features in the historical feature set; the sub-feature is the result output by a single feature extraction layer in the feature extraction network.

2. The driver behavior monitoring method based on the online video understanding network according to claim 1, characterized in that The step of covering the feature extraction result of the image data of the i-th frame with a set part of the historical feature set to obtain the combined feature extraction result of the image data of the i-th frame includes: Cover the feature extraction result of the image data of the i-th frame with the first combined historical feature set to obtain the combined feature extraction result of the image data of the i-th frame; The first combined historical feature includes the j-th sub-feature in the historical feature set; the j-th sub-feature is the result output by the j-th feature extraction layer of the feature extraction network; where j is an element in the combined serial number set, and the combined serial number set is a proper subset of the feature extraction layer serial number set.

3. The driver behavior monitoring method based on an online video understanding network according to any one of claims 1 to 2, characterized in that, The event monitoring tasks include driver smoking monitoring task, driver handheld phone call monitoring task, driver distracted attention monitoring task, driver fatigue monitoring task, direct sunlight monitoring task, video acquisition device occlusion monitoring task, and video acquisition device angle anomaly monitoring task.

4. The driver behavior monitoring method based on an online video understanding network according to claim 3, characterized in that, The step of obtaining the driver behavior monitoring result according to the event monitoring task score includes: Judge the score of the event monitoring task. If the scores of the event monitoring tasks for a set number of consecutive frames are all greater than the threshold, then consider the event as occurring as the driver behavior monitoring result; otherwise, consider the event as not occurring as the driver behavior monitoring result.

5. A driver behavior monitoring system based on an online video understanding network, characterized in that, Including: A data acquisition module for acquiring online video data; The online video data includes image data of at least two consecutive frames; A feature extraction module for inputting the current frame image data into the feature extraction network in the online video understanding network to obtain the feature extraction result of the current frame image data; A feature coverage module for covering a set part of the feature extraction result of the current frame's image data with the feature extraction result of the historical frame image data to obtain the combined feature extraction result of the current frame image data; A target prediction module for inputting the combined feature extraction result into the target prediction network in the online video understanding network to obtain the score of the event monitoring task for the current frame image data; A monitoring result module for obtaining the driver behavior monitoring result based on the score of the event monitoring task; The online video understanding network is obtained by training with online video samples; The feature coverage module includes: An initial frame sub-module for determining that the serial number i of the current frame is 1, and then obtaining the feature extraction result of the reference image through the feature extraction network as the historical feature set; the reference image is a completely black image with the same size as the image data; A non-initial frame sub-module for determining that the serial number i of the current frame is not 1, and then obtaining the feature extraction result of the (i - 1)-th frame image data through the feature extraction network as the historical feature set; A frame feature coverage sub-module for covering a set part of the feature extraction result of the i-th frame's image data with the historical feature set to obtain the combined feature extraction result of the i-th frame's image data; The frame feature coverage sub-module is further used for: Covering the feature extraction result of the i-th frame's image data with a second combined historical feature set to obtain the combined feature extraction result of the i-th frame's image data; The second combined historical feature includes a set part of the sub-features in the historical feature set; the sub-feature is the result output by a single feature extraction layer in the feature extraction network.

6. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the driver behavior monitoring method based on the online video understanding network according to any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the driver behavior monitoring method based on the online video understanding network according to any one of claims 1 to 4.

8. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the driver behavior monitoring method based on the online video understanding network according to any one of claims 1 to 4.

Images