Driver behavior detection method, device and equipment and storage medium

Abstract

The invention provides a driver behavior detection method, device and equipment and a storage medium, and the method comprises the steps: obtaining a to-be-detected image in a target vehicle, inputting the to-be-detected image into a preset behavior detection model, and enabling the to-be-detected image to comprise a driver image; performing multi-scale processing on each to-be-detected image to obtain each feature image of different scales; feature fusion is carried out on feature tensors obtained by stacking the feature images, and a first feature tensor is determined; adjusting the channel dimensions of the feature images to be consistent and then splicing the feature images to obtain a second feature tensor; and performing target frame identification after fusion based on the first feature tensor and the second feature tensor, and determining whether the driver has a distraction behavior or not. Therefore, the recognition of the tiny target in the distraction behavior detection of the driver is improved, and the detection accuracy is high.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a driver behavior detection method, device, equipment and storage medium. Background Art

[0002] In the field of transportation, driver distraction detection is one of the key technologies to improve road safety and reduce traffic accidents. Driver distraction refers to the situation in which the driver's attention is diverted from the driving task due to various reasons and no longer focuses on the road and driving behavior, thereby increasing the risk of traffic accidents. This phenomenon not only seriously threatens the driver's own safety, but also affects the safety of other road users. However, with the popularization of devices such as smartphones and in-car entertainment systems, the distraction factors faced by drivers during driving are increasing, further exacerbating driving risks. Therefore, it is necessary to detect driver distraction. In related technologies, driver distraction detection usually uses network models to monitor and analyze the driver's behavioral characteristics in real time, such as facial expressions, line of sight, hand movements, and vehicle operations, to identify whether the driver is in a distracted state, and issue a warning in time to remind the driver to refocus.

[0003] However, during the detection process, it is difficult to accurately identify small changes in driver behavior. For example, facial expression changes such as increased blinking frequency, reduced smiling, yawning, and hand shaking due to fatigue, changes in the direction of the steering wheel, etc., are all small targets that are difficult to detect. There are difficulties such as a small number of annotated objects, low object density, and severe lack of information, resulting in low accuracy in detecting driver distraction behaviors. In addition, the network models used for detection are often computationally complex and resource intensive, making it difficult to meet the real-time requirements of in-vehicle edge devices. Summary of the invention

[0004] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0005] In view of the above-mentioned shortcomings of the prior art, the present application discloses a driver behavior detection method, device, equipment and storage medium to solve at least one of the above-mentioned technical problems.

[0006] In a first aspect, the present application provides a driver behavior detection method, comprising: obtaining an image to be detected in a target vehicle, inputting the image to be detected into a preset behavior detection model, wherein the image to be detected includes a driver image; performing multi-scale processing on the image to be detected to obtain feature images of different scales; performing feature fusion on feature tensors obtained by stacking each of the feature images to determine a first feature tensor; adjusting the channel dimensions of each of the feature images to be consistent and then splicing them to obtain a second feature tensor; performing target frame recognition based on the fusion of the first feature tensor and the second feature tensor to determine whether the driver has distracted behavior.

[0007] In one embodiment of the present application, the feature tensor obtained by stacking each of the feature images is subjected to feature fusion to determine the first feature tensor, including: adjusting the resolution of each of the feature images to obtain each of the feature images with the same resolution; horizontally stacking each of the feature images with the same resolution to obtain the feature tensor; performing image segmentation on the feature tensor to determine each target feature map of different scales; and performing feature fusion on each of the target feature maps to determine the first feature tensor.

[0008] In one embodiment of the present application, the channel dimensions of each of the feature images are adjusted to be consistent and then stitched to obtain a second feature tensor, including: adjusting the channel dimension of each of the feature images to a preset dimension; if the scale of the feature image is larger than the target scale, downsampling the feature image, and the target scale is the median of each scale; if the scale of the feature image is smaller than the target scale, upsampling the feature image; after sampling, stitching each of the feature images to obtain the second feature tensor.

[0009] In one embodiment of the present application, the target frame recognition is performed based on the fusion of the first feature tensor and the second feature tensor to determine whether the driver has distracted behavior, including: fusing the first feature tensor and the second feature tensor to obtain an image to be predicted; performing bounding box prediction on the image to be predicted to generate multiple prediction frames of different categories and a confidence level of the driver's distracted behavior corresponding to each prediction frame, wherein the prediction frames of different categories correspond to different distracted behaviors of the driver; filtering the prediction frames of each category based on a non-maximum suppression method to obtain target frames of each category; and determining whether the driver has distracted behavior based on the target frames of different categories and the confidence levels corresponding to the target frames.

[0010] In one embodiment of the present application, the preset behavior detection model is compressed, including: converting the weight tensor of each layer of the neural network included in the preset behavior detection model to obtain each weight vector of each layer of the neural network; generating a pruning mask for each layer of the neural network, the pruning mask being a Boolean type tensor that minimizes the distortion of the neural network after pruning the weight tensor, and the pruning mask and the weight tensor have the same shape; pruning the weight tensor of the neural network in the same layer based on the pruning mask to adjust each weight vector in the weight tensor, determine a first weight tensor, and use the weight vector in the first weight tensor as a first weight vector Quantity; screen each of the first weight vectors in the target neural network to determine a target weight vector, the target neural network is determined by a preset pruning order, and the target weight vector is the first weight vector with the smallest absolute value in the target neural network; prune the target weight vector in the target neural network, and calculate the first distortion value of the target neural network and the second distortion value of the preset behavior detection model; compare the first distortion value with the preset first distortion range, and the second distortion value with the preset second distortion range, respectively, and determine whether to screen the new target weight vector for pruning based on the comparison result to complete the compression of the preset behavior detection model.

[0011] In one embodiment of the present application, the screening of each first weight vector in the target neural network to determine a target weight vector includes: determining the neural network to be pruned according to the preset pruning order and using it as the target neural network; sorting each first weight vector in the target nerve in ascending order to obtain a weight sequence; screening each first weight vector in the target nerve based on the weight sequence to determine a target weight vector; after pruning the target weight vector, if the first distortion value is within a preset first distortion range and the second distortion value is within a preset second distortion range, re-sorting each first weight vector in the target nerve to determine a new target weight vector; wherein the square of each first weight vector in the target neural network is respectively calculated with the sum of the squares of each first weight vector to determine the score of each first weight vector; and each first weight vector is sorted in ascending order according to the score to determine the weight sequence.

[0012] In one embodiment of the present application, the method of determining whether to select a new target weight vector for pruning based on the comparison result includes: if the first distortion value is within a preset first distortion range, and the second distortion value is within a preset second distortion range, then selecting a new target weight vector in the target neural network and pruning it; if the first distortion value is not within the preset first distortion range, and the second distortion value is within the preset second distortion range, then redetermining the target neural network; until the second distortion value is no longer within the preset second distortion range, or each of the neural networks has been pruned, pruning is stopped to complete the compression of the preset behavior detection model.

[0013] In a second aspect, the present application provides a driver behavior detection device, which includes: an acquisition module, used to acquire an image to be detected in a target vehicle, and input the image to be detected into a preset behavior detection model, wherein the image to be detected includes a driver image; a preset behavior detection model, used to perform multi-scale processing on each of the images to be detected to obtain feature images of different scales; perform feature fusion on the feature tensors obtained by stacking each of the feature images to determine a first feature tensor; adjust the channel dimensions of each of the feature images to be consistent and then splice them to obtain a second feature tensor; perform target frame recognition based on the fusion of the first feature tensor and the second feature tensor to determine whether the driver has distracted behavior.

[0014] In a third aspect, the present application also provides an electronic device, comprising: a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement the method described in the above embodiments.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of a computer, the computer executes the method described in the above embodiments.

[0016] Beneficial effects of the present application: The present application proposes a driver behavior detection method, device, equipment and storage medium. By acquiring the image to be detected in the target vehicle, inputting a preset behavior detection model, the image to be detected includes the driver image, and then using the preset behavior detection model to perform multi-scale processing on the image to be detected to obtain feature images of different scales, and then performing feature fusion on the feature tensors obtained by stacking each feature image, determining the first feature tensor to capture all the information in the image to be detected and the local information of the small target, and adjusting the channel dimensions of each feature image to be consistent and splicing them to obtain the second feature tensor to prevent the loss of information related to the small target in the image to be detected, and finally performing target frame recognition based on the fusion of the first feature tensor and the second feature tensor to determine whether the driver has distracted behavior, thereby improving the accuracy of detection. In addition, the weight connections of each layer of the neural network in the model are pruned to make the model more lightweight, and the real-time performance of the model detection is ensured even if it is deployed on the edge device in the vehicle.

[0017] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0019] Figure 1 is a schematic diagram of an implementation environment of a driver behavior detection method shown in an exemplary embodiment of the present application;

[0020] Figure 2 is a flow chart of a driver behavior detection method shown in an exemplary embodiment of the present application;

[0021] Figure 3 is a schematic diagram of multi-scale feature fusion shown in an exemplary embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a driver behavior detection process shown in an exemplary embodiment of the present application;

[0023] Figure 5 is a schematic diagram of an amplitude pruning network shown in an exemplary embodiment of the present application;

[0024] Figure 6is a block diagram of a driver behavior detection device shown in an exemplary embodiment of the present application;

[0025] Figure 7 It is a structural diagram of a computer system suitable for implementing the electronic device of the present application, shown as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0027] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0028] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0029] It should be noted that small targets in an image generally refer to objects with a resolution lower than 32*32 pixels, while tiny targets generally refer to objects with a resolution lower than 15*15 pixels.

[0030] See also Figure 1 , is a schematic diagram of an implementation environment of a driver behavior detection method shown in an exemplary embodiment of the present application. Figure 1As shown, the implementation environment includes a target vehicle 110 and an edge device 120, wherein the edge device 120 is embedded in the target vehicle 110, and the edge device 120 includes but is not limited to a vehicle-mounted system, an on-board computer, an on-board controller, etc., and is used to implement a driver behavior detection method, and a preset behavior detection model is deployed on the edge device 120. By acquiring an image to be detected in the target vehicle 110, the image to be detected is input into the preset behavior detection model in the edge device 120, and multi-scale processing is performed on the image to be detected to obtain feature images of different scales, and then feature fusion is performed on the feature tensors obtained by stacking the feature images to determine a first feature tensor, and the channel dimensions of the feature images are adjusted to be consistent and then spliced ​​to obtain a second feature tensor, and finally, target frame recognition is performed based on the fusion of the first feature tensor and the second feature tensor to determine whether the driver has distracted behavior. In this way, the global or high-level semantic information from multiple scale images is combined to improve the accuracy of detection.

[0031] See also Figure 2 , is a flowchart of a driver behavior detection method shown in an exemplary embodiment of the present application. The method can be applied to Figure 1 The implementation environment shown in FIG. 1 is shown in FIG. 1 . It should be understood that the method may also be applicable to other exemplary implementation environments, and this embodiment does not limit the implementation environment to which the method is applicable. Figure 2 As shown, in an exemplary embodiment, the driver behavior detection method includes at least steps S210 to S250, which are described in detail as follows:

[0032] Step S210, obtaining an image to be detected in the target vehicle, and inputting the image to be detected into a preset behavior detection model, wherein the image to be detected includes an image of the driver.

[0033] In one embodiment of the present application, the image to be detected is an image actually taken when the driver is driving in the target vehicle. Before the image to be detected is input into the preset behavior detection model, each image to be detected is compressed so that the image to be detected is adjusted to a preset size that meets the input of the preset behavior detection model. In this way, the size of the image to be detected input into the preset behavior detection model is consistent, which helps to learn more robust and generalized features using the model while reducing the amount of calculation of the preset behavior detection network model, so as to improve the accuracy of driver distraction behavior detection.

[0034] Step S220, performing multi-scale processing on the image to be detected to obtain feature images of different scales.

[0035] In one embodiment of the present application, the preset behavior detection module includes a multi-scale processing model, which is constructed by a multi-layer neural network and is used to perform multiple downsampling processes (reducing resolution) on each image to be detected, capture image features at different spatial scales, and achieve fine capture of tiny features. In order to reduce noise in the feature image, each feature image downsampled can also be Gaussian smoothed. Among them, the downsampling process can be achieved by average pooling or interpolation of the image to be detected, and there is no limitation on this.

[0036] See also Figure 3 , is a schematic diagram of a multi-scale feature fusion shown in an exemplary embodiment of the present application. Figure 3 As shown, the scale processing model can be a Backbone network, which is used to extract high-level features from the image to be detected, that is, to extract feature images such as P1, P2, P3, P4, and P5.

[0037] In one embodiment of the present application, before step S230 and step S240, Gaussian smoothing and scaling may be performed on the feature image of each scale, and the expression is as follows:

[0038] F σ (w,h)=G σ (w,h)×f(w,h) Formula (1)

[0039]

[0040] Among them, G in formula (1) and (2) σ (w,h) represents a Gaussian filter, f(w,h) represents the input Gaussian filter with a width of w and a height of h as the feature image, F σ (w,h) represents the feature image after smoothing by the Gaussian filter, and σ represents the scaling parameter of the standard deviation of the Gaussian filter. In this way, the Gaussian filter is used to remove noise in the feature image and blur the image, while retaining the edge information of the image, making the structural feature information of the feature image more obvious, which helps to more accurately identify and analyze the features in the feature image in subsequent image processing.

[0041] In one embodiment of the present application, Figure 1 As shown, the preset behavior detection model also includes a multi-scale triple feature fusion encoding module, which is constructed by a multi-layer neural network and is used to implement steps S230 and S240.

[0042] Step S230, performing feature fusion on the feature tensors obtained by stacking the feature images to determine a first feature tensor.

[0043] Specifically, the resolution of each feature image is adjusted to obtain feature images of the same resolution; the feature images of the same resolution are horizontally stacked to obtain a feature tensor; the feature tensor is segmented to determine feature maps of targets of different scales; and feature fusion is performed on the feature maps of the targets to determine a first feature tensor.

[0044] In one embodiment of the present application, the nearest neighbor interpolation method can be used to adjust images of different resolutions to the same resolution. Due to the fact that the scales and resolutions of feature images at different levels are usually different under multi-scale processing, the deeper the level, the smaller the scale (size) and resolution of the feature map. Therefore, in order to retain the detail information in the feature image to the greatest extent, the low-resolution feature image is adjusted to high resolution when adjusting the resolution so that it is aligned with the high-resolution feature map in the spatial dimension, ensuring that in the subsequent feature stacking or feature fusion process, too much detail information will not be lost due to the reduction in resolution, so as to further improve the accuracy of detecting driver behavior. Then, the feature maps of the same resolution and different scales are stacked horizontally in the spatial dimension. In this way, the shallow feature map and the deep feature map are combined, which helps the preset behavior detection model to retain the local detail information of the image (including the detail information of the tiny target) while enhancing the global feature connectivity, so as to more accurately understand the content of the image to be detected. Among them, the shallow feature map is responsible for capturing the local detail information of the image, such as low-level features such as edges and textures; the deep feature map abstracts the high-level features of the feature image, which are more global and abstract, and can express complex patterns and concepts in the image.

[0045] In one embodiment of the present application, due to the differences in semantic levels and other aspects of feature images of different scales, directly fusing these feature images with large differences may cause information confusion or feature conflicts, thereby affecting the performance of the model, and direct fusion may cause feature redundancy, that is, some features are reflected multiple times in the feature tensor. Therefore, it is necessary to perform image segmentation on the stacked feature tensor, which helps to process these features more specifically in the subsequent feature fusion process, thereby improving the detection accuracy of the model.

[0046] In one embodiment of the present application, the first feature tensor obtained by feature fusion can also be amplified to improve feature information, thereby preventing subsequent missed detection or misdetection of small targets and further improving the accuracy of model detection.

[0047] In this way, while maintaining high resolution, multi-scale feature extraction is enhanced, which helps to improve the accuracy of detection by combining global high-level semantic information and local detail information from multiple scale images.

[0048] Step S240, adjusting the channel dimensions of each feature image to be consistent and then splicing them to obtain a second feature tensor.

[0049] Specifically, feature encoding of each feature image includes: adjusting the channel dimension of each feature image to a preset dimension; if the scale of the feature image is larger than the target scale, downsampling the feature image, and the target scale is the median of each scale; if the scale of the feature image is smaller than the target scale, upsampling the feature image; after sampling, splicing each feature image to obtain a second feature tensor.

[0050] In one embodiment of the present application, in order to avoid losing the feature information of a tiny target in a feature image, especially after the feature image is blurred, some detail information is easily lost, and these detail information is crucial for the detection of tiny targets. Feature images of different scales are convolved so that the channel dimension of each feature image is adjusted to 1, that is, the preset dimension is 1, so that the number of channels of feature images of different scales is consistent, so that they can be directly spliced ​​later, so that feature information of different levels can be fully utilized, and the performance of the model and the accuracy of detection are improved.

[0051] In one embodiment of the present application, downsampling the feature image whose scale is larger than the target scale is beneficial to retain the high-resolution features of the large-scale feature image and improve the effectiveness and diversity of the image; upsampling the feature image whose scale is smaller than the target scale is beneficial to maintain the richness of the local features of the low-resolution image, prevent the information of the tiny target features from being lost in the subsequent fusion process, and enhance the detection effect of the tiny target. Among them, the hybrid structure of maximum pooling and average pooling can be used for downsampling, and the convolution processing can be used for upsampling.

[0052] In one embodiment of the present application, the feature images with the same number of channels are concatenated in the channel dimension to obtain a second feature tensor, which is expressed as follows:

[0053] P TFE =Concant(P l ,P m ,P s ) Formula (3)

[0054] Among them, P in formula (3) l , P m , P s Represents feature images of different scales, P m is the feature image of the target scale, P l The scale of P is larger than the target scale. s The scale is smaller than the target scale, P TFE Represents the second feature tensor. Concat means concatenating two or more feature maps in a certain dimension.

[0055] Through the above method, the local fine details of tiny target objects can be captured to a greater extent, improving the accuracy of subsequent detection.

[0056] It should be noted that step S230 and step S240 can be executed in parallel without limiting the execution order.

[0057] Step S250, performing target frame recognition based on the fusion of the first feature tensor and the second feature tensor to determine whether the driver has distracted behavior.

[0058] Specifically, Figure 1 As shown, the first feature tensor and the second feature tensor are fused to obtain the images to be predicted Q1, P2, and P3. Target frame recognition includes: performing bounding box prediction on the image to be predicted, generating multiple prediction frames of different categories and the confidence of each prediction frame corresponding to the driver's distraction behavior, and different categories of prediction frames correspond to different distraction behaviors of the driver, such as looking at the phone, eating, talking with passengers, etc.; filtering each category of prediction frames based on the non-maximum suppression method to obtain target frames of each category; judging according to the different categories of target frames and the corresponding confidence of the target frames to determine whether the driver has distraction behavior.

[0059] In one embodiment of the present application, the bounding box prediction, that is, the image to be predicted is input into the detection head, and the detection head is constructed by a multi-layer neural network, usually a convolutional neural network (CNN). The generated prediction box is used to locate the key area in the image to be predicted that may contain the driver's distraction behavior. Each prediction box corresponds to one or more categories, and each prediction box is associated with a confidence, which indicates the probability that the corresponding distraction behavior does exist in the prediction box. In addition, there may be multiple similar features in the image to be predicted, resulting in multiple overlapping prediction boxes predicted in the same area. These prediction boxes may point to distraction behaviors of the same category. Therefore, the non-maximum suppression (NMS) method sorts the prediction boxes of each category according to the confidence, and then traverses the prediction boxes of each category separately, retains the prediction box with the highest confidence in each category, and suppresses (i.e., deletes or reduces the weight) other prediction boxes with a high degree of overlap with it, and determines the target boxes of each category. If the confidence of any category target box reaches the preset threshold, it is determined that the driver has the distraction behavior corresponding to the category; if the confidence of any category target box is less than the preset threshold, it is determined that the driver does not have the distraction behavior corresponding to the category. In this way, the accuracy and reliability of driver behavior detection are high, the amount of calculation is reduced, and the detection efficiency and real-time performance are improved.

[0060] See also Figure 4 , is a schematic diagram of a driver behavior detection process shown in an exemplary embodiment of the present application. Figure 4As shown in the figure, the detection of driver distraction behavior has high real-time requirements for the model, and deep learning models usually have a large number of parameters and complex structures, resulting in a large model file size and extremely high demand for computing and storage resources. However, vehicle-side edge devices (such as on-board computers, sensors, etc.) often have limited resources, resulting in poor real-time performance of the model. Therefore, in order to ensure that the preset behavior detection model is deployed in the edge device in the target vehicle with higher real-time performance, it needs to be compressed. Figure 4 The adaptive amplitude global pruning module in is used to implement the model compression step as described below. In this way, after compressing the preset behavior detection model, the detection result of whether the driver has distracted behavior can be obtained more quickly.

[0061] Specifically, the model is compressed, including: converting the weight tensor of each layer of the neural network included in the preset behavior detection model to obtain each weight vector of each layer of the neural network; generating a pruning mask for each layer of the neural network, the pruning mask is a Boolean type tensor that minimizes the distortion of the neural network after pruning the weight tensor, and the pruning mask and the weight tensor have the same shape; based on the pruning mask, the weight tensor of the same layer of the neural network is pruned to adjust the weight vectors in the weight tensor, determine the first weight tensor, and use the weight vector in the first weight tensor as the first weight vector.

[0062] In one embodiment of the present application, a preset behavior detection model is deployed on the target vehicle. (d) Represents the weight tensor (i.e., weight matrix) of the dth layer. When the input to the preset behavior detection model is x, the expression for calculating the output is as follows:

[0063] f(x;W (1:d) )=W (d) σ(…W (2) σ(W (1) x)…) Formula (4)

[0064] Among them, W in formula (4) (d) Represents the weight tensor of the dth layer, W (1:d) =(W (1) ,...,W (d) ) represents the set of weight tensors of each layer of the neural network, and σ represents the ReLU (Rectified Linear Unit) activation function.

[0065] In one embodiment of the present application, each weight vector of each layer is sorted in ascending order. If W[n] represents the nth weight vector (i.e., a one-dimensional vector), then |W[1]|≤|W[2]|. The ascending order is based on the absolute value of each weight vector.

[0066] In one embodiment of the present application, the neural network is pruned based on amplitude, and during the pruning process, the distortion of the neural network output caused by the pruning operation is minimized, so that the difference between the pruned neural network output and the original neural network output is as small as possible. The weights of the neural network are pruned based on amplitude, and the number of parameters and computational complexity of the model are reduced by removing those weights that have little impact on the model performance, while maintaining the performance of the model as much as possible.

[0067] In one embodiment of the present application, when the weight tensor W∈R of a layer of neural network m×n When , the pruning mask is an m×n matrix consisting of only two elements, 0 and 1, where 0 indicates pruning the weight vector at the corresponding position in the weight tensor W, and 1 indicates retaining the weight vector at the corresponding position in the weight tensor W. Within the preset input bounds of the preset detection model, the expression for generating the pruning mask is as follows:

[0068]

[0069] Where W in formula (5) represents the weight tensor, M represents the pruning mask, Represents the first weight tensor formed by pruning the weight tensor W by the pruning mask M, min||WM·W|| F represents the minimization of the Frobenius norm between W and M·W, represents the weight vector of the i-th row and j-th column in W (the weight vector with the largest absolute value in the weight tensor), M ij Indicates the pruning mask and W ij The corresponding elements are set to 1 to retain important weight vectors.

[0070] See also Figure 5 , is a schematic diagram of an amplitude pruning network according to an exemplary embodiment of the present application. Figure 5 The CNNs in the figure are preset behavior detection models. After amplitude-based pruning, the model compression effect is obvious.

[0071] Specifically, compressing the model also includes: screening each first weight vector in the target neural network to determine a target weight vector, the target neural network is determined by a preset pruning order, and the target weight vector is the first weight vector with the smallest absolute value in the target neural network; pruning the target weight vector in the target neural network, calculating the first distortion value of the target neural network and the second distortion value of the preset behavior detection model; comparing the first distortion value with the preset first distortion range, the second distortion value with the preset second distortion range, and determining whether to screen a new target weight vector for pruning based on the comparison result to complete the compression of the preset behavior detection model.

[0072] In one embodiment of the present application, when pruning each layer of the neural network, at least one weight connection is retained, that is, at least one weight vector is retained.

[0073] In one embodiment of the present application, a neural network to be pruned is determined according to a preset pruning order and used as a target neural network; each first weight vector in the target nerve is sorted in ascending order to obtain a weight sequence; each first weight vector in the target nerve is screened based on the weight sequence to determine a target weight vector; after pruning the target weight vector, if the first distortion value is within a preset first distortion range and the second distortion value is within a preset second distortion range, the first weight vectors in the target nerve are re-sorted to determine a new target weight vector; wherein the square of each first weight vector in the target neural network is respectively calculated with the sum of the squares of each first weight vector to determine the score of each first weight vector; each first weight vector is sorted in ascending order according to the score to determine the weight sequence, and the target weight vector is the first weight vector with the smallest score in the weight sequence.

[0074] In one embodiment of the present application, the preset second distortion range of the preset behavior detection model, that is, the upper limit of the distortion of the model is expressed as:

[0075]

[0076] Wherein, W in formula (6) represents the weight tensor, (1:d) represents the set of weight tensors of each layer of the neural network, that is, the set of weight tensors before pruning, (1:i-1) represents the set of weight tensors after pruning, that is, the set of weight tensors of the neural network vectors from the 1st layer to the i-1th layer, and (i+1:d) represents the set of weight tensors of the neural network vectors from the i+1th layer to the dth layer. In addition, there is no activation function on the right side of formula (6), which is equivalent to pruning layer by layer based on the amplitude until the required global sparsity constraint is met. Does not affect pruning decisions.

[0077] In one embodiment of the present application, the expression for determining the score of the first weight vector is:

[0078]

[0079] Where score(u,W) in formula (7) represents the score of W[u], W[u] represents the score of the first weight vector ranked u in the weight tensor, W[u] represents the score of the first weight vector ranked v in the weight tensor, ∑ v≥u (W[v]) 2 Represents the sum of the squares of all first weight vectors in the weight tensor.

[0080]

[0081] Among them, it can be seen from formula (8) that the later the ranking of the first weight vector in the weight sequence is, the greater the score is.

[0082] In one embodiment of the present application, if the first distortion value is within a preset first distortion range, and the second distortion value is within a preset second distortion range, a new target weight vector is screened and pruned in the target neural network; if the first distortion value is not within the preset first distortion range, and the second distortion value is within the preset second distortion range, the target neural network is re-determined; until the second distortion value is no longer within the preset second distortion range, or all neural networks have been pruned, pruning is stopped to complete the compression of the preset behavior detection model.

[0083] In one embodiment of the present application, if the number of pruning times of each layer of the neural network reaches a preset number, pruning is stopped; or, if the number of first weight vectors in each layer of the neural network reaches a preset number, pruning is stopped.

[0084] In this way, the preset behavior detection model is iteratively pruned by using a feedback mechanism, and only one weight vector with the smallest score in the neural network layer is pruned in each iteration until the required global sparsity is met. In this way, the compression effect of the model is further improved, the amount of model calculation and complexity are reduced, and the accuracy and real-time performance of driver distraction behavior detection are guaranteed while achieving model lightweight.

[0085] The driver behavior detection method provided in the present application has the following advantages: First, it combines the global information and high-dimensional information in the multi-scale feature image from the image to be detected, and captures the local details of the tiny target, thereby improving the accuracy of detection. Second, the weight connections of each layer of the neural network in the model are pruned to make the model more lightweight, ensuring the real-time performance of the model detection even when deployed on the edge device in the vehicle.

[0086] See also Figure 6 , is a block diagram of a driver behavior detection device shown in an exemplary embodiment of the present application. The device can be applied to Figure 1 It should be understood that the present application does not limit the implementation environment of the device. Figure 6 As shown, in an exemplary embodiment, the driver behavior detection device includes at least an acquisition module 610 and a preset behavior detection model 620, which are described in detail as follows:

[0087] An acquisition module 610 is used to acquire an image to be detected in a target vehicle and input the image to be detected into a preset behavior detection model, wherein the image to be detected includes an image of a driver;

[0088] The preset behavior detection model 620 is used to perform multi-scale processing on the image to be detected to obtain feature images of different scales; perform feature fusion on the feature tensors obtained by stacking the feature images to determine the first feature tensor; adjust the channel dimensions of the feature images to be consistent and then splice them to obtain the second feature tensor; perform target frame recognition based on the fusion of the first feature tensor and the second feature tensor to determine whether the driver has distracted behavior.

[0089] It should be noted that the driver behavior detection device provided in the above embodiment and the driver behavior detection method provided in the above embodiment belong to the same concept, wherein the contents of the operations performed by each module have been described in detail in the method embodiment and will not be repeated here.

[0090] The present application also provides an electronic device, comprising: a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement the driver behavior detection method as in the above embodiment.

[0091] See also Figure 7 , shows a schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application. It should be noted that, Figure 7 The computer system 700 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0092] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 702 or the program loaded from the storage part 708 to the random access memory (RAM) 703, such as executing the method in the above embodiment. In the RAM 703, various programs and data required for system operation are also stored. The CPU 701, ROM 702 and RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0093] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, a mouse, etc.; an output section 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed so that a computer program read therefrom is installed into the storage section 708 as needed.

[0094] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 709, and / or installed from a removable medium 711. When the computer program is executed by a central processing unit (CPU) 701, various functions defined in the system of the present application are executed.

[0095] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes the vehicle-side data forwarding method as described above. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.

[0096] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, system or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, system or device. A computer program contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0097] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0098] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not constitute limitations on the units themselves in some cases.

[0099] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A driver behavior detection method, characterized in that: include: Acquire an image to be detected in a target vehicle, and input the image to be detected into a preset behavior detection model, wherein the image to be detected includes a driver image; Performing multi-scale processing on the image to be detected to obtain feature images of different scales; Performing feature fusion on the feature tensors obtained by stacking the feature images to determine a first feature tensor; Adjusting the channel dimensions of the feature images to be consistent and then splicing them to obtain a second feature tensor; Based on the fusion of the first feature tensor and the second feature tensor, target frame recognition is performed to determine whether the driver has distracted behavior.

2. The driver behavior detection method according to claim 1, characterized in that: The step of performing feature fusion on the feature tensors obtained by stacking the feature images to determine the first feature tensor includes: Adjusting the resolution of each of the characteristic images to obtain each of the characteristic images with the same resolution; Stacking the feature images with the same resolution horizontally to obtain the feature tensor; Performing image segmentation on the feature tensor to determine feature maps of each target at different scales; Perform feature fusion on each of the target feature maps to determine the first feature tensor.

3. The driver behavior detection method according to claim 1, characterized in that: The step of adjusting the channel dimensions of the feature images to be consistent and then splicing them to obtain a second feature tensor includes: Adjusting the channel dimension of each of the feature images to a preset dimension; If the scale of the feature image is larger than the target scale, the feature image is downsampled, and the target scale is the median of each scale; If the scale of the feature image is smaller than the target scale, upsampling the feature image; After sampling, each of the feature images is spliced ​​to obtain the second feature tensor.

4. The driver behavior detection method according to claim 1, characterized in that: The step of performing target frame recognition based on the fusion of the first feature tensor and the second feature tensor to determine whether the driver has distracted behavior includes: Fusing the first feature tensor and the second feature tensor to obtain an image to be predicted; Performing bounding box prediction on the image to be predicted, generating a plurality of prediction boxes of different categories and a confidence level of each prediction box corresponding to the driver's distraction behavior, wherein the prediction boxes of different categories correspond to different distraction behaviors of the driver; Based on the non-maximum suppression method, the prediction boxes of each category are respectively screened to obtain the target boxes of each category; A judgment is made based on the target frames of different categories and the confidence levels corresponding to the target frames to determine whether the driver has distracted behavior.

5. The driver behavior detection method according to any one of claims 1 to 4, characterized in that: Compress the preset behavior detection model, including: Converting the weight tensor of each layer of the neural network included in the preset behavior detection model to obtain each weight vector of the neural network in each layer; Generate a pruning mask for each layer of the neural network, wherein the pruning mask is a Boolean type tensor that minimizes the distortion of the neural network after pruning the weight tensor, and the pruning mask has the same shape as the weight tensor; Pruning the weight tensor of the neural network in the same layer based on the pruning mask to adjust each weight vector in the weight tensor, determine a first weight tensor, and use the weight vector in the first weight tensor as a first weight vector; Screening each of the first weight vectors in the target neural network to determine a target weight vector, wherein the target neural network is determined by a preset pruning order, and the target weight vector is the first weight vector with the smallest absolute value in the target neural network; Pruning the target weight vector in the target neural network, and calculating a first distortion value of the target neural network and a second distortion value of the preset behavior detection model; The first distortion value and the preset first distortion range, the second distortion value and the preset second distortion range are compared respectively, and based on the comparison results, it is determined whether to select the new target weight vector for pruning to complete the compression of the preset behavior detection model.

6. The driver behavior detection method according to claim 5, characterized in that: The step of screening each of the first weight vectors in the target neural network to determine a target weight vector includes: Determining the neural network to be pruned according to the preset pruning order and using it as the target neural network; Sorting the first weight vectors in the target nerve in ascending order to obtain a weight sequence; Screening each of the first weight vectors in the target nerve based on the weight sequence to determine a target weight vector; After pruning the target weight vector, if the first distortion value is within a preset first distortion range, and the second distortion value is within a preset second distortion range, then reordering the first weight vectors in the target nerve to determine a new target weight vector; wherein the square of each of the first weight vectors in the target neural network is respectively calculated with the sum of the squares of the first weight vectors to determine the score of each of the first weight vectors; The first weight vectors are sorted in ascending order according to the scores to determine the weight sequence.

7. The driver behavior detection method according to claim 6, characterized in that: The determining whether to select the new target weight vector for pruning based on the comparison result includes: If the first distortion value is within a preset first distortion range, and the second distortion value is within a preset second distortion range, selecting and pruning the new target weight vector in the target neural network; If the first distortion value is not within the preset first distortion range, and the second distortion value is within the preset second distortion range, re-determining the target neural network; Until the second distortion value is no longer within the preset second distortion range, or each of the neural networks has been pruned, pruning is stopped to complete the compression of the preset behavior detection model.

8. A driver behavior detection device, characterized in that: include: An acquisition module is used to acquire an image to be detected in a target vehicle and input the image to be detected into a preset behavior detection model, wherein the image to be detected includes a driver image; A preset behavior detection model is used to perform multi-scale processing on the image to be detected to obtain feature images of different scales; perform feature fusion on the feature tensors obtained by stacking the feature images to determine a first feature tensor; adjust the channel dimensions of the feature images to be consistent and then splice them to obtain a second feature tensor; perform target frame recognition based on the fusion of the first feature tensor and the second feature tensor to determine whether the driver has distracted behavior.

9. An electronic device, characterized in that: include: processor, memory, and communications bus; The communication bus is used to connect the processor and the memory; The processor is configured to execute the computer program stored in the memory to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is used to make a computer execute the method according to any one of claims 1 to 7.

Images