Tracking method, model training method, device and equipment

Abstract

The embodiment of the invention discloses a vehicle driver tracking method, a model training method, a device and equipment. The method includes: acquiring a driver status image of the vehicle via an image sensor on the vehicle; inputting the driver state image into a pre-trained driver state recognition model to obtain a driver recognition result; and the identification result of the driver is input into the pre-trained multi-target tracking model, whether the driver leaves the vehicle or not is determined, the method is suitable for end-side deployment, and the vehicle leaving condition of the driver can be rapidly, accurately and automatically identified, so that illegal riding or vehicle loss is avoided.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a driver tracking method for a vehicle, a model training method, device and equipment. BACKGROUND

[0002] In order to avoid the driver riding the vehicle such as a two-wheeled vehicle in violation of the rules, or to avoid the vehicle loss event caused by the vehicle management confusion, a driver presence detection function can be configured on the vehicle.

[0003] The methods described in this section can not be the methods that have been previously conceived or adopted. Unless otherwise indicated, it should not be assumed that any of the methods described in this section are considered prior art merely because of their inclusion in this section. Similarly, issues mentioned in this section should not be assumed to have been admitted to be prior art in any jurisdiction merely because of their inclusion in this section. SUMMARY

[0004] The present application provides a driver tracking method for a vehicle, a model training method, device and equipment, which can quickly and accurately automatically identify the driver leaving the vehicle to avoid riding in violation of the rules or vehicle loss.

[0005] According to an aspect of the present application, a driver tracking method for a vehicle is provided, which comprises the following steps:

[0006] acquiring a driver state image of the vehicle via an image sensor on the vehicle;

[0007] inputting the driver state image into a pre-trained driver state recognition model to obtain a recognition result of the driver;

[0008] inputting the recognition result of the driver into a pre-trained multi-target tracking model to determine whether the driver leaves the vehicle.

[0009] Optionally, the vehicle is a handlebar-controlled vehicle.

[0010] Optionally, the image sensor is installed on the dashboard of the vehicle or installed in an area of the vehicle at a preset distance from the dashboard.

[0011] Optionally, the vehicle is a two-wheeled vehicle.

[0012] Optionally, the driver state recognition model is a first neural network model trained by the following method:

[0013] acquiring a driver state change data set, the driver state change data set comprising a plurality of first image samples of the vehicle during stopping or driving;

[0014] For each first image sample in the plurality of first image samples, the following operations are performed:

[0015] Edge information and background information of the first image sample are extracted;

[0016] The first image sample is subjected to receptive field enhancement processing according to characteristic information of the driver state recognition;

[0017] The driver state in the first image sample is recognized;

[0018] A first neural network model is trained based on the edge information, the background information and the recognized driver state of the first image sample to establish a driver state recognition model.

[0019] Optionally, training the first neural network model based on the edge information, the background information and the recognized driver state of the first image sample to establish the driver state recognition model comprises:

[0020] The first neural network model is subjected to redundant detection target and shallow feature processing according to the characteristic information of the driver state recognition.

[0021] Optionally, training the first neural network model based on the edge information, the background information and the recognized driver state of the first image sample to establish the driver state recognition model comprises:

[0022] For the recognized driver state, a loss function between a prediction box of the driver state and a labeled bounding box is determined;

[0023] The first neural network model is trained based on the edge information, the background information and the loss function of the first image sample to establish the driver state recognition model.

[0024] Optionally, the loss function is an Area_CIoU loss function.

[0025] Optionally, the multi-target tracking model is a second neural network model trained by the following method:

[0026] A driver tracking dataset is obtained, and the driver tracking dataset includes a plurality of second image samples with recognized results of drivers labeled;

[0027] For each second image sample in the plurality of second image samples, the following operations are performed:

[0028] Network adjustment is performed on pedestrian re-identification of the second neural network model according to the characteristic information of the driver state recognition;

[0029] extract feature information of the second image sample according to the second neural network model adjusted through the network;

[0030] identify a driver tracking result in the second image sample;

[0031] train the second neural network model based on the feature information of the second image sample and the identified driver tracking result to establish a multi-target tracking model.

[0032] Optionally, the training of the second neural network model based on the feature information of the second image sample and the identified driver tracking result to establish a multi-target tracking model comprises:

[0033] perform knowledge distillation training according to the second neural network model adjusted through the network.

[0034] Optionally, the training of the first neural network model based on the edge information, the background information and the identified driver state of the first image sample to establish a driver state recognition model comprises:

[0035] determine a ReLU function as an activation function of the first neural network model;

[0036] train the first neural network model based on the edge information, the background information, the activation function and the identified driver state of the first image sample to establish a driver state recognition model.

[0037] Optionally, the method further comprises:

[0038] perform a graph optimization operation on a convolution layer, a batch normalization layer and an activation function of the driver state recognition model after the driver state recognition model is established.

[0039] According to another aspect of the present application, a model training method of a driver state recognition model is provided, which comprises:

[0040] obtain a driver state change dataset, the driver state change dataset comprising a plurality of first image samples of a vehicle during stopping or driving;

[0041] for each first image sample in the plurality of first image samples, perform the following operations:

[0042] extract edge information and background information of the first image sample;

[0043] perform a receptive field enhancement processing on the first image sample according to the characteristic information of the driver state recognition;

[0044] identify a driver state in the first image sample;

[0045] Based on the edge information, background information and the recognized driver state of the first image sample, a first neural network model is trained to establish a driver state recognition model.

[0046] According to another aspect of the present application, a model training method of a multi-target tracking model is provided, the method comprising:

[0047] Obtaining a driver tracking dataset, the driver tracking dataset comprising a plurality of second image samples labeled with driver state recognition results;

[0048] For each second image sample in the plurality of second image samples, the following operations are performed:

[0049] According to the characteristic information of the driver state recognition, network adjustment is performed on pedestrian re-identification of a second neural network model;

[0050] Through the network-adjusted second neural network model, feature information of the second image sample is extracted;

[0051] Recognizing driver tracking results in the second image sample;

[0052] Based on the feature information of the second image sample and the recognized driver tracking results, a second neural network model is trained to establish a multi-target tracking model.

[0053] According to another aspect of the present application, a driver tracking device of a vehicle is provided, the device comprising:

[0054] An image acquisition unit configured to acquire a driver state image of the vehicle via an image sensor on the vehicle;

[0055] A driver state recognition unit configured to input the driver state image into a pre-trained driver state recognition model to obtain an identification result of the driver;

[0056] A multi-target tracking unit for inputting the identification result of the driver into a pre-trained multi-target tracking model to determine whether the driver has left the vehicle.

[0057] According to another aspect of the present application, an electronic device is provided, the electronic device comprising:

[0058] At least one processor; and

[0059] A memory in communication connection with the at least one processor; wherein,

[0060] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to perform the driver tracking method of the vehicle, or the model training method of the driver state recognition model, or the model training method of the multi-target tracking model according to any one of the embodiments of the application.

[0061] According to another aspect of the application, a computer readable storage medium is provided, which stores computer instructions for enabling a processor to implement the driver tracking method of the vehicle, or the model training method of the driver state recognition model, or the model training method of the multi-target tracking model according to any one of the embodiments of the application when executed by the processor.

[0062] According to another aspect of the application, a computer program product is provided, which comprises a computer program for implementing the driver tracking method of the vehicle, or the model training method of the driver state recognition model, or the model training method of the multi-target tracking model according to any one of the embodiments of the application when executed by a processor.

[0063] The technical scheme of the embodiment of the application acquires the driver state image of the vehicle through the image sensor on the vehicle; inputs the driver state image into the pre-trained driver state recognition model to obtain the recognition result of the driver; and inputs the recognition result of the driver into the pre-trained multi-target tracking model to determine whether the driver leaves the vehicle, thereby solving the detection problem of whether the driver of the vehicle leaves the vehicle, and being suitable for end-side deployment, and automatically realizing the recognition of the driver leaving the vehicle quickly and accurately to avoid illegal riding or loss of the vehicle.

[0064] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the application, nor is it used to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0065] In order to more clearly illustrate the technical solutions in the embodiments of the application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0066] Figure 1 is a flowchart of a driver tracking method of a vehicle according to an embodiment of the application;

[0067] Figure 2ais a flow chart of a model training method of a driver state recognition model according to Embodiment Two of the present application;

[0068] Figure 2b is a structural schematic diagram of a model receptive field enhancement processing and a redundant detection target according to Embodiment Two of the present application;

[0069] Figure 2c is a structural schematic diagram of a model shallow feature processing according to Embodiment Two of the present application;

[0070] Figure 2d is a structural schematic diagram of a driver state recognition model according to Embodiment Two of the present application;

[0071] Figure 3a is a flow chart of a model training method of a multi-target tracking model according to Embodiment Three of the present application;

[0072] Figure 3b is a model training structural schematic diagram of a multi-target tracking model according to Embodiment Three of the present application;

[0073] Figure 4 is a structural schematic diagram of a driver tracking device of a vehicle according to Embodiment Four of the present application;

[0074] Figure 5 is a structural schematic diagram of an electronic device for implementing the method of the present application. DETAILED DESCRIPTION

[0075] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should belong to the scope of protection of the present application.

[0076] It is to be understood that the terminology "first", "second" and the like used in the specification and the claims of the application as well as the foregoing drawings is merely intended to distinguish between similar objects and not necessarily for describing a special sequential order or priority. It is to be understood that the data so used can be interchanged, under appropriate circumstances, such that the embodiments of the application described herein can be practiced in other than the particular sequences or order shown herein. Furthermore, the term "comprising" and "including" and any variations thereof are intended to cover a non-exclusive inclusion, such that processes, methods, systems, articles, or apparatuses that comprise, include, or are otherwise accompanied by a list of steps or elements utilising any of these terms are not necessarily limited to the specific order or combination of steps or elements as set forth herein.

[0077] Embodiment One

[0078] Figure 1 is a flowchart of a driver tracking method of a vehicle according to an embodiment of the application. The embodiment can be applied to the case of detecting whether the driver of the vehicle is away from the vehicle. The method can be executed by a driver tracking device of the vehicle, which can be implemented in the form of hardware and / or software. The driver tracking device of the vehicle can be configured in an electronic device such as a computer, a vehicle controller, or a mobile phone, etc. As shown in Figure 1 the method comprises:

[0079] Step 110: acquiring a driver state image of the vehicle via an image sensor on the vehicle.

[0080] The driver state image is acquired after authorization by the user, and the acquisition method is reasonable and legal. The image sensor can be any suitable sensor capable of capturing images, such as a camera, a video camera, a camera lens, etc. The image sensor can be located at any position on the front of the vehicle as long as it can capture the driver of the vehicle in real time, for example, installed near or on the dashboard of the vehicle, preferably installed in the direction of the center axis of the vehicle. Optionally, the image sensor is a fisheye lens facing the upper body of the driver to track and detect the gestures of the driver.

[0081] Optionally, the vehicle is a vehicle controlled by a handlebar. For example, the vehicle is a two-wheeled vehicle such as a bicycle, a two-wheeled motorcycle, a two-wheeled electric bicycle, etc., or other three-wheeled vehicles, four-wheeled vehicles, etc. controlled by a handlebar.

[0082] Step 120: inputting the driver state image into a pre-trained driver state recognition model to obtain a recognition result of the driver.

[0083] In some embodiments, preprocessing of the driver state image can also be included to convert the driver state image into an image in a preset format. For example, the driver state image may not be directly applicable as input to the driver state recognition model, and therefore, the image can be converted into a preset format (for example, converted into yuv format by the Video Input module, or converted into a format suitable for the neural network model) so that it can be applied to the driver state recognition model to achieve rapid and accurate recognition of the driver state.

[0084] The driver state can be the position state of the driver, and the position of the driver in the driver state image can be specifically detected by the driver state recognition model.

[0085] Step 130: input the recognition result of the driver into the pre-trained multi-target tracking model to determine whether the driver has left the vehicle.

[0086] By tracking according to the recognition result of the driver, it can be determined whether the driver has left the vehicle, so as to strengthen the safety monitoring of the vehicle when the driver leaves the vehicle, and avoid loss of the vehicle; or to provide a safety prompt when the driver leaves the vehicle, and avoid illegal riding.

[0087] Embodiment two

[0088] Figure 2a is a flowchart of a model training method of a driver state recognition model according to embodiment two of the present application. The technical solution in this embodiment can be combined with each optional scheme in one or more of the above embodiments. As shown in Figure 2a The method comprises:

[0089] Step 210: obtain a driver state change data set, which includes a plurality of first image samples during the stopping or driving of the vehicle.

[0090] Specifically, the driver state change data set can be specifically an image of the driver moving in position on or around the vehicle. For example, an image of the driver on or around the vehicle. In the driver state change data set, the position of the driver can be labeled.

[0091] Step 220: for each first image sample in the plurality of first image samples, extract edge information and background information of the first image sample.

[0092] The first neural network model can be a YOLOv8 model, or other models based on deep neural network algorithms, to achieve detection of each image.

[0093] Since the edge information can locate the contour boundary of the target object, and the background information can determine the environmental range where the target object is located, simultaneously extracting both the edge information and the background information to train the first neural network model can accurately determine the position in a more complex scene, reduce false positives and missed cases, and better infer the complete shape and position of the occluded part when the target object is partially occluded, thereby improving the accuracy of the established driver state recognition model.

[0094] Since the obtained driver state change data set contains a large number of detection images of different angles, different times, different regions, different weather, and preferably a relatively balanced number of different categories of targets, the diversity and richness of the detection data set can be ensured, which is beneficial to improve the accuracy of the established driver state recognition model and adapt to diversified application scenarios.

[0095] Step 230: For each first image sample in the plurality of first image samples, performing receptive field enhancement processing on the first image sample according to the characteristic information of driver state recognition.

[0096] In the receptive field enhancement processing, the model structure can be improved according to the characteristic information of driver state recognition. The characteristic information of driver state recognition can be the characteristics that the driver has when detecting the position, such as the driver being a large target. The large target can be an object greater than or equal to 96x96 pixels. In the embodiment of the present application, the original YOLOv8 model can be improved to improve its detection capability for large targets.

[0097] Optionally, in order to improve the receptive field support of the first neural network model for larger targets to capture more global information, a dilated convolution (Dilated Convolution) can be added in front of the detection head of the YOLOv8 model. Preferably, the dilated rate of the dilated convolution is 3, which can realize the receptive field of the convolution to be 7x7. Compared with the original convolution kernel, the receptive field is expanded by 5.5 times, which can realize accurate driver state recognition.

[0098] The dilated rate of the dilated convolution can define the spacing between elements in the convolution kernel, which can effectively expand the receptive field of the convolution operation without increasing the amount of calculation.

[0099] For example, in the receptive field enhancement processing, the equivalent convolution kernel can be represented as k' = k + (k-1)x(d-1). Where k is the dilated convolution kernel size, d is the dilated number, and k is the equivalent convolution kernel. The receptive field can be represented as RF i+1 = RF i +(k'-1)xS i . Where RF i+1 is the current layer receptive field, and RFi For the upper receptive field, S i is a step size.

[0100] Step 240, for each first image sample in the plurality of first image samples, performing identifying a driver state in the first image sample.

[0101] Step 250, for each first image sample in the plurality of first image samples, performing training a first neural network model based on edge information, background information and the identified driver state of the first image sample, to establish a driver state recognition model.

[0102] Optionally, training the first neural network model based on the edge information, the background information and the identified driver state of the first image sample to establish the driver state recognition model comprises: performing a redundant detection target removal and a shallow feature removal on the first neural network model according to characteristic information of driver state recognition.

[0103] Through the redundant detection target removal and the shallow feature removal, the YOLOv8 model can be more suitable for detecting large target objects such as drivers, so as to improve the accuracy of driver state recognition.

[0104] Specifically, the redundant detection target removal can be removing detection heads in the YOLOv8 model that are more suitable for detecting small and medium target objects. Figure 2b is a structural schematic diagram of a model receptive field enhancement processing and a redundant detection target removal according to Embodiment Two of the present application. As Figure 2b shown, preferably, redundant detection heads Detect1 and Detect2 in the YOLOv8 model are removed, and only the detection head Detect3 is retained. Through the redundant detection target removal, the accuracy of driver state detection can be improved while reducing the amount of calculation.

[0105] In the YOLOv8 model, through the PAN-FPN structure, multi-level up-sampling and down-sampling are performed to fuse features of different scales, so as to enhance the expression ability of the model. In the present application, based on the consideration of the characteristic information of driver state recognition, in the PAN-FPN structure, the middle and deep features are more important, while the shallow features will introduce noise, leading to an increase in misjudgment of driver state recognition. Therefore, the YOLOv8 model can be subjected to shallow feature removal, so as to simplify the model architecture and improve the recognition accuracy of the model. Figure 2c is a structural schematic diagram of model shallow feature removal according to Embodiment Two of the present application. As Figure 2c shown, preferably, in the PAN-FPN structure of the YOLOv8 model, the shallow feature processing of P3 is removed, and the processing of the middle feature (P4) and the deep feature (P5) is retained.

[0106] Figure 2d is a structural schematic diagram of a driver state recognition model provided according to Embodiment Two of the present application. As shown in the figure, by removing redundant detection targets and shallow features from the YOLOv8 model and performing receptive field enhancement processing before the detection head Detect3, the accuracy of driver state recognition can be improved on the basis of simplifying the model structure. Figure 2d

[0107] Optionally, the first neural network model is trained based on the edge information, the background information and the recognized driver state of the first image sample to establish the driver state recognition model, including: determining a loss function between a prediction box of the driver state and a labeled bounding box for the recognized driver state; and training the first neural network model based on the edge information, the background information and the loss function of the first image sample to establish the driver state recognition model.

[0108] In driver state recognition, the design of the loss function directly affects the tendency and accuracy of the model in target recognition. In the present application, the design of the loss function can make the model more inclined to locate large target objects and improve the classification accuracy. When the driver state recognition model is established, the loss function can be the deviation between the prediction box of the driver state and the labeled bounding box.

[0109] Optionally, the loss function is an Area_CIoU loss function. The expression of the Area_CIoU loss function is wherein area represents the area of the prediction box bbox of the driver state, max_area represents the maximum area of bbox in the driver state change data set, wherein d is the distance between the center points of the prediction box and the labeled bounding box, such as the Euclidean distance, c is the diagonal length of the smallest closed region that can simultaneously contain the prediction box and the labeled bounding box, a and b are balance parameters, and v is a parameter for measuring the consistency of the aspect ratio. is the aspect ratio of the labeled bounding box, is the aspect ratio of the prediction box. IoU is the intersection over union of the overlapping degree of the prediction box and the labeled bounding box.

[0110] In the present application, the weight of the CIoU loss function is adjusted according to the area of the prediction box, and the area of the prediction box and the weight of the CIoU loss function are in a positive correlation, i.e., the larger the area of the prediction box, the greater the weight of the CIoU loss function. By adjusting the weight of the CIoU loss function according to the area of the prediction box, the model can be more inclined to the positioning accuracy of large target objects to improve the accuracy of driver state recognition. When adjusting the weight of the CIoU loss function according to the area of the prediction box, the weight of the CIoU loss function is adjusted by a square root function, which can prevent training instability caused by too large weight.​

[0111] The technical solution of the embodiment of the present invention solves the problem of low accuracy of the model in driver state recognition by removing redundant detection targets, removing shallow features, and enhancing the receptive field when establishing a driver state recognition model, and adjusting the weight of the CIoU function according to the area of ​​the predicted box. It improves the speed and accuracy of driver state recognition, reduces the misjudgment rate, enhances the generalization ability of the model, and adapts to recognition in multiple scenarios.

[0112] Example 3

[0113] Figure 3a 1 is a flow chart of a model training method for a multi-target tracking model provided in accordance with the third embodiment of the present invention. The technical solution in this embodiment can be combined with the various optional solutions in one or more of the above embodiments. Figure 3a As shown, the method includes:

[0114] Step 310: Acquire a driver tracking dataset, where the driver tracking dataset includes a plurality of second image samples labeled with recognition results of the driver.

[0115] Specifically, the annotations in the driver tracking dataset can be prediction boxes obtained by the driver state recognition model, or annotation bounding boxes annotated manually.

[0116] Step 320: For each second image sample in the plurality of second image samples, perform network adjustment on the pedestrian re-identification of the second neural network model according to the characteristic information of the driver state recognition.

[0117] Optionally, the second neural network model can be the multi-target tracking algorithm DeepSort. Pedestrian re-identification can be the ReID in the ResNet50 model that performs the re-identification part in the multi-target tracking algorithm DeepSort. The ResNet50 model has a strong feature extraction capability, but is inefficient. Based on the fact that the characteristic information of the driver state recognition in the application scenario of the present invention is a large target, and the aspect ratio of the prediction box or the annotation bounding box is greater than 1 and the category is single, the network is adjusted for pedestrian re-identification. Preferably, the input and output dimensions of pedestrian re-identification are adjusted, for example, the input dimension is increased according to a first preset adjustment ratio, and the output dimension is reduced according to a second preset adjustment ratio. Preferably, the input dimension is 256×128 and the output dimension is 64, which improves the reasoning performance of the model.

[0118] When adjusting the person re-identification network, the backbone network can be replaced. Preferably, the backbone network for person re-identification is replaced with the lightweight deep learning model architecture MobileNetV2 to improve the model's expressiveness and make it more suitable for driver tracking applications.

[0119] Step 330: Execute the second neural network model adjusted by the network for each second image sample in the plurality of second image samples to extract feature information of the second image sample.

[0120] Step 340 : For each second image sample in the plurality of second image samples, identify a driver tracking result in the second image sample.

[0121] Step 350: For each second image sample in the plurality of second image samples, perform training on the second neural network model based on the feature information of the second image sample and the identified driver tracking result to establish a multi-target tracking model.

[0122] Optionally, based on the feature information of the second image sample and the identified driver tracking result, the second neural network model is trained to establish a multi-target tracking model, including: performing knowledge distillation training based on the second neural network model after network adjustment.

[0123] Figure 3b Schematic diagram of the model training structure of a multi-target tracking model provided by the third embodiment of the present invention. Figure 3b As shown in the figure, in the knowledge distillation training, the original ResNet50 model is used as the teacher and the adjusted MobileNetV2 is used as the student. A ResNet50 teacher model is trained first. When training the student model, it is constrained by the original loss function (Loss_FN2) and the loss function (Loss_FN1) of the predicted output of ResNet50 and the predicted output of MobileNetV2. Preferably, the loss function when training the adjusted MobileNetV2 student model can be total_loss = α × Loss_FN1 + (1-α) × Loss_FN2. α is the adjustment coefficient and can be a value in the range [0.4, 0.6], such as 0.5.

[0124] The technical solution of the embodiment of the present invention improves the expressive power of the model and the ability to predict whether the driver will leave the vehicle by adjusting the network for pedestrian re-identification when establishing a multi-target tracking model. Specifically, the network structure is replaced, the input and output dimensions are adjusted, and the model is trained through knowledge distillation technology.

[0125] In a specific application of the embodiment of the application, whether the driver in the ROI region is determined by the established driver state recognition model, and whether the driver is the same person is determined by the established multi-target tracking model, so as to determine whether the driver leaves. After the driver state recognition model and the multi-target tracking model are established, the trained model can be fitted and finally converted into an onnx offline model to decouple from the original deep learning framework, so that it can run on various operating systems and hardware architectures without relying on the training framework. After obtaining the corresponding onnx offline model, the pruning of the DFL module can be performed to avoid operations such as Reshape, Transpose, Softmax, etc., thereby simplifying the operation and further improving the model operation speed.

[0126] The onnx offline model can be deployed on the end side, such as the vehicle side. In order to make the onnx offline model more suitable for the end side, the activation function can be adjusted, and the graph optimization operation can also be performed.

[0127] Optionally, the first neural network model is trained based on the edge information, background information and recognized driver state of the first image sample to establish the driver state recognition model, including: determining the ReLU function as the activation function of the first neural network model; and training the first neural network model based on the edge information, background information, activation function and recognized driver state of the first image sample to establish the driver state recognition model.

[0128] The activation function SiLU is replaced by ReLU. The formula of the SiLU activation function is SiLU = x * sigmoid, The formula of the ReLU activation function is The sigmoid function has an exponential operation. In some low-configuration neural network operation units, such as two-wheeled vehicles, the exponential operation cannot be performed when the Yolov8 network model is used for driver state recognition due to the limitation of device computing power. In this case, the data needs to be transported to a higher-power end, such as the cloud, model training end (ARM) or demand side platform (DSP), etc. for exponential operation and then transported to the low-configuration neural network operation unit. Due to multiple data transportation, the model inference speed is greatly reduced. The embodiment of the application replaces the complex Sigmoid function with the ReLU function, which can reduce data transportation and reduce data operation complexity, thereby improving the inference speed during driver state recognition.

[0129] Optionally, the driver tracking method of the vehicle further includes: after the driver state recognition model is established, performing a graph optimization operation on the convolution layer, the batch normalization layer and the activation function of the driver state recognition model.

[0130] By graph optimization, the network structure is changed into an operator, and memory is accessed once, so that the inference performance can be greatly improved. The formula of the convolution layer Conv can be understood as y = w x x, where y is the convolution calculation result, w is the weight, which is a parameter determined by a large amount of data learning, and x is the input data such as a picture. The formula of the batch normalization layer BN can be understood as The input x of the BN layer is the output y of the Conv layer, E[x] is the mean, is the variance, and γ and β are adjustment coefficients. The input x of the ReLU function is the output y of the BN layer. By graph optimization, the following formula can be obtained wherein, is the variance, w is the weight, μ B -b is the mean, and γ and β are adjustment coefficients.

[0131] By performing the activation function replacement and the graph optimization, the number of memory accesses can be reduced, the model inference performance can be improved, the model of the driver state recognition model and the multi-target tracking model can be quickly deployed on the end side, the deployment cost and time cost can be reduced, and the application in the intelligent transportation system or the security monitoring system and the like is facilitated.

[0132] Embodiment Four

[0133] Figure 4 is a structural schematic diagram of a driver tracking device of a vehicle according to an embodiment four of the present application. As Figure 4 shown, the device comprises:

[0134] An image acquisition unit 410 configured to acquire a driver state image of the vehicle via an image sensor on the vehicle;

[0135] A driver state recognition unit 420 configured to input the driver state image into a pre-trained driver state recognition model to obtain a recognition result of the driver;

[0136] A multi-target tracking unit 430 configured to input the recognition result of the driver into a pre-trained multi-target tracking model to determine whether the driver leaves the vehicle.

[0137] In some embodiments, the driver tracking device of the vehicle can further comprise a preprocessing unit configured to preprocess the driver state image to convert the driver state image into an image in a preset format.

[0138] It can be understood that, Figure 4 each unit of the driver tracking device of the vehicle shown in the above embodiment can be the same as the reference Figure 1The various steps in the driver tracking method of the vehicle described correspond. Thus, the operations, features and advantages described for the driver tracking method of the vehicle apply equally to the driver tracking apparatus of the vehicle and the units comprised therein. For the sake of brevity, certain operations, features and advantages are not described again here.

[0139] In the technical solutions of the embodiments of the present application, the collection, storage, use, processing, transmission, provision and disclosure of the personal information of the user (such as the driver state image) involved all comply with the relevant legal regulations and do not violate public order and good customs.

[0140] Embodiment five

[0141] Figure 5 A structural schematic diagram of an electronic device 10 that can be used to implement embodiments of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (such as headsets, glasses, watches, etc.), and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0142] As shown in Figure 5 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program that can be executed by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0143] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, a loudspeaker, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0144] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes various methods and processes described above, such as the driver tracking method of a vehicle, the model training method of a driver state recognition model, or the model training method of a multi-target tracking model.

[0145] In some embodiments, the driver tracking method of a vehicle, the model training method of a driver state recognition model, or the model training method of a multi-target tracking model can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the driver tracking method of a vehicle, the model training method of a driver state recognition model, or the model training method of a multi-target tracking model described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the driver tracking method of a vehicle, the model training method of a driver state recognition model, or the model training method of a multi-target tracking model by any other appropriate means, such as by means of firmware.

[0146] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0147] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, can cause instructions defined in the flow charts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package and partially on a remote machine or entirely on a remote machine or server.

[0148] In the context of the present application, a computer readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer readable storage medium can be a machine readable signal medium. More specific examples of the machine readable signal medium will include one or more lines of electrical communication, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0149] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0150] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0151] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0152] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be executed in parallel, executed in series, or executed in different orders, as long as the desired results of the technical solutions of the present disclosure can be achieved, and the present disclosure is not limited herein.

[0153] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for tracking a vehicle driver, characterized in that: The method comprises the following steps: acquiring a driver status image of the vehicle via an image sensor on the vehicle; Inputting the driver status image into a pre-trained driver status recognition model to obtain a driver recognition result; The driver identification result is input into a pre-trained multi-target tracking model to determine whether the driver leaves the vehicle.

2. The method according to claim 1, characterized in that The vehicle is a handlebar-controlled vehicle.

3. The method according to claim 1, characterized in that The image sensor is mounted on the dashboard of the vehicle or in an area of ​​the vehicle at a preset distance from the dashboard.

4. The method according to claim 1, wherein The vehicle is a two-wheeled vehicle.

5. A model training method for a driver state recognition model, characterized in that: include: Acquire a driver state change dataset, where the driver state change dataset includes a plurality of first image samples of the vehicle when the vehicle is stopped or moving; For each first image sample in the plurality of first image samples, perform the following operations: extracting edge information and background information of the first image sample; performing receptive field enhancement processing on the first image sample according to the characteristic information of the driver state recognition; identifying a driver state in the first image sample; Based on the edge information, background information and the identified driver state of the first image sample, a first neural network model is trained to establish a driver state recognition model.

6. A model training method for a multi-target tracking model, characterized in that: include: Acquire a driver tracking dataset, wherein the driver tracking dataset includes a plurality of second image samples annotated with driver state recognition results; For each second image sample in the plurality of second image samples, perform the following operations: Performing network adjustment on pedestrian re-identification of the second neural network model according to the characteristic information of the driver state recognition; extracting feature information of the second image sample through the second neural network model after network adjustment; identifying a driver tracking result in the second image sample; Based on the feature information of the second image sample and the identified driver tracking result, the second neural network model is trained to establish a multi-target tracking model.

7. A vehicle driver tracking device, characterized in that: include: an image acquisition unit configured to acquire an image of a driver state of the vehicle via an image sensor on the vehicle; a driver state recognition unit configured to input the driver state image into a pre-trained driver state recognition model to obtain a recognition result of the driver; The multi-target tracking unit is used to input the recognition result of the driver into a pre-trained multi-target tracking model to determine whether the driver leaves the vehicle.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the vehicle driver tracking method described in any one of claims 1 to 4, or the model training method of the driver state recognition model described in claim 5, or the model training method of the multi-target tracking model described in claim 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which are used to enable a processor to implement the vehicle driver tracking method described in any one of claims 1 to 4, or the model training method for the driver state recognition model described in claim 5, or the model training method for the multi-target tracking model described in claim 6 when executed.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the vehicle driver tracking method according to any one of claims 1 to 4, or the model training method for the driver state recognition model according to claim 5, or the model training method for the multi-target tracking model according to claim 6.

Images