Head orientation estimation method and system, and vehicle-mounted visual model training method and device
Through iterative training and target vision big model annotation data, the accuracy of the head orientation estimation of the vehicle-mounted vision model in complex scenarios is improved, the problem of low accuracy in the existing technology is solved, and more efficient driver monitoring and autonomous driving are achieved System security.
Patent Information
- Application Number
- CN202410904156.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-11-22
AI Technical Summary
Existing vehicle-mounted visual models have low accuracy in estimation of head orientation in complex scenarios, making it difficult to deal with posture diversity, occlusion, lighting changes, low resolution, and changes in expression and hair style.
Through iterative training methods, the target visual model is used to mark the original data head-oriented, generate a large amount of labeled data, train the first vehicle visual model to improve its discriminant characteristics, and then obtain the second vehicle visual model, which can be used in complex scenarios More accurately estimates the head orientation.
It significantly improves the accuracy of head orientation estimation, can better adapt to complex scenarios, and enhances the safety of the driver monitoring system and the reliability of the autonomous driving mode.
Smart Images

Figure CN119007165A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of computer vision technology, and in particular, relates to a method and system for estimating the orientation of a human head, and a method and device for training a vehicle-mounted vision model. Background Art
[0002] In the field of computer vision, head pose estimation is a key task, which aims to determine the orientation of a person's head in three-dimensional space.
[0003] For the estimation of the driver's head orientation, it is possible to detect in real time whether the driver is concentrating on driving during long-distance driving. By identifying the driver's head orientation, the driver monitoring system (DMS) can determine whether the driver has kept his eyes on the road ahead, or whether he has deviated from the normal driving posture due to fatigue, distraction, etc. If the driver monitoring system detects that the driver's head orientation is abnormal, such as looking down at the phone or looking out the window for a long time, it can issue a warning or take other measures, such as automatic deceleration, automatic switching to automatic driving mode, etc., to avoid traffic accidents caused by the driver's inattention to driving. In automatic driving mode, the driver monitoring system can determine whether the driver is ready to take over control of the vehicle by identifying the driver's head orientation, thereby more safely switching between automatic driving and manual driving.
[0004] Currently, the task of estimating head orientation faces many challenges in complex scenes, including pose diversity, occlusion, illumination changes, low resolution, and changes in facial expressions and hair styles. In-vehicle vision models are required to accurately estimate head orientation, but existing in-vehicle vision models have low accuracy in estimating head orientation when dealing with these complex scenes. Summary of the invention
[0005] The embodiments of the present application provide a head orientation estimation method, system, and vehicle-mounted vision model training method and device to improve the accuracy of the vehicle-mounted vision model in processing head orientation estimation tasks in complex scenes.
[0006] In a first aspect, an embodiment of the present application provides a method for estimating a head orientation, comprising:
[0007] Acquire a video or image of the driver's head;
[0008] estimating the head orientation of the driver's head video or image according to the second vehicle-mounted vision model to obtain an estimated value of the driver's head orientation;
[0009] Among them, the second vehicle-mounted visual model is obtained by iteratively training the first vehicle-mounted visual model based on the second data set; the second data set includes multiple second data, the second data is marked with head orientation, and the head orientation of the second data is the estimated value of the head orientation of the target visual large model.
[0010] In a second aspect, an embodiment of the present application provides a head orientation estimation system, including: a camera and a terminal device;
[0011] The camera is used to obtain and capture the driver's head video;
[0012] The terminal device has a second vehicle-mounted visual model built in, and the second vehicle-mounted visual model is used to estimate the head orientation of the driver's head video to obtain an estimated value of the driver's head orientation;
[0013] Among them, the second vehicle-mounted visual model is obtained by iteratively training the first vehicle-mounted visual model based on the second data set; the second data set includes multiple second data, the second data is marked with head orientation, and the head orientation of the second data is the estimated value of the head orientation of the target visual large model.
[0014] In a third aspect, an embodiment of the present application provides a vehicle-mounted visual model training method, comprising:
[0015] Determine a second data set, the second data set includes a plurality of second data, the second data is an image and / or video showing a human head, and the second data is annotated with a head orientation, and the head orientation of the second data is an estimated value of the head orientation of the target visual macro model;
[0016] The first vehicle-mounted vision model is iteratively trained using the second data set to obtain a second vehicle-mounted vision model, where the second vehicle-mounted vision model is used to estimate the direction of a human head for target data containing a human head.
[0017] In a possible implementation manner of the third aspect, determining the second data set includes:
[0018] Determine a first data set, where the first data set includes a plurality of first data, each of the first data is an image and / or video showing a human head, and the first data does not mark the direction of the human head;
[0019] Inputting any first data in the first data set into the target visual large model to obtain a first head orientation estimation value of the any first data;
[0020] Inputting the arbitrary first data into the first vehicle-mounted visual model to obtain a second head orientation estimation value of the arbitrary first data;
[0021] If the difference between the first head orientation estimation value and the second head orientation estimation value of any one of the first data is greater than a first difference threshold, the any one of the first data is used as one of the second data in the second data set.
[0022] In a possible implementation manner of the third aspect, determining the first data set includes:
[0023] Perform scene screening on the original data without labeled head orientation to screen out data including multiple scenes;
[0024] The screened data are preprocessed to obtain the first data set; the preprocessing includes performing one or more of the following on the data: cropping, scaling, and standardization.
[0025] In a possible implementation manner of the third aspect, after obtaining the first head orientation estimation value of any one of the first data, the method further includes:
[0026] performing smoothing processing on the first head orientation estimation value;
[0027] After obtaining the second head orientation estimation value of any one of the first data, the method further includes:
[0028] The second head orientation estimation value is smoothed.
[0029] In a possible implementation manner of the third aspect, the iteratively training the first vehicle-mounted vision model by using the second data set includes:
[0030] Monitoring the accuracy of the head orientation estimate of the first vehicle-mounted vision model during training;
[0031] If it is determined that the accuracy of the head orientation estimation value of the first vehicle-mounted vision model is improved, the first difference threshold is updated to the second difference threshold, and the first vehicle-mounted vision model is iteratively trained through the second data set again until the difference between the head orientation estimation value of the first vehicle-mounted vision model for any second data of a set proportion in the second data set and the head orientation estimation value of the target vision model for any second data is less than the second difference threshold, and the second difference threshold is less than the first difference threshold.
[0032] In a possible implementation manner of the third aspect, the method further includes:
[0033] Acquire a third data set, where the third data set includes a plurality of third data, where the third data is an image and / or video showing a human head, and the third data is marked with a direction of the human head;
[0034] The first visual large model is trained according to the third data set to obtain the target visual large model.
[0035] In a possible implementation manner of the third aspect, the method further includes: migrating the supervision information of the target large visual model to the second vehicle-mounted visual model through knowledge distillation.
[0036] In a fourth aspect, an embodiment of the present application provides a second vehicle-mounted visual model training device, the device comprising:
[0037] a determination module, configured to determine a second data set, wherein the second data set includes a plurality of second data, wherein the second data is an image and / or video showing a human head, and the second data is annotated with a head orientation, and the head orientation of the second data is an estimated value of the head orientation of the target visual macromodel;
[0038] The training module is used to iteratively train the first vehicle-mounted vision model using the second data set to obtain a second vehicle-mounted vision model, wherein the second vehicle-mounted vision model is used to estimate the head orientation of target data containing a human head.
[0039] In the fifth aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for estimating the head orientation as described in the first aspect or the method for training a vehicle-mounted visual model as described in any one of the third aspects is implemented.
[0040] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the head orientation estimation method as described in the first aspect, or the vehicle-mounted visual model training method as described in any one of the third aspects.
[0041] In the seventh aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes the head orientation estimation method described in the first aspect above, or the vehicle-mounted visual model training method as described in any one of the third aspects.
[0042] The beneficial effect of the embodiment of the present application compared with the prior art is: the embodiment of the present application uses a large target vision model to annotate the original data with head orientation, obtains a large amount of second data annotated with head orientation, and trains the first vehicle-mounted vision model through the second data set, so that the first vehicle-mounted vision model can learn highly discriminative features from the original data. The second vehicle-mounted vision model obtained after the training is completed has significantly improved accuracy in estimating head orientation. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0044] Figure 1 is a flowchart of a method for estimating head orientation provided by an embodiment of the present application;
[0045] Figure 2 is a structural block diagram of a head orientation estimation system provided by an embodiment of the present application;
[0046] Figure 3 It is a flowchart of a vehicle-mounted visual model training method provided in one embodiment of the present application;
[0047] Figure 4 It is a structural block diagram of a vehicle-mounted visual model training device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0048] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0049] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0050] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0051] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0052] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0053] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.
[0054] Before introducing the embodiments of the present application, the following terms are first introduced:
[0055] 1) Visual big models are artificial intelligence technologies that can simulate the working principle of the human visual system and perform in-depth analysis and processing of images. They are built using a large amount of data and advanced algorithms, and have a high degree of recognition, analysis, and expression capabilities. These models usually contain tens of millions of parameters or more, which ensures their powerful processing capabilities. Visual big models are generally built using structures such as convolutional neural networks (CNN) or deformable convolutional neural networks (Transformers), and continuously optimize their algorithms through machine learning.
[0056] The visual big model has excellent performance in various visual tasks, especially showing rich representation capabilities when dealing with complex scenes and subtle changes. The visual big model can capture rich visual details and hierarchical structures, which are beyond the reach of the on-board visual model. There are many types of visual big models, including but not limited to: Moco (Momentum Contrast for Unsupervised Visual Representation Learning), SimCLR (Simple Framework for Contrastive Learning of Visual Representations), MAE (Masked Autoencoders), EVA (Exploring the Limits of Masked Visual Representation Learning at Scale), CLIP (Contrastive Language-Image Pre-training), DINOv2 (Learning Robust Visual Features without Supervision). Preferably, the visual big model uses DINOv2.
[0057] DINOv2 uses self-supervised learning, which learns powerful visual representations by pre-training on large-scale unlabeled data. The idea behind this method is to learn features by letting the large model estimate the consistency between images from different perspectives, rather than relying on manually labeled data. This not only reduces the reliance on a large amount of labeled data, but also enables the large model to learn more generalized feature representations.
[0058] DINOv2 is well suited for head orientation estimation tasks because it can capture subtle differences in human faces and learn key information about head orientation from them. Compared to traditional supervised learning models, DINOv2 can understand complex visual patterns without explicit labels, which makes it perform better when processing head images with masks, glasses, or in different lighting conditions.
[0059] 2) The in-vehicle vision model is a small model deployed on the terminal device. Due to its local operation and low computing power, its performance is limited when processing complex scenes, resulting in a large deviation between the estimated head orientation and the true value in complex scenes. Specifically, complex scenes include but are not limited to: head occlusion, lighting changes, low resolution, the head is far away from the camera in the car, facial expressions change, hairstyle change, wearing a hat, sunglasses, or a mask, etc.
[0060] Figure 1A schematic diagram of a process of estimating a head orientation disclosed in an embodiment of the present application is shown, and the method includes:
[0061] S110: The terminal device obtains the driver's head video or image.
[0062] As an example, the terminal device may be a server, a mobile phone, a tablet computer, or a vehicle.
[0063] For example, the terminal device can obtain the driver's head video in real time or periodically according to a preset period through an image acquisition device (such as a camera) installed in the car; for example, the driver's head image is collected every 1 second. Exemplarily, the driver monitoring system (DMS) collects a frame of image every 40ms, and estimates the head orientation of the frame image every 60ms through the second on-board visual model. Optionally, it is also possible to obtain the driver's head video or image when it is detected that the set conditions are met. For example, by detecting the pressure of the driver's seat and backrest, it is determined whether the driver has reached for a mobile phone or sat still for a long time, and when these set conditions occur, the driver's head video or image is obtained.
[0064] S120: The terminal device estimates the head orientation of the driver's head video or image according to the second on-board visual model to obtain an estimated value of the driver's head orientation; wherein the second on-board visual model is obtained by iteratively training the first on-board visual model based on the second data set; the second data set includes multiple second data, and the second data is marked with a head orientation, and the head orientation of the second data is the estimated value of the head orientation of the target visual large model.
[0065] The terminal device has a second vehicle-mounted visual model, or the terminal device can obtain the second vehicle-mounted visual model from a cloud server or other device.
[0066] The second vehicle-mounted vision model may be obtained by training the terminal device itself, or may be provided to the terminal device after being trained by other devices, which is not limited in the embodiments of the present application.
[0067] As an example, taking the terminal device as a vehicle, the second vehicle-mounted visual model is arranged in the vehicle-mounted chip of the vehicle.
[0068] The target visual large model needs to be placed in a device with strong computing power, fast data transmission speed and large storage space, usually in a local server or cloud server.
[0069] Specifically, the head orientation has angle values in three directions, namely, yaw, pitch, and roll.
[0070] As an example, the target visual big model may be the first visual big model, which may be understood as the initial visual big model. Alternatively, the target visual big model is obtained by fine-tuning (ie, training) the first visual big model through labeled head orientation data.
[0071] Traditional machine learning methods rely on hand-designed feature extractors, which are often unable to cope with changes in head orientation in complex scenes. In contrast, the vehicle-mounted vision model of the embodiment of the present application adopts a deep learning model, preferably a convolutional neural network (CNN), which can automatically learn the hierarchical feature representation of the image, so as to better capture the posture and orientation of the head. In addition, the deep learning model can enhance its generalization ability through data enhancement technology. For example, different real-world situations are simulated by means of random rotation, scaling, cropping and color transformation. These techniques not only improve the performance of the deep learning model on the training data, but also enable the deep learning model to better adapt to new and unseen data. And by reducing the amount of calculation and model size through network pruning, quantization and other technologies, the deep learning model can be run under limited computing resources to meet the needs of real-time processing.
[0072] The first vehicle vision model can be understood as an initial vehicle vision model (an untrained vehicle vision model) or an existing vehicle vision model. The second vehicle vision model is obtained by iteratively training the first vehicle vision model based on the second data set.
[0073] In this embodiment, the original data is annotated with the head direction through the target vision large model to obtain a large amount of second data annotated with the head direction. The existing first vehicle-mounted vision model is trained with the second data set, so that the first vehicle-mounted vision model can directly learn highly discriminative features from the original image data. The second vehicle-mounted vision model obtained after the training is completed has significantly improved accuracy in estimating the head direction.
[0074] In some optional implementations, the second data is data in which a difference between an estimated value of the head orientation of the target large visual model and an estimated value of the head orientation of the first vehicle-mounted visual model is greater than a first difference threshold.
[0075] Specifically, the target visual large model and the first vehicle-mounted visual model respectively estimate the head orientation of the same data, and compare the head orientation estimation values obtained by the two. If the difference is greater than the first difference threshold, the data is used as the second data. Therefore, the second data in the second data set are all data in which the head orientation estimation value of the first vehicle-mounted visual model differs greatly from the head orientation estimation value of the target visual large model. The second data is pre-labeled by the target visual large model, and the data showing significant differences between the head orientation estimation value of the first vehicle-mounted visual model and the head orientation estimation value of the target visual large model are mined, and the mined data is used as the training data of the first vehicle-mounted visual model, which not only greatly increases the labeled data, but also greatly improves the training efficiency. The second vehicle-mounted visual model obtained after iterative training of the first vehicle-mounted visual model based on the second data set can greatly improve the accuracy of head orientation estimation for complex scenes.
[0076] Figure 2 A structural block diagram of a head orientation estimation system disclosed in an embodiment of the present application is shown, and the system includes: a camera 210 and a terminal device 220.
[0077] The camera 210 is used to collect the driver's head video and to provide the collected driver's head video to the terminal device 220.
[0078] The terminal device 220 has a built-in second vehicle-mounted vision model 221, which is used to estimate the head orientation of the driver's head video to obtain an estimated value of the driver's head orientation; wherein the second vehicle-mounted vision model is obtained by iteratively training the first vehicle-mounted vision model based on the second data set; the second data set includes multiple second data, the second data is marked with a head orientation, and the head orientation of the second data is an estimated value of the head orientation of the target visual large model.
[0079] This embodiment deploys the trained second vehicle-mounted vision model on the vehicle to estimate the driver's head direction in real time, thereby helping the driver monitoring system (DMS) to determine whether the driver maintains his gaze on the road ahead, or whether he deviates from the normal driving posture due to fatigue, distraction, etc.
[0080] As an example, the second data is data in which a difference between an estimated value of a head orientation of a target visual large model and an estimated value of a head orientation of a first vehicle-mounted visual model is greater than a first difference threshold.
[0081] The training efficiency can be greatly improved by training the first onboard vision model using the second data that is inconsistent with the estimated value of the target vision model.
[0082] Figure 3A schematic diagram of a process flow of a vehicle-mounted visual model training method provided in an embodiment of the present application is shown, and the method includes:
[0083] S310: The terminal device determines a second data set, the second data set includes multiple second data, the second data is an image and / or video showing a human head, and the second data is marked with a head orientation, and the head orientation of the second data is an estimated value of the head orientation of the target visual large model.
[0084] Specifically, video data is also composed of images. Therefore, whether the second data is an image or a video, the image is ultimately processed and estimated.
[0085] The target vision large model has higher accuracy in estimating the head orientation of the second data. Therefore, the estimated value of the head orientation of the second data by the target vision large model can be used as the "true value", thereby greatly increasing the training data labeled with head orientation.
[0086] S320: The terminal device iteratively trains the first vehicle-mounted vision model using the second data set to obtain a second vehicle-mounted vision model, where the second vehicle-mounted vision model is used to estimate the direction of a human head for target data containing a human head.
[0087] In this embodiment, by using the target visual large model's estimated value of the head orientation of the second data as training data for the first vehicle-mounted visual model, the accuracy of the target visual large model's estimation of the head orientation of the original data obtained after training will be greatly improved.
[0088] In some optional implementations, determining the second data set in S310 includes:
[0089] S410: Determine a first data set, where the first data set includes a plurality of first data, each of which is an image and / or video showing a human head, and the first data does not mark the direction of the human head.
[0090] Specifically, the first data set can be obtained from public data sets, inventory data, etc. The first data in the first data set can be an image, a video, or a video and an image; the image needs to show a person's head, and each image frame or part of the image frames in the video needs to show a person's head. Preferably, the first data set includes video data showing the driver's head, and these data can reflect the driving scene.
[0091] S420: Input any first data in the first data set into the target visual model to obtain a first head orientation estimation value of any first data.
[0092] Furthermore, after obtaining the first head orientation estimation value, the method further includes performing a smoothing process on the first head orientation estimation value.
[0093] S430: Input any first data into the first vehicle-mounted visual model to obtain a second head orientation estimation value of any first data.
[0094] Furthermore, after obtaining the second head orientation estimation value, the method further includes performing a smoothing process on the second head orientation estimation value.
[0095] Specifically, when the target visual large model and the first vehicle-mounted visual model process the first data in the first data set, a time series processing method is adopted, specifically, the estimated values of the continuous image frames of the video are smoothed, thereby solving the influence of the single-frame angle jump and improving the estimation stability of the target visual large model and the first vehicle-mounted visual model. Taking 25 frames per second of video as an example, that is, 25 head images are collected per second, under normal circumstances, there will be no significant changes between two adjacent image frames. In order to solve the influence of the single-frame angle jump, the estimated value after smoothing is used, and the angle between two adjacent image frames will not change significantly.
[0096] S440: If the difference between the first head orientation estimation value and the second head orientation estimation value of any one of the first data is greater than a first difference threshold, use any one of the first data as a second data in the second data set.
[0097] It should be noted that when comparing the first head orientation estimation value and the second head orientation estimation value, the first head orientation estimation value and the second head orientation estimation value of the same data are compared.
[0098] Specifically, this embodiment mines out data with significant differences by comparing the performance of the target visual large model and the first vehicle-mounted visual model on different data. The first head orientation estimate estimated by the target visual large model is relatively accurate, and the first head orientation estimate can be used as the "true value", while the second head orientation estimate estimated by the first vehicle-mounted visual model may differ greatly from the true value. If the difference between the first head orientation estimate and the second head orientation estimate of an image is less than the first difference threshold, it means that the second head orientation estimate estimated by the first vehicle-mounted visual model is relatively accurate, and the image does not have much training value for the first vehicle-mounted visual model. If the difference between the first head orientation estimate and the second head orientation estimate of an image is greater than the first difference threshold, it means that the second head orientation estimate estimated by the first vehicle-mounted visual model is inaccurate, and it can be considered that the image is an image in a complex scene, then the image has a greater training value for the first vehicle-mounted visual model, and the image needs to be placed in the second data set. By comparing the performance differences between the two models on the head orientation task, we can accurately identify the data that the first on-board vision model cannot process, provide targeted training data for the small model, and improve the targeting and efficiency of data enhancement.
[0099] Specific screening strategy: The default value of N is 5, and the default value of L is 5. min Represents the minimum loss function. By comparing the target visual model fine-tuned by DINOv2 and the first vehicle visual model trained by CNN, the output value and The model performance is evaluated by the absolute difference in different directions (yaw, pitch, roll). If the absolute difference in any of these three directions is less than the threshold N degrees, it is considered that the estimated outputs of the first on-board vision model and the target vision large model are sufficiently consistent, and the image sample is considered to be a simple image sample, and the first on-board vision model can accurately identify the head orientation value. If the difference between the outputs of the first on-board vision model and the target vision large model in any direction exceeds N degrees, it is considered that the estimation of the first on-board vision model in this direction is not consistent enough, and the estimated value of the target vision large model can be directly used as the true value to improve the generalization ability of the first on-board vision model.
[0100] The second data in the second data set of the embodiment of the present application are all data that are inaccurately estimated by the second vehicle-mounted vision model. Therefore, training the first vehicle-mounted vision model by using the second data set can, on the one hand, reduce the training of data that is not very helpful to the first vehicle-mounted vision model, and on the other hand, can quickly improve the accuracy of the first vehicle-mounted vision model in estimating the direction of the human head.
[0101] In some optional implementations, determining the first data set in S410 includes:
[0102] S510: Perform scene screening on the original data without marking the head orientation, so as to screen out data including multiple scenes.
[0103] S520: Preprocess the filtered data to obtain a first data set; the preprocessing includes performing one or more of the following on the data: cropping, scaling, and standardization.
[0104] Specifically, scene screening from images and / or video data without annotated head orientation can be done manually, but considering that it takes a lot of time and effort, it is preferred to use a large target visual model for automated screening. The screened scenes include but are not limited to: complex scenes such as head occlusion, lighting changes, low resolution, long distance between the head and the camera in the car, expression changes, hairstyle changes, wearing a hat, sunglasses, and a mask.
[0105] This embodiment estimates the head orientation of the first data in the first data set through the target vision model, filters out data from different scenes, makes the first data set more comprehensive, and improves the accuracy and robustness of the second vehicle-mounted vision model in various complex scenes.
[0106] In some optional implementations, iteratively training the first vehicle-mounted vision model using the second data set in S320 includes:
[0107] S610: Monitor the accuracy of the head orientation estimation value of the first vehicle-mounted vision model during the training process.
[0108] Specifically, during the training process, the accuracy of the estimation of the first onboard visual model will be continuously monitored and compared with the estimation value of the target visual model to determine whether the accuracy of the estimation of the first onboard visual model is improved.
[0109] S620: If it is determined that the accuracy of the head orientation estimation value of the first vehicle-mounted vision model is improved, the first difference threshold is updated to the second difference threshold, and the iterative training of the first vehicle-mounted vision model is re-executed using the second data set until the difference between the head orientation estimation value of any second data of the first vehicle-mounted vision model for a set proportion of the second data in the second data set and the head orientation estimation value of any second data of the target vision model is less than the second difference threshold, and the second difference threshold is less than the first difference threshold.
[0110] Specifically, since all the data in the second data set are data that are inaccurately estimated by the second vehicle-mounted visual model, after training, the estimated value of the first vehicle-mounted visual model gradually tends to the estimated value of the target visual model. Exemplarily, if during the training process, it is detected that 10% of the data in the second data set can be accurately estimated by the first vehicle-mounted visual model, that is, the difference between the first head orientation estimate and the second head orientation estimate of 10% of the image data in the second data set is less than 5°, then the first difference threshold is adjusted to the second difference threshold, for example, the second difference threshold can be 4.5°, and the iterative training is repeated until the difference between the first head orientation estimate and the second head orientation estimate of a set proportion of the second data in the second data set is less than the preset target difference threshold, at which time the second vehicle-mounted visual model is obtained. The set ratio can be 100%, or 80%, etc. The preset target difference threshold can be selected as 3° or 2°. It should be noted that when comparing the difference threshold between the first head orientation estimate and the second head orientation estimate, since both the first head orientation estimate and the second head orientation estimate include angles in three directions, namely, yaw, pitch and roll, the three directions of the first head orientation estimate and the second head orientation estimate should be compared separately, and only when the differences in all three directions are less than the first difference threshold, is it considered that the difference between the first head orientation estimate and the second head orientation estimate is less than the first difference threshold.
[0111] This embodiment forms a closed-loop feedback mechanism by monitoring the accuracy of head orientation estimation of the first vehicle-mounted vision model during the training process and lowering the first difference threshold accordingly when the accuracy improves, so that the performance of the first vehicle-mounted vision model continues to improve. Ultimately, the first vehicle-mounted vision model achieves accuracy and stability similar to that of the target large vision model in actual applications, thereby determining the first vehicle-mounted vision model as the second vehicle-mounted vision model.
[0112] In some optional embodiments, the method further comprises:
[0113] S710: Acquire a third data set, where the third data set includes a plurality of third data, where the third data is an image and / or video showing a human head, and the third data is marked with a direction of the human head.
[0114] Specifically, the third data annotation may be manually annotated data.
[0115] S720: Train the first visual large model according to the third data set to obtain a target visual large model.
[0116] Specifically, the first vision big model can be used as a pre-trained model to capture high-quality feature representations that can be applied to specific tasks through a fine-tuning process. Since the representations learned by the first vision big model in the pre-training stage are highly generalized, even a small amount of labeled data is sufficient to train the model to achieve high performance, which is very beneficial for data-limited scenarios. By performing the head labeling task with the fine-tuned target vision big model, the first head orientation estimate can be made closer to the true value.
[0117] Exemplarily, fine-tuning the target visual model can be achieved in the following way: the input image size of the first visual model is 224 pixels x 224 pixels, and four data enhancement methods are used: random perturbation of the face frame, horizontal flipping, Gaussian blur, and color enhancement, so that the first visual model input is more robust and stable. In order to further improve the stability of the head orientation model regression, multi-branch regression is performed using independent heads for the three angle regressions of head posture yaw, pitch, and roll.
[0118] To address the problem of difficult training convergence, the regression space can be discretized and an intermediate classification layer can be introduced to learn the distribution of the data. At the same time, the direct regression of the posture is converted into the expected value of the data distribution and the pre-set anchor point. In addition, a weighted loss function is introduced to adjust the contribution of different distribution data to network training.
[0119] Assume that the loss function L of each branch of yaw, pitch, and roll is angleThe angle can be one of the three angles of yaw, pitch, and roll. The above loss function design can be obtained α is set to 1, where y represents the true value of the head direction. Represents the estimated value of the model, that is, the regression value output by the model. H represents the cross entropy loss (CrossEntropy Loss) in the classification layer. When preparing training data, for the input training data [pitch, yaw, roll], [0, 180] degrees can be divided into 61 bins, and then labels are generated according to the bins where the three angles are located, and used to train the classification layer; according to the probability distribution from the classification layer and the values of the 61 bins, the expected regression value is obtained, and finally the classification loss and regression loss are integrated in a weighted manner. MSE stands for Mean Squared Error. In particular, for the yaw angle value yaw, which may cross the boundary of -180 to 180 degrees, when the difference between the estimated value and the true value is greater than 180 degrees, the difference is adjusted to the threshold period of -180 to 180 to correctly handle this periodicity. So the total loss function L can be obtained, L = L yaw +L pitch +L roll .
[0120] This embodiment uses a small amount of data with head orientation annotations to train a first visual large model (such as DINOv2), and applies the obtained target visual large model to the head orientation estimation task. It can effectively handle complex scenes such as posture diversity, occlusion, lighting changes, wearing masks, hats, sunglasses, etc., and has high accuracy and stability.
[0121] In some optional implementations, the vehicle-mounted vision model training method further includes:
[0122] S810: Migrate the supervision information of the target large visual model to the second vehicle-mounted visual model through knowledge distillation.
[0123] Knowledge distillation is a technique that guides the training of a second onboard visual model by leveraging the knowledge of the target visual model. The basic idea is to allow the second onboard visual model to learn the decision boundaries and feature representations of the target visual model, thereby improving its performance.
[0124] In this embodiment, the estimated value of the target visual large model is used as a soft label, that is, the output probability distribution of the target visual large model is used as a target to guide the training of the second vehicle-mounted visual model.
[0125] Exemplarily, the supervision information of the target visual model can be transferred to the second onboard visual model by knowledge distillation in the following way: the image is input into the second onboard visual model, and the image is classified into yaw, pitch and roll by the classifier. The cross entropy loss function is used as the first distillation loss function to measure the difference between the output of the second onboard visual model and the soft label of the target visual model. The cross entropy loss function can be expressed as: Among them, y i ' is the true labeling probability distribution estimated by the target visual model, is the estimated output of the second onboard vision model, and i represents the index of the category.
[0126] In addition to the cross entropy loss function, the mean square error loss function (MSE) is also combined as the second distillation loss function to comprehensively consider the losses of classification and regression tasks. The comprehensive loss function can be expressed as: L KD =λL CE +(1-λ)L MSE ,λ is used to balance the classification loss and regression loss.
[0127] When the sample has the target visual model estimate as the true value, the loss function L = L angle +L KD , where L angle That is, using the estimated value of the target visual model as the true value for supervision, L KD It is supervised by knowledge distillation. When the sample does not have the target visual model estimate as the true value, the loss function L = L KD , note that L angle and L KD Contains three directions: yaw, pitch, and roll.
[0128] On the one hand, this embodiment uses the pre-labeled data generated by the target visual large model to perform true value supervised training of the first vehicle-mounted visual model to ensure that the second vehicle-mounted visual model obtained after training can learn accurate head direction estimation capabilities; on the other hand, the knowledge distillation technology is used to further optimize the second vehicle-mounted visual model. In the knowledge distillation process, the output of the target visual large model is used as a soft label to guide the training of the second vehicle-mounted visual model, thereby transferring the rich representation capabilities and fine-grained knowledge of the target visual large model to the second vehicle-mounted visual model.
[0129] Through this dual-track training strategy that combines truth-value supervision and knowledge distillation, the second on-board vision model can achieve more accurate and stable head orientation estimation in complex and changeable on-board environments. Not only does it improve the performance of the second on-board vision model, but it also utilizes the powerful representation capabilities of the target vision large model to achieve the optimal balance between data efficiency and computational efficiency, which is particularly suitable for on-board application scenarios with limited resources and high requirements for real-time response. For data from complex scenes, the estimated value of the existing second on-board vision model differs from that of the target vision large model by more than ten degrees. After training with the method of the embodiment of the present application, the estimated value of the second on-board vision model differs from that of the target vision large model by 2-3°, which greatly improves the estimation accuracy of the second on-board vision model.
[0130] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0131] Corresponding to the second vehicle-mounted visual model training method in the above embodiment, Figure 4 A structural block diagram of the vehicle-mounted visual model training device 900 provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.
[0132] Reference Figure 4 , the device comprises:
[0133] Determination module 910 is used to determine a second data set, the second data set includes multiple second data, the second data is an image and / or video showing a human head, and the second data is marked with a head orientation, and the head orientation of the second data is an estimated value of the head orientation of the target visual large model.
[0134] The training module 920 iteratively trains the first vehicle-mounted vision model using the second data set to obtain a second vehicle-mounted vision model, where the second vehicle-mounted vision model is used to estimate the head orientation for target data containing a human head.
[0135] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0136] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0137] An embodiment of the present application also provides a terminal device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on at least one processor, and when the processor executes the computer program, the steps in any of the above-mentioned method embodiments are implemented.
[0138] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.
[0139] An embodiment of the present application provides a computer program product. When the computer program product runs on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.
[0140] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to the terminal device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.
[0141] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0142] The computer program code for performing the operation of the embodiment of the application can be written in one or more programming languages or their combination, and the programming language includes object-oriented programming languages-such as python, Java, Smalltalk, C++, and also includes conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed completely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer by any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect to the Internet).
[0143] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0144] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0145] In the embodiments provided in the present application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0146] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0147] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A method for estimating a head orientation, characterized in that: include: Acquire a video or image of the driver's head; estimating the head orientation of the driver's head video or image according to the second vehicle-mounted vision model to obtain an estimated value of the driver's head orientation; The second vehicle-mounted vision model is obtained by iteratively training the first vehicle-mounted vision model based on the second data set; The second data set includes a plurality of second data, wherein the second data is marked with a head orientation, and the head orientation of the second data is an estimated value of the head orientation of the target visual large model.
2. A head orientation estimation system, characterized in that: include: Cameras and terminal equipment; The camera is used to obtain and capture the driver's head video; The terminal device has a second vehicle-mounted visual model built in, and the second vehicle-mounted visual model is used to estimate the head orientation of the driver's head video to obtain an estimated value of the driver's head orientation; The second vehicle-mounted vision model is obtained by iteratively training the first vehicle-mounted vision model based on the second data set; The second data set includes a plurality of second data, wherein the second data is marked with a head orientation, and the head orientation of the second data is an estimated value of the head orientation of the target visual large model.
3. A vehicle-mounted vision model training method, characterized in that: include: Determine a second data set, the second data set includes a plurality of second data, the second data is an image and / or video showing a human head, and the second data is annotated with a head orientation, and the head orientation of the second data is an estimated value of the head orientation of the target visual macro model; The first vehicle-mounted vision model is iteratively trained using the second data set to obtain a second vehicle-mounted vision model, where the second vehicle-mounted vision model is used to estimate the direction of a human head for target data containing a human head.
4. The vehicle-mounted visual model training method according to claim 3, characterized in that: The determining of the second data set comprises: Determine a first data set, where the first data set includes a plurality of first data, each of the first data is an image and / or video showing a human head, and the first data does not mark the direction of the human head; Inputting any first data in the first data set into the target visual large model to obtain a first head orientation estimation value of the any first data; Inputting the arbitrary first data into the first vehicle-mounted visual model to obtain a second head orientation estimation value of the arbitrary first data; If the difference between the first head orientation estimation value and the second head orientation estimation value of any one of the first data is greater than a first difference threshold, the any one of the first data is used as one of the second data in the second data set.
5. The vehicle-mounted visual model training method according to claim 4, characterized in that: The determining of the first data set comprises: Perform scene screening on the original data without labeled head orientation to screen out data including multiple scenes; The screened data are preprocessed to obtain the first data set; the preprocessing includes performing one or more of the following on the data: cropping, scaling, and standardization.
6. The vehicle-mounted visual model training method according to claim 4, characterized in that: After obtaining the first head orientation estimation value of any one of the first data, the method further includes: performing smoothing processing on the first head orientation estimation value; After obtaining the second head orientation estimation value of any one of the first data, the method further includes: The second head orientation estimation value is smoothed.
7. The vehicle-mounted visual model training method according to claim 4, characterized in that: The iterative training of the first vehicle-mounted vision model using the second data set includes: Monitoring the accuracy of the head orientation estimate of the first vehicle-mounted vision model during training; If it is determined that the accuracy of the head orientation estimation value of the first vehicle-mounted vision model is improved, the first difference threshold is updated to the second difference threshold, and the first vehicle-mounted vision model is iteratively trained through the second data set again until the difference between the head orientation estimation value of the first vehicle-mounted vision model for any second data of a set proportion in the second data set and the head orientation estimation value of the target vision model for any second data is less than the second difference threshold, and the second difference threshold is less than the first difference threshold.
8. The vehicle-mounted visual model training method according to claim 3, characterized in that: The method further comprises: Acquire a third data set, where the third data set includes a plurality of third data, where the third data is an image and / or video showing a human head, and the third data is marked with a direction of the human head; The first visual large model is trained according to the third data set to obtain the target visual large model.
9. The vehicle-mounted visual model training method according to any one of claims 3 to 8, characterized in that: The method also includes: migrating the supervision information of the target large visual model to the second vehicle-mounted visual model through knowledge distillation.
10. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for estimating the head orientation as described in claim 1 or the method for training a vehicle-mounted visual model as described in any one of claims 3 to 9 is implemented.
11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for estimating the head orientation as described in claim 1 or the method for training a vehicle-mounted visual model as described in any one of claims 3 to 9 is implemented.
Citation Information
Patent Citations
Visual perception system for accurately perceiving head action of motor vehicle driver
CN112329566A
Human face pose estimation method and system based on knowledge distillation
CN112818969A
Sight line estimation method based on pseudo label
CN114973356A
Training method and device of automatic driving perception model, equipment and medium
CN115879535A
Perception model training method and device, computer equipment and storage medium
CN116681123A
