Abnormality detection method, model training method, device and vehicle

Abstract

The invention relates to an anomaly detection method and device, a model training method and device and a vehicle. The anomaly detection method comprises the following steps: acquiring a target recognition image of a vehicle cabin; performing anomaly detection according to the target recognition image through a pre-trained detection model to obtain anomaly detection information, the anomaly detection information being used for representing an anomaly reason of the target recognition image; the detection model is obtained by training of training data, the training data comprises an image sample and a label of the image sample, the label of the image sample comprises a main task label and an auxiliary task label, the main task label is used for representing an abnormal reason of the image sample, and the auxiliary task label is used for representing whether the image sample contains target image information or not; the target image information is used for assisting in detecting abnormal reasons. According to the technical scheme, on the basis that additional hardware setting is not needed, the anomaly detection precision of the target recognition image of the intelligent cockpit is improved, the anomaly false alarm rate is reduced, and then the user safety is improved.

Description

Technical Field

[0001] This disclosure relates to the field of intelligent cockpit technology, and in particular to an anomaly detection method, model training method, device, and vehicle. Background Technology

[0002] With the development of smart cockpit technology, anomaly detection systems are typically installed in smart cockpits to detect anomalies. These systems can detect whether the cockpit's recognition images (i.e., images captured by cameras) are abnormal. By detecting anomalies in the cockpit's recognition images, user safety can be ensured. Summary of the Invention

[0003] To overcome the technical problems of requiring additional hardware settings and failing to effectively guarantee the anomaly detection accuracy of target recognition images in smart cockpits, this disclosure provides an anomaly detection method, model training method, device, and vehicle that improve the anomaly detection accuracy of target recognition images in smart cockpits and reduce the false alarm rate without requiring additional hardware settings, thereby enhancing user safety.

[0004] According to a first aspect of the present disclosure, an anomaly detection method is provided, comprising: acquiring a target recognition image of a vehicle cabin; performing anomaly detection on the target recognition image using a pre-trained detection model to obtain anomaly detection information, wherein the anomaly detection information is used to characterize the cause of anomaly in the target recognition image; wherein the detection model is trained using training data, the training data including: image samples and labels of the image samples, the labels of the image samples including a main task label and an auxiliary task label, the main task label being used to characterize the cause of anomaly in the image sample, the auxiliary task label being used to characterize whether the image sample contains target image information, and the target image information being used to assist in detecting the cause of anomaly.

[0005] This technical solution improves the anomaly detection accuracy of smart cockpit images from the algorithm side without requiring additional hardware setup. Specifically, based on the target recognition images of the smart cockpit, a pre-trained detection model is used to detect anomalies and identify the causes of these anomalies. The training data for the pre-trained detection model includes two types of task labels: a primary task label and an auxiliary task label. The auxiliary task label assists in detecting the causes of anomalies, resulting in higher anomaly detection accuracy for the pre-trained model. This allows for precise localization of the causes of anomalies in the smart cockpit target recognition images, reducing the false alarm rate. Therefore, this technical solution can improve the anomaly detection accuracy of smart cockpit target recognition images and reduce the false alarm rate without requiring additional hardware setup, thereby enhancing user safety.

[0006] In some possible implementations, the pre-trained detection model includes: a feature extraction network and a pre-trained main task classification head. The step of using the pre-trained detection model to perform anomaly detection based on the target recognition image and obtain anomaly detection information includes: extracting target recognition image features from the target recognition image using the feature extraction network; and performing anomaly detection based on the target recognition image features using the pre-trained main task classification head to obtain anomaly detection information.

[0007] Since the auxiliary task classification head only plays a role during training, during model inference, only the prediction results of the main task classification head need to be extracted to obtain anomaly detection information.

[0008] In some possible implementations, acquiring the target recognition image of the vehicle cabin includes: acquiring the target recognition image of the vehicle cabin captured by a camera mounted on the A-pillar of the vehicle; and / or, acquiring the target recognition image of the vehicle cabin captured by a camera mounted on the steering wheel of the vehicle.

[0009] By installing cameras on the A-pillar and / or steering wheel, it is possible to acquire target recognition images of the driver. By checking whether the target recognition images are abnormal, it can be determined whether the driver's condition is abnormal, thereby improving driving safety.

[0010] In some possible implementations, the abnormal causes of the target recognition image include: camera obstruction, no one in the driving position, face obstruction, and abnormal face posture, wherein the target image information is face image information.

[0011] Since images from occluded scenes and overexposed scenes have similar effects—for example, the image appears washed out and does not include faces, thus constituting an image anomaly—while overexposed images still contain faces and are not considered anomalies, using the presence or absence of facial image information as an auxiliary task label can help the model better distinguish between these two scenarios, thereby avoiding misjudgments of anomalies in overexposed scenes.

[0012] In some possible implementations, the anomaly detection method further includes: in response to the anomaly detection information indicating that the cause of the anomaly in the target recognition image is camera occlusion, controlling the target device of the vehicle to issue an alarm regarding the camera occlusion.

[0013] When the camera is obstructed in the target recognition image, timely alarm control of relevant equipment can resolve camera malfunctions promptly and improve security.

[0014] Secondly, embodiments of this disclosure provide a model training method, comprising: acquiring training data, the training data including: image samples and labels of the image samples, the labels of the image samples including a main task label and an auxiliary task label, the main task label being used to characterize the cause of anomalies in the image samples, the auxiliary task label being used to characterize whether the image samples contain target image information, the target image information being used to assist in detecting the cause of anomalies; and training a detection model to be trained multiple times based on the training data to obtain a pre-trained detection model, the pre-trained detection model being used to detect the cause of anomalies in recognition images of vehicle cabins.

[0015] Because the training data has two types of task labels, primary task label and secondary task label, the secondary task label can help detect the cause of anomalies. The detection model is trained based on this training data, which makes the anomaly detection accuracy of the pre-trained detection model higher. In this way, the accurate localization of the cause of anomalies in the target recognition image of the intelligent cockpit can be achieved, reducing the false alarm rate.

[0016] In some possible implementations, training the detection model to be trained based on the training data to obtain a pre-trained detection model includes: training the detection model to be trained multiple times based on the training data to obtain the pre-trained detection model; wherein each training process of the detection model to be trained includes: using the detection model to be trained to make predictions based on the image samples to obtain prediction information, the prediction information including: main task prediction information and auxiliary task prediction information, the main task prediction information being used to characterize the cause of the anomaly of the image sample, and the auxiliary task prediction information being used to characterize whether the image sample contains target image information; and training the detection model to be trained based on the prediction information and the labels of the image samples.

[0017] By training the detection model multiple times, the model can fully learn the various causes of image anomalies and distinguish between these anomalies and non-anomalies, thereby improving the model's anomaly detection accuracy.

[0018] In some possible implementations, training the detection model to be trained based on the prediction information and the labels of the image samples includes: determining a main task loss based on the difference between the main task label and the main task prediction information; determining an auxiliary task loss based on the difference between the auxiliary task label and the auxiliary task prediction information; determining the loss of the detection model to be trained based on the main task loss and the auxiliary task loss; and training the detection model to be trained based on the loss of the detection model to be trained.

[0019] Based on the main task loss and auxiliary task loss, the loss of the detection model to be trained is determined, and then the detection model to be trained is trained based on the model loss to improve the anomaly detection accuracy of the model.

[0020] In some possible implementations, determining the loss of the detection model to be trained based on the main task loss and the auxiliary task loss includes: determining the proportion of auxiliary task loss; and determining the loss of the detection model to be trained based on the main task loss, the auxiliary task loss, and the proportion of auxiliary task loss.

[0021] By configuring the proportion of loss for auxiliary tasks, the training of the detection model to be trained can be effectively balanced between the main task and auxiliary tasks, achieving a training effect where the main task is primary and the auxiliary task is secondary.

[0022] In some possible implementations, determining the proportion of auxiliary task loss includes: determining the proportion of auxiliary task loss based on the proportion of the current training round to the total number of training rounds.

[0023] Based on the proportion of training rounds relative to the total number of training rounds, the proportion of auxiliary task loss is determined, so that the proportion of auxiliary task loss changes dynamically as the number of training rounds of the model increases, thereby improving the robustness of model training.

[0024] In some possible implementations, the detection model to be trained includes: a feature extraction network, a main task classification head to be trained, and an auxiliary task classification head. The step of using the detection model to be trained to make predictions based on the image samples to obtain prediction information includes: extracting image sample features from the image samples using the feature extraction network; making predictions based on the image sample features using the main task classification head to be trained to obtain the main task prediction information; and making predictions based on the image sample features using the auxiliary task classification head to obtain the auxiliary task prediction information.

[0025] Image sample features are extracted using a feature extraction network, and then two task classification heads are used to predict different types of information. The introduction of an auxiliary task allows the model to focus on target image information within the image, thereby reducing the model's false positive rate.

[0026] According to a third aspect of the present disclosure, an anomaly detection apparatus is provided, the anomaly detection apparatus being configured to perform the anomaly detection method described in the first aspect of the present disclosure.

[0027] According to a fourth aspect of the present disclosure, a model training apparatus is provided, the model training apparatus being configured to perform the model training method described in the second aspect of the present disclosure.

[0028] According to a fifth aspect of the present disclosure, a vehicle is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement the anomaly detection method described in the first aspect of the present disclosure.

[0029] According to a sixth aspect of the present disclosure, a computer-readable storage medium is provided, having stored thereon computer program instructions that, when executed by a processor, implement the anomaly detection method of the first aspect of the present disclosure or the model training method of the second aspect of the present disclosure.

[0030] According to a seventh aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the anomaly detection method described in the first aspect of the present disclosure or the model training method described in the second aspect of the present disclosure.

[0031] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0032] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0033] Figure 1 This is a flowchart illustrating an anomaly detection method according to an exemplary embodiment.

[0034] Figure 2 This is an image showing a camera obstruction situation according to an exemplary embodiment.

[0035] Figure 3 This is an image of an overexposed scene according to an exemplary embodiment.

[0036] Figure 4 This is an example diagram illustrating a model training framework according to an exemplary embodiment.

[0037] Figure 5 This is a block diagram illustrating an anomaly detection device according to an exemplary embodiment.

[0038] Figure 6 This is a block diagram illustrating a vehicle according to an exemplary embodiment. Figure 7 This is a block diagram illustrating a model training apparatus according to an exemplary embodiment. Figure 8 This is a block diagram illustrating a vehicle according to an exemplary embodiment. Detailed Implementation

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0040] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0041] As mentioned in the background section, in the monitoring scenario of smart cockpits, it is possible to detect whether the images of the smart cockpit are abnormal in order to improve user safety.

[0042] In related technologies, to ensure the accuracy of anomaly detection in smart cockpit recognition images, a hardware-based method is used to adaptively adjust the camera's fill light to obtain optimal imaging, thereby improving the quality of the smart cockpit's recognition images and thus enhancing anomaly detection accuracy. However, this approach requires additional hardware configuration; furthermore, improving image quality does not necessarily guarantee accurate anomaly detection.

[0043] Therefore, the relevant technologies have technical problems such as requiring additional hardware settings and not being able to effectively guarantee the accuracy of anomaly detection in the intelligent cockpit's recognized images.

[0044] Based on this, the present disclosure provides a technical solution that improves the anomaly detection accuracy of intelligent cockpit detection images from the algorithm side without requiring additional hardware settings. Specifically, based on the target recognition images of the intelligent cockpit, a pre-trained detection model is used for anomaly detection to identify the causes of anomalies in the target recognition images. The training data of the pre-trained detection model includes two types of task labels: a primary task label and an auxiliary task label. The auxiliary task label assists in detecting the causes of anomalies, resulting in higher anomaly detection accuracy for the pre-trained detection model. This, in turn, enables precise localization of the causes of anomalies in intelligent cockpit target recognition images, reducing the false alarm rate.

[0045] Therefore, this technical solution can improve the anomaly detection accuracy of target recognition images in smart cockpits and reduce the false alarm rate without the need for additional hardware settings, thereby improving user safety.

[0046] Figure 1 This is a flowchart illustrating an anomaly detection method according to an exemplary embodiment. This anomaly detection method can be applied to a vehicle's intelligent cockpit monitoring system, such as... Figure 1As shown, the anomaly detection method includes: Step S11: Obtain target recognition image of the vehicle cabin.

[0047] Step S12: Anomaly detection is performed on the target recognition image using a pre-trained detection model to obtain anomaly detection information, which is used to characterize the cause of anomalies in the target recognition image.

[0048] In step S11, the target recognition image of the vehicle cabin can be an image collected from a monitored object inside the vehicle cabin. The monitored object could be, for example, the driver in the driver's seat or a passenger in a non-driver's seat.

[0049] In some embodiments, the target recognition image can be acquired using a camera positioned at a corresponding location on the vehicle.

[0050] As an optional implementation, step S11 includes: acquiring a target recognition image of the vehicle cabin captured by a camera mounted on the A-pillar of the vehicle; and / or, acquiring a target recognition image of the vehicle cabin captured by a camera mounted on the steering wheel of the vehicle.

[0051] In this implementation, by using cameras mounted on the A-pillar and / or steering wheel, target recognition images of the driver can be acquired. By checking whether the target recognition images are abnormal, the driver's state can be characterized as abnormal, thereby improving driving safety.

[0052] The A-pillar, a component of the vehicle's external structure, refers to the pillar between the windshield and the left and right front doors. It is a key part of the vehicle's main structural support, sharing the responsibility of supporting the vehicle body's strength with the B and C pillars. By installing a camera on the A-pillar, it's possible to capture target images for driver recognition from the side.

[0053] Installing a camera on the vehicle's steering wheel allows for the capture of a target recognition image of the driver from the front.

[0054] It is understandable that, depending on the actual monitoring needs in different application scenarios, cameras can be placed in one or more locations; there is no limitation here. For example, if it is necessary to monitor passengers in the back of a vehicle, a camera installed on the back of the front seat can be used to achieve target recognition and image acquisition.

[0055] In this embodiment of the disclosure, the training data of the pre-trained detection model includes: image samples and labels of the image samples. The labels of the image samples include main task labels and auxiliary task labels. The main task labels are used to characterize the cause of the anomaly in the image samples, and the auxiliary task labels are used to characterize whether the image samples contain target image information. The target image information is used to assist in detecting the cause of the anomaly.

[0056] In some embodiments, the target recognition image can be directly input into a pre-trained detection model, which can output anomaly detection information that characterizes the cause of the anomaly.

[0057] In some embodiments, the target recognition image may be preprocessed before being input into a pre-trained detection model. Image preprocessing methods, such as image enhancement and scaling, are not limited here.

[0058] In this embodiment of the disclosure, the abnormal reasons for the target recognition image may include: camera obstruction, no one in the driving position, face obstruction, and abnormal face posture; correspondingly, the target image information may be face image information.

[0059] Since images from occluded scenes and overexposed scenes have similar effects—for example, the image appears washed out and does not include faces, thus constituting an image anomaly—while overexposed images still contain faces and are not considered anomalies, using the presence or absence of facial image information as an auxiliary task label can help the model better distinguish between these two scenarios, thereby avoiding misjudgments of anomalies in overexposed scenes.

[0060] Images with obstructed camera views have a similar effect to images in overexposed scenes. Figure 2 This is an image illustrating a camera obstruction situation according to an exemplary embodiment, such as... Figure 2 As shown, when the camera is obstructed, some areas of the target recognition image do not include the corresponding image information, and the image in these areas appears washed out.

[0061] Figure 3 This is an image of an overexposed scene illustrated according to an exemplary embodiment, such as... Figure 3 As shown, in overexposed scenes, the target recognition image includes a human face, and the overall image is also too white.

[0062] Therefore, target recognition images in overexposed scenes are prone to false alarms. The model training method of this disclosure can avoid such false alarms.

[0063] Regarding abnormal facial poses, such as: excessively large facial poses, excessively small facial poses, and distorted facial poses.

[0064] This implementation method is suitable for scenarios where the monitored object is a person, and the image effect of one or more abnormal causes is similar to the image effect in an overexposed scene.

[0065] It is understandable that if the monitored object is not a person, but another object, such as a pet or a static object, then the target image information can be pet image information, static object information, etc., without any limitation.

[0066] Furthermore, if the image effect of the cause of the anomaly is not distinguished from the image effect in a non-abnormal scene by whether or not it includes information about the monitored object, then the target image information can be other information that can help distinguish between abnormal and non-abnormal images, and is not limited here.

[0067] In some embodiments, the labels of image samples may be set manually or configured through artificial intelligence technology, without limitation.

[0068] In some embodiments, the main task labels may include five types. These five main task labels can respectively represent the four abnormal causes mentioned above and normal driving (i.e., no abnormalities in the target recognition image). As an example, label 0 represents normal driving; label 1 represents camera obstruction; label 2 represents no one in the driving position; label 3 represents face obstruction; and label 4 represents abnormal face pose.

[0069] Regarding auxiliary task labels, two types can be set: one that includes target image information and one that does not. For example, label 1 includes target image information; label 0 does not include target image information.

[0070] Furthermore, based on image samples and their labels, the model can be trained to obtain a pre-trained detection model. The specific training method for the model will be described in subsequent embodiments.

[0071] In this embodiment of the disclosure, the pre-trained detection model may include: a feature extraction network and a pre-trained main task classification head. The anomaly detection information obtained by performing anomaly detection based on the target recognition image through the pre-trained detection model may include: extracting target recognition image features from the target recognition image through the feature extraction network; and performing anomaly detection based on the target recognition image features through the pre-trained main task classification head to obtain anomaly detection information.

[0072] In this implementation, the auxiliary task classification head only plays a role during the training process. Therefore, during model inference, only the prediction results of the main task classification head need to be extracted to obtain the anomaly detection information.

[0073] The technical solution of this disclosure can optimize the recognition accuracy of abnormal scenarios such as camera obstruction in overexposed cabin scenes at the algorithm level. Furthermore, without requiring additional hardware or data, it reduces the probability of false alarms for camera obstruction during normal driving in overexposed scenarios through potentially related auxiliary tasks.

[0074] Furthermore, after receiving anomaly detection information, corresponding feedback can be executed.

[0075] Therefore, as an optional implementation, the anomaly detection method further includes: in response to the anomaly detection information indicating that the cause of the anomaly in the target recognition image is camera occlusion, controlling the target device of the vehicle to issue an alarm regarding the camera occlusion.

[0076] In this implementation, when the camera obstructs the target recognition image, timely alarms can be triggered by controlling relevant devices to resolve camera malfunctions promptly and improve security.

[0077] In some embodiments, the target device may be: a speaker, a warning light, a vehicle display screen, a steering wheel, a seat, etc.

[0078] As an example, the speaker can be controlled to output an alarm voice message, the content of which is used to indicate that the camera is obstructed.

[0079] As an example, a warning light can be controlled to flash to alert the driver to a dangerous situation inside the vehicle, allowing the driver to promptly check for any abnormalities in the vehicle's cameras.

[0080] As an example, the vehicle's infotainment system can be controlled to display an alarm message indicating that the camera is obstructed.

[0081] As an example, steering wheel vibration can be controlled to alert the driver to a potential danger inside the vehicle, allowing the driver to promptly check for any abnormalities in the in-vehicle cameras.

[0082] As an example, the seat angle can be adjusted to indicate a dangerous situation inside the vehicle, allowing the driver to promptly check for any abnormalities in the in-vehicle cameras.

[0083] It is understood that the alarm methods described above are merely examples. Different alarm methods can be configured in different application scenarios, and no limitation is made here.

[0084] Besides camera obstruction, other abnormal causes can also trigger timely alarms / notifications using appropriate methods. Since the severity of these other abnormal causes is lower than that of camera obstruction, the corresponding alarm / notification intensity can be lower.

[0085] Figure 4 This is a flowchart illustrating a model training method according to an exemplary embodiment, such as... Figure 4 As shown, the model training method includes the following steps: Step S41: Obtain training data. The training data includes: image samples and their labels. The image sample labels include a main task label and an auxiliary task label. The main task label is used to characterize the cause of anomalies in the image samples, and the auxiliary task label is used to characterize whether the image samples contain target image information. The target image information is used to assist in detecting the cause of anomalies. Step S42: Based on the training data, train the detection model to be trained to obtain a pre-trained detection model. The pre-trained detection model is used to detect the causes of anomalies in the recognition images of the vehicle cabin.

[0086] It is understood that the detection model obtained through this training method can be used to implement the anomaly detection scheme in the aforementioned embodiments.

[0087] As an optional implementation method, the model training process includes: acquiring training data; and training the detection model to be trained multiple times based on the training data to obtain a pre-trained detection model.

[0088] The training process of the detection model to be trained includes: making predictions based on image samples using the detection model to be trained, and obtaining prediction information, which includes: main task prediction information and auxiliary task prediction information. The main task prediction information is used to characterize the cause of the anomaly in the image sample, and the auxiliary task prediction information is used to characterize whether the image sample contains target image information; and training the detection model to be trained based on the prediction information and the labels of the image samples.

[0089] In this implementation, by training the detection model to be trained multiple times, the model can fully learn the various causes of image anomalies and distinguish between the various causes of image anomalies and non-abnormal causes, thereby improving the model's anomaly detection accuracy.

[0090] In some embodiments, the image samples used in each training process of the model can be different. For example, the first training uses a subset of image samples, the second training uses the image samples selected in the first training plus new image samples, and so on, so that the image samples used in each training are different, thereby improving the generalization ability of the model.

[0091] In some embodiments, image samples can be input into a detection model to be trained. The detection model to be trained can output prediction information. Based on the difference between the output prediction information and the true label, the model can be continuously adjusted to achieve the training effect of the model.

[0092] Therefore, as an optional implementation, the detection model to be trained is trained based on the prediction information and the labels of the image samples, including: determining the main task loss based on the difference between the main task label and the main task prediction information; determining the auxiliary task loss based on the difference between the auxiliary task label and the auxiliary task prediction information; determining the loss of the detection model to be trained based on the main task loss and the auxiliary task loss; and training the detection model to be trained based on the loss of the detection model to be trained.

[0093] In this implementation, the loss of the detection model to be trained is determined based on the loss of the main task and the loss of the auxiliary task. Then, the detection model to be trained is trained based on the loss of the model to improve the anomaly detection accuracy of the model.

[0094] In some embodiments, the auxiliary task loss may account for a certain proportion of the model loss.

[0095] Therefore, as an optional implementation, the loss of the detection model to be trained is determined based on the main task loss and the auxiliary task loss, including: determining the proportion of auxiliary task loss; and determining the loss of the detection model to be trained based on the main task loss, the auxiliary task loss, and the proportion of auxiliary task loss.

[0096] In this implementation, by configuring the proportion of auxiliary task loss, the training of the detection model to be trained can be effectively balanced between the main task and the auxiliary task, achieving the training effect of the main task as the primary task and the auxiliary task as the secondary task.

[0097] In some embodiments, the percentage of loss due to auxiliary tasks can be a fixed preset percentage.

[0098] In some embodiments, the percentage of loss due to auxiliary tasks can be dynamically adjustable.

[0099] Therefore, as an optional implementation method, determining the proportion of auxiliary task loss includes: determining the proportion of auxiliary task loss based on the proportion of the current training rounds to the total number of training rounds.

[0100] In this implementation, the proportion of auxiliary task loss is determined based on the ratio of training rounds to total training rounds, so that the proportion of auxiliary task loss changes dynamically as the number of training rounds of the model increases, thereby improving the robustness of model training.

[0101] In some embodiments, the higher the proportion of the current training epoch relative to the total training epochs, the smaller the auxiliary task loss. That is, as the number of training epochs increases, the proportion of auxiliary task loss gradually decreases.

[0102] As an example, the auxiliary task loss can be expressed as:

[0103] Where t represents the current training round, and T represents the total training rounds. This indicates the proportion of auxiliary task loss in the current training round. Also, 0.25 represents the initial value of the auxiliary task loss; this initial value can vary in different application scenarios and is not limited here.

[0104] In some embodiments, the detection model to be trained includes: a feature extraction network, a main task classification head to be trained, and an auxiliary task classification head. Then, the detection model to be trained makes predictions based on image samples to obtain prediction information, including: extracting image sample features from the image samples using the feature extraction network; making predictions based on the image sample features using the main task classification head to be trained to obtain main task prediction information; and making predictions based on the image sample features using the auxiliary task classification head to obtain auxiliary task prediction information.

[0105] In this implementation, image sample features are extracted using a feature extraction network, and then two task classification heads are used to predict different types of information. The introduction of an auxiliary task allows the model to focus on target image information within the image, thereby reducing the model's false positive rate.

[0106] The feature extraction network can be a backbone network, such as MobileNetV3, or it can be a convolutional neural network. This feature extraction network can be a pre-trained network or a network to be trained; there are no specific limitations here.

[0107] A classification head is a component located at the end of a model used to perform classification tasks. A classification head can consist of one or more fully connected layers, and its function is to map the extracted features to a specific classification label space.

[0108] In this embodiment of the disclosure, two types of classification heads can be set in the model: one for the main task classification, i.e., the classification of abnormal causes; and the other for the classification of whether or not target image information is included.

[0109] Figure 5 This is an example diagram illustrating a model training framework according to an exemplary embodiment. Figure 5 In the process, the target image information is facial information. The classification labels of the main task classification head include: camera obstruction, no driver in the driver's seat, face obstruction, excessive facial posture, and normal driving. The classification labels of the auxiliary task classification head include: no face and face.

[0110] like Figure 5As shown, image samples are first input into the backbone network of the model, where feature extraction is performed. The image features extracted by the backbone network can be input into the main task classification head and the auxiliary task classification head, respectively. The main task classification head can output prediction information representing anomalies, while the auxiliary task classification head can output prediction information representing whether a face is present. Furthermore, based on the difference between the two predictions and the true labels, the model parameters can be adjusted through backpropagation and other methods to achieve the desired model training effect.

[0111] As an example, each image sample x i After being input into the detection model to be trained, the main features z are first extracted through a feature extraction network. i : ,in, This represents a feature extraction network.

[0112] After extracting the main features z i Then, input it into the first category header (Main Task / Exception Category Header). The predicted category of the main task is obtained. : .

[0113] Meanwhile, the main feature z i It will also be entered into the second category header (auxiliary task category header). To obtain the predicted category of the auxiliary task : .

[0114] Based on the prediction results of the main task and the auxiliary task, as well as the label corresponding to each image sample, the total loss can be calculated. : ,in, This indicates the loss of the main task. This indicates the loss of auxiliary tasks. This indicates the percentage of losses incurred by auxiliary tasks.

[0115] In some embodiments, and It could be the cross-entropy loss between the predicted information and the true label.

[0116] In this embodiment of the disclosure, the auxiliary task only plays a role in assisting training during the training process, and it may not be used during the inference process.

[0117] Figure 6 This is a block diagram illustrating an anomaly detection device 600 according to an exemplary embodiment. (Refer to...) Figure 6 The device includes: The image acquisition module 501 is configured to acquire target recognition images of the vehicle cabin.

[0118] The detection module 502 is configured to: perform anomaly detection based on the target recognition image using a pre-trained detection model to obtain anomaly detection information, wherein the anomaly detection information is used to characterize the cause of anomalies in the target recognition image; wherein the training data of the pre-trained detection model includes: image samples and labels of the image samples, wherein the labels of the image samples include a main task label and an auxiliary task label, wherein the main task label is used to characterize the cause of anomalies in the image samples, and the auxiliary task label is used to characterize whether the image samples contain target image information, wherein the target image information is used to assist in detecting the cause of anomalies.

[0119] Optionally, the pre-trained detection model includes a feature extraction network and a pre-trained main task classification head. The detection module 602 is further configured to: extract target recognition image features from the target recognition image through the feature extraction network; and perform anomaly detection based on the target recognition image features through the pre-trained main task classification head to obtain anomaly detection information.

[0120] Optionally, the image acquisition module 601 is further configured to: acquire a target recognition image of the vehicle cabin captured by a camera mounted on the A-pillar of the vehicle; and / or acquire a target recognition image of the vehicle cabin captured by a camera mounted on the steering wheel of the vehicle.

[0121] Optionally, the device further includes a control module configured to: in response to the anomaly detection information indicating that the cause of the anomaly in the target recognition image is camera occlusion, control the target device of the vehicle to issue an alarm regarding the camera occlusion.

[0122] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0123] Figure 7 This is a block diagram illustrating a model training apparatus 700 according to an exemplary embodiment, such as... Figure 7 As shown, the model training device 700 includes: The data acquisition module 701 is configured to acquire training data, which includes image samples and labels of the image samples. The labels of the image samples include a main task label and an auxiliary task label. The main task label is used to characterize the cause of the anomaly in the image sample, and the auxiliary task label is used to characterize whether the image sample contains target image information. The target image information is used to assist in detecting the cause of the anomaly.

[0124] The training module 702 is configured to: train the detection model to be trained based on the training data to obtain a pre-trained detection model, wherein the pre-trained detection model is used to detect the causes of anomalies in the recognition images of the vehicle cabin.

[0125] Optionally, the training module 702 is further configured to: train the detection model to be trained multiple times based on the training data to obtain a pre-trained detection model; wherein each training process of the detection model to be trained includes: using the detection model to be trained to make predictions based on the image samples to obtain prediction information, the prediction information including: main task prediction information and auxiliary task prediction information, the main task prediction information being used to characterize the cause of the anomaly of the image sample, and the auxiliary task prediction information being used to characterize whether the image sample contains target image information; and training the detection model to be trained based on the prediction information and the labels of the image samples.

[0126] Optionally, the training module 702 is further configured to: determine the main task loss based on the difference between the main task label and the main task prediction information; determine the auxiliary task loss based on the difference between the auxiliary task label and the auxiliary task prediction information; determine the loss of the detection model to be trained based on the main task loss and the auxiliary task loss; and train the detection model to be trained based on the loss of the detection model to be trained.

[0127] Optionally, the training module 702 is further configured to: determine the proportion of auxiliary task loss; and determine the loss of the detection model to be trained based on the main task loss, the auxiliary task loss, and the proportion of auxiliary task loss.

[0128] Optionally, the training module 702 is further configured to determine the auxiliary task loss ratio based on the ratio of the current training round to the total training rounds.

[0129] Optionally, the training module 702 is further configured to: extract image sample features from the image samples through the feature extraction network; make predictions based on the image sample features through the main task classification head to be trained to obtain the main task prediction information; and make predictions based on the image sample features through the auxiliary task classification head to obtain the auxiliary task prediction information.

[0130] This disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the steps of the anomaly detection method or model training method provided in this disclosure.

[0131] Figure 8This is a block diagram illustrating a vehicle 800 according to an exemplary embodiment. For example, vehicle 800 may be a hybrid vehicle, a non-hybrid vehicle, an electric vehicle, a fuel cell vehicle, or other types of vehicle. Vehicle 800 may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle.

[0132] Reference Figure 8 The vehicle 800 may include various subsystems, such as an infotainment system 810, a perception system 820, a decision control system 830, a drive system 840, and a computing platform 850. The vehicle 800 may also include more or fewer subsystems, and each subsystem may include multiple components. Furthermore, each subsystem and each component of the vehicle 800 can be interconnected via wired or wireless means.

[0133] In some embodiments, the infotainment system 810 may include a communication system, an entertainment system, and a navigation system, etc.

[0134] The perception system 820 may include several sensors for sensing information about the environment surrounding the vehicle 800. For example, the perception system 820 may include a global positioning system (which may be GPS, BeiDou, or other positioning systems), an inertial measurement unit (IMU), lidar, millimeter-wave radar, ultrasonic radar, and a camera device.

[0135] The decision control system 830 may include a computing system, a vehicle controller, a steering system, a throttle, and a braking system.

[0136] The drive system 840 may include components that provide powered motion to the vehicle 800. In one embodiment, the drive system 840 may include an engine, an energy source, a transmission system, and wheels. The engine may be one or a combination of internal combustion engines, electric motors, and compressed air engines. The engine is capable of converting energy provided by the energy source into mechanical energy.

[0137] Some or all of the functions of the vehicle 800 are controlled by a computing platform 850. The computing platform 850 may include at least one processor 851 and a memory 852, the processor 851 being able to execute instructions 853 stored in the memory 852.

[0138] The processor 851 can be any conventional processor, such as a commercially available CPU. The processor may also include graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems on chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0139] The memory 852 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0140] In addition to instruction set 853, memory 852 can also store data, such as road maps, route information, vehicle position, direction, speed, and other data. The data stored in memory 852 can be used by computing platform 850.

[0141] In this embodiment of the disclosure, the processor 851 may execute instruction 853 to complete all or part of the steps of the above-described anomaly detection method.

[0142] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for performing the above-described anomaly detection method or model training method when executed by the programmable device.

[0143] Furthermore, the term “exemplary” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “exemplary” is not necessarily to be construed as advantageous compared to other aspects or designs. Rather, the use of the term “exemplary” is intended to present the concept in a concrete manner. As used herein, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless otherwise specified or clear from the context, “X applies A or B” is intended to mean any of the natural inclusive arrangements. That is, “X applies A or B” satisfies any of the foregoing instances if X applies A; X applies B; or both X applies A and B. Additionally, unless otherwise specified or clear from the context to refer to the singular form, the articles “a” and “an” as used in this application and the appended claims are generally understood to mean “one or more.”

[0144] Similarly, although this disclosure has been shown and described with respect to one or more implementations, equivalent variations and modifications will occur to those skilled in the art upon reading and understanding this specification and the accompanying drawings. This disclosure includes all such modifications and variations and is limited only by the scope of the claims. In particular, with respect to the various functions performed by the components described above (e.g., elements, resources, etc.), unless otherwise indicated, the terminology used to describe such components is intended to correspond to any component (functionally equivalent) that performs the specific function of the described component, even if structurally not equivalent to the disclosed structure. Furthermore, although specific features of this disclosure may have been disclosed with respect to only one of several implementations, such features may be combined with one or more other features of other implementations, as may be desired and advantageous to any given or particular application. Moreover, with regard to the terms “comprising,” “owning,” “having,” “having,” or variations thereof as used in the detailed description or claims, such terms are intended to be inclusive in a manner similar to the term “including.”

[0145] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0146] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

[0147] It should be understood that, unless otherwise specifically indicated, features of various embodiments of this disclosure described herein can be combined with each other. As used herein, the term “and / or” includes any one of the relevant listed items and any combination of any two or more; similarly, “at least one of…” includes any one of the relevant listed items and any combination of any two or more.

[0148] Although terms such as “first,” “second,” and “third” may be used herein to describe various components, parts, regions, layers, or sections, these components, parts, regions, layers, or sections are not limited to these terms. Rather, these terms are used only to distinguish one component, part, region, layer, or section from another. Therefore, without departing from the teachings of the examples described herein, the first component, part, region, layer, or section mentioned in the examples may also be referred to as the second component, part, region, layer, or section. Furthermore, the terms “first” and “second” are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as “first” or “second” may explicitly or implicitly include at least one of that feature. In the description herein, “a plurality” means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. An anomaly detection method, characterized in that, include: Acquire target recognition images of the vehicle cabin; Anomaly detection is performed on the target recognition image using a pre-trained detection model to obtain anomaly detection information, which is used to characterize the cause of the anomaly in the target recognition image. The detection model is trained using training data, which includes image samples and labels for the image samples. The labels for the image samples include a main task label and an auxiliary task label. The main task label is used to characterize the cause of the anomaly in the image sample, and the auxiliary task label is used to characterize whether the image sample contains target image information. The target image information is used to assist in detecting the cause of the anomaly.

2. The anomaly detection method according to claim 1, characterized in that, The detection model includes a feature extraction network and a pre-trained main task classification head. The pre-trained detection model performs anomaly detection based on the target recognition image to obtain anomaly detection information, including: The feature extraction network extracts recognition image features from the target recognition image. Anomaly detection information is obtained by using the pre-trained main task classification head and performing anomaly detection based on the features of the recognized image.

3. The anomaly detection method according to claim 1, characterized in that, The acquisition of the target recognition image of the vehicle cabin includes: Acquire target recognition images of the vehicle cabin captured by a camera mounted on the A-pillar of the vehicle; and / or, Acquire target recognition images of the vehicle cabin captured by a camera mounted on the vehicle's steering wheel.

4. The anomaly detection method according to any one of claims 1 to 3, characterized in that, The abnormal reasons for the target recognition image include: camera obstruction, no one in the driving position, face obstruction, and abnormal face posture. The target image information is face image information.

5. The anomaly detection method according to claim 4, characterized in that, The anomaly detection method further includes: In response to the anomaly detection information indicating that the cause of the anomaly in the target recognition image is camera obstruction, the vehicle's target device is controlled to issue an alarm regarding the camera obstruction.

6. A model training method, characterized in that, include: Acquire training data, which includes: image samples and labels of the image samples. The labels of the image samples include a main task label and an auxiliary task label. The main task label is used to characterize the cause of the anomaly in the image sample, and the auxiliary task label is used to characterize whether the image sample contains target image information. The target image information is used to assist in detecting the cause of the anomaly. Based on the training data, the detection model to be trained is trained to obtain a pre-trained detection model, which is used to detect the causes of anomalies in the recognition images of the vehicle cabin.

7. The model training method according to claim 6, characterized in that, The step of training the detection model to be trained based on the training data to obtain a pre-trained detection model includes: Based on the training data, the detection model to be trained is trained multiple times to obtain a pre-trained detection model; The training process of the detection model to be trained includes the following for each iteration: The detection model to be trained makes predictions based on the image samples to obtain prediction information, which includes: main task prediction information and auxiliary task prediction information. The main task prediction information is used to characterize the cause of the anomaly in the image samples, and the auxiliary task prediction information is used to characterize whether the image samples contain target image information. The detection model to be trained is trained based on the prediction information and the labels of the image samples.

8. The model training method according to claim 7, characterized in that, The step of training the detection model to be trained based on the prediction information and the labels of the image samples includes: The main task loss is determined based on the difference between the main task label and the main task prediction information; The auxiliary task loss is determined based on the difference between the auxiliary task label and the auxiliary task prediction information; The loss of the detection model to be trained is determined based on the main task loss and the auxiliary task loss. The detection model to be trained is trained based on the loss of the detection model to be trained.

9. The model training method according to claim 8, characterized in that, The step of determining the loss of the detection model to be trained based on the main task loss and the auxiliary task loss includes: Determine the percentage of losses from auxiliary tasks; The loss of the detection model to be trained is determined based on the main task loss, the auxiliary task loss, and the proportion of the auxiliary task loss.

10. The model training method according to claim 9, characterized in that, Determining the percentage of loss for auxiliary tasks includes: The percentage of loss in the auxiliary task is determined based on the percentage of this training round relative to the total number of training rounds.

11. The model training method according to claim 7, characterized in that, The detection model to be trained includes: a feature extraction network, a main task classification head to be trained, and an auxiliary task classification head. The step of using the detection model to be trained to make predictions based on the image samples to obtain prediction information includes: The feature extraction network extracts image sample features from the image samples. The main task prediction information is obtained by using the main task classification head to be trained and making predictions based on the features of the image samples. The auxiliary task prediction information is obtained by using the auxiliary task classification head to make predictions based on the features of the image samples.

12. An anomaly detection device, characterized in that, The anomaly detection device is configured to perform the anomaly detection method according to any one of claims 1 to 5.

13. A model training device, characterized in that, The model training apparatus is configured to perform the model training method according to any one of claims 6 to 11.

14. A vehicle, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to execute the executable instructions to implement the anomaly detection method according to any one of claims 1 to 5.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the anomaly detection method according to any one of claims 1 to 5, or the model training method according to any one of claims 6 to 11.

16. A computer program product, characterized in that, It includes a computer program that, when executed by a processor, implements the anomaly detection method according to any one of claims 1 to 5, or the model training method according to any one of claims 6 to 11.

Images