Abnormal distribution detection method based on automatic driving scene

Abstract

The invention discloses an abnormal distribution detection method based on an automatic driving scene, and relates to the technical field of machine learning and deep learning. Acquiring a to-be-detected image in an automatic driving scene; performing hierarchical feature extraction on the to-be-detected image to obtain features of the to-be-detected image at different hierarchies; projecting the features of the to-be-detected image at different levels to the orthogonal basis of the corresponding level to obtain the features of each level after dimension reduction processing; inputting each level of features after dimension reduction processing into a cross-layer score aggregator, and calculating the score of the to-be-detected image through the cross-layer score aggregator; and comparing the score of the to-be-detected image with a preset score threshold, and determining an abnormal distribution detection result of the to-be-detected image. According to the method, abnormal distribution detection can be accurately carried out on the image in the automatic driving scene.

Description

Technical Field

[0001] The present invention relates to the technical fields of machine learning and deep learning, and particularly relates to an outlier distribution detection method based on an autonomous driving scenario. Background Art

[0002] As an important development direction of the modern automotive industry, the core of autonomous driving technology is to enable vehicles to drive safely and effectively without human intervention. To achieve this goal, autonomous driving systems rely on complex perception, decision-making, and control modules, which need to process and interpret a large amount of sensor data, including visual and distance information provided by cameras, radars, lidars (Light Detection and Ranging), etc.

[0003] The perception module of an autonomous driving system is responsible for identifying and understanding the environment around the vehicle, including other vehicles, pedestrians, road signs, traffic signals, etc. This module usually relies on neural network models to process and classify sensor inputs. However, current neural network models are all default trained on a specific dataset, so when faced with out-of-distribution data samples of other categories outside the training dataset, the trained autonomous driving neural network model cannot accurately identify the types of data samples. The misclassification of such out-of-distribution data samples may lead to serious safety risks, especially in high-speed driving or complex traffic environments. For example, misidentifying a construction area as a normal road, or failing to timely identify an obstacle on the road, may lead to accidents.

[0004] Therefore, there is an urgent need in the prior art for a method that can accurately perform outlier distribution detection in an autonomous driving scenario. Summary of the Invention

[0005] Based on this, it is necessary to provide an outlier distribution detection method based on an autonomous driving scenario for the above technical problems, which can accurately perform outlier distribution detection on images in an autonomous driving scenario.

[0006] The present invention adopts the following technical solutions:

[0007] The present invention provides an outlier distribution detection method based on an autonomous driving scenario, including:

[0008] Obtain an image to be detected in an autonomous driving scenario;

[0009] Perform hierarchical feature extraction on the image to be detected to obtain features of the image to be detected at different levels;

[0010] Project the features of the image to be detected at different levels onto the corresponding level of orthogonal basis to obtain the features after dimensionality reduction processing at each level;

[0011] Input the features after dimensionality reduction at each level into a cross-layer score aggregator, and calculate the score of the image to be detected through the cross-layer score aggregator; the cross-layer score aggregator is a module that comprehensively processes features at different levels;

[0012] Compare the score of the image to be detected with a preset score threshold to determine the abnormal distribution detection result of the image to be detected.

[0013] Optionally, perform hierarchical feature extraction on the image to be detected to obtain features of the image to be detected at different levels, including:

[0014] Input the image to be detected into a pre-trained neural network model, and perform hierarchical feature extraction on the image to be detected through the neural network model to obtain initial features of the image to be detected at different levels; the initial features are multi-dimensional features;

[0015] Convert the initial features at different levels into one-dimensional vectors through a hierarchical feature extractor to obtain features of the image to be detected at different levels.

[0016] Optionally, the calculation method of the features after dimensionality reduction at each level is:

[0017]

[0018] where F' l represents the feature after dimensionality reduction at the l-th level, F l represents the feature at the l-th level, and P l represents the orthogonal basis at the l-th level.

[0019] Optionally, the construction method of the orthogonal basis at each level includes:

[0020] Obtain training samples within the distribution data from an autonomous driving system, an open-source autonomous driving scenario dataset, and a scenario target dataset;

[0021] Perform hierarchical feature extraction on the training samples to obtain features of the training samples at different levels;

[0022] Determine the projection matrix of the training samples at different levels according to the features of the training samples at different levels;

[0023] For the projection matrix at any level, perform eigenvalue decomposition on the projection matrix to obtain an orthogonal matrix, and determine the orthogonal basis of the training samples at the level according to the orthogonal matrix.

[0024] Optionally, the projection matrix is:

[0025]

[0026] where, Denote the projection matrix of the $l$-th layer as $F$ l (x i ) represents the feature of the $i$-th training sample $x$ i ; $\text{Var}(F$ l (x i )) = $F$ l (x i ) $\cdot$ $F$ l (x i ) represents the variance of the feature $F$ l (x i ); $\text{Cov}(F$ l (x i ), $F$ l (x v )) = $F$ l (x i ) $\cdot$ $F$ l (x v ) is the covariance of the features of two training samples $x$ i and $x$ v .

[0027] Optionally, determine the orthogonal basis of the training samples at the hierarchy according to the orthogonal matrix, including:

[0028] Determine the dimension $K$ of the hierarchical dimensionality reduction according to the features of the training samples at the hierarchy l ;

[0029] Determine the first column to the $(K$ l + 1)-th column in the orthogonal matrix as the orthogonal basis.

[0030] Optionally, the calculation method of the dimensionality reduction dimension $K$ l is:

[0031]

[0032] where $A$ l and $A$ L respectively represent the average values of the features of the training samples at the $l$-th layer and the $L$-th (last) layer, $N$ represents the number of training samples, $F$ l (x j ) and $F$ L (x j ) respectively represent the features of the training sample $x$ j at the $l$-th layer and the $L$-th layer, $M$ L represents the dimension of the $L$-th layer; $F$ l (x i ) and $F$ L (x i ) respectively represent the features of the training sample $x$ i at the $l$-th layer and the $L$-th layer.

[0033] Optionally, the score of the image to be detected is calculated by a cross-layer score aggregator, including:

[0034] For any level, normalize the features after dimensionality reduction of the level to obtain normalized features;

[0035] Calculate the Euclidean distance between the normalized features and the normalized features of the historical detected image; the historical detected image is the image for which abnormal distribution detection was performed before the image to be detected;

[0036] Obtain the p-th shortest distance in the Euclidean distance;

[0037] Determine the ratio of the sum of the p-th shortest distances in the Euclidean distance at all levels to the total number of all levels as the score of the image to be detected.

[0038] Optionally, compare the score of the image to be detected with a preset score threshold to determine the abnormal distribution detection result of the image to be detected, including:

[0039] If the score of the image to be detected is greater than the preset score threshold, determine that the abnormal distribution detection result of the image to be detected is that the image to be detected is data with an abnormal distribution;

[0040] If the score of the image to be detected is less than or equal to the preset score threshold, determine that the abnormal distribution detection result of the image to be detected is that the image to be detected is data with a normal distribution.

[0041] Optionally, the method further includes:

[0042] If the abnormal distribution detection result of the image to be detected is data with an abnormal distribution, upload the image to be detected to the server and give an alarm reminder on the display interface of the main control console of the autonomous driving vehicle.

[0043] The present invention provides an abnormal distribution detection device based on an autonomous driving scenario, including:

[0044] An acquisition module, configured to acquire an image to be detected in an autonomous driving scenario;

[0045] An extraction module, configured to perform hierarchical feature extraction on the image to be detected to obtain features of the image to be detected at different levels;

[0046] A projection module, configured to project the features of the image to be detected at different levels onto the orthogonal basis of the corresponding level to obtain features after dimensionality reduction processing at each level;

[0047] A calculation module, configured to input the features after dimensionality reduction processing at each level into a cross-layer score aggregator, and calculate the score of the image to be detected through the cross-layer score aggregator; the cross-layer score aggregator is a module for comprehensively processing features at different levels;

[0048] A comparison module for comparing the score of the image to be detected with a preset score threshold to determine the abnormal distribution detection result of the image to be detected.

[0049] The present invention provides a computer-readable storage medium storing a computer program, which when executed by a processor implements the above-mentioned abnormal distribution detection method based on an autonomous driving scenario.

[0050] The present invention provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, it implements the above-mentioned abnormal distribution detection method based on an autonomous driving scenario.

[0051] The above-mentioned at least one technical solution adopted by the present invention can achieve the following beneficial effects:

[0052] Perform hierarchical feature extraction on the image to be detected to obtain features at different levels, then project the features onto an orthogonal basis for dimensionality reduction processing, and then use a cross-layer score aggregator to calculate the score of the image to be detected. Finally, based on a preset score threshold, make an abnormal distribution detection decision to determine whether the image to be detected is an abnormal distribution sample.

[0053] The present invention extracts features of the image to be detected at different levels. Shallow features contain detailed information, and deep features have stronger semantic information. Such multi-level feature representation can more comprehensively describe the image to be detected. In this way, through shallow feature-driven enhancement, the detection ability for unknown images to be detected can be improved. At the same time, compared with traditional methods that rely on deep features, the present invention can better utilize the covariate shift of the image to enhance the recognition of abnormal distributions, making up for the insensitivity of out-of-abnormal-distribution detection to covariate shifts in domains such as style backgrounds, and enhancing the detection ability of the autonomous driving system for unknown situations, thereby improving the accuracy of abnormal distribution detection for images to be detected in an autonomous driving scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The schematic embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0055] Figure 1 It is a schematic flowchart of an abnormal distribution detection method based on an autonomous driving scenario provided by the present invention;

[0056] Figure 2 It is a schematic diagram of an alarm on the display screen of the main console of an autonomous driving vehicle provided by the present invention;

[0057] Figure 3Structural diagram of a voice alarm module provided by the present invention;

[0058] Figure 4 Flowchart of the network architecture training of an outlier distribution detection method based on an autonomous driving scenario;

[0059] Figure 5 Flowchart of the actual application of an outlier distribution detection method based on an autonomous driving scenario;

[0060] Figure 6 Schematic diagram of an outlier distribution detection device based on an autonomous driving scenario provided by the present invention;

[0061] Figure 7 Schematic diagram of a computer device for implementing an outlier distribution detection method based on an autonomous driving scenario provided by the present invention.

[0062] Description of reference numerals:

[0063] 1. Power supply input unit; 2. Relay output unit; 3. Relay status indicator light; 4. External microphone interface; 5. On-board microphone; 6. Speaker interface; 7. Voice upgrade port; 8. Power light; 9. Speaker. Detailed implementation manners

[0064] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the specific embodiments and corresponding drawings of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0065] The perception module in the prior art has the following challenges when facing out-of-distribution data (i.e., one or several types of data that did not participate in the training process):

[0066] (1) Limitations of the autonomous driving dataset: Due to the diversity and complexity of actual road conditions, it is almost impossible to construct a training dataset covering all possible situations. Therefore, there is a great possibility that the autonomous driving system will encounter rare or abnormal data and sample events not covered by the training data, and it is very likely that it will not be able to make a correct response.

[0067] (2) Limited model generalization ability: The real world is an open world, but current neural network models are defaulted to be trained on a specific closed-set dataset. This problem exists not only in the field of autonomous driving but also in other fields. Therefore, when faced with out-of-distribution data samples of other categories outside the training dataset, the trained autonomous driving neural network model cannot accurately identify them and may even make incorrect classifications overconfidently.

[0068] (3) Safety risks: The misclassification of such out-of-distribution data samples may lead to serious safety risks, especially in high-speed driving or complex traffic environments. For example, misidentifying a construction area as a normal road or failing to timely identify an obstacle on the road may result in accidents.

[0069] Some current studies are also dedicated to improving the detection ability of out-of-distribution data samples, but there are still deficiencies:

[0070] (1) Dependence on deep features: Many existing detection methods for out-of-distribution data mainly rely on features extracted from deep network layers. Although these features can capture the semantic changes in the data, the offsets between data not only include semantic offsets but also covariate offsets in the domain. Only focusing on the features of the deep network layer will lead to ignoring the impact of covariate offsets between data on the detection of abnormal distributions.

[0071] (2) Lack of real-time performance: Some out-of-distribution detection methods cannot meet the real-time requirements in terms of processing speed, which is insufficient for an autonomous driving system that requires quick response.

[0072] Based on this, there is an urgent need for a new method for detecting abnormal distributions based on autonomous driving scenarios. It should be noted that this method can be applied to the control module of autonomous vehicles to detect whether there are out-of-distribution data in the autonomous driving scenario through the control module.

[0073] The following will detail the technical solutions provided by each embodiment of the present invention in conjunction with the accompanying drawings.

[0074] Figure 1 It is a schematic flowchart of a method for detecting abnormal distributions based on autonomous driving scenarios in the present invention, specifically including the following steps:

[0075] S101, obtain the image to be detected in the autonomous driving scenario.

[0076] Among them, the process of obtaining the image to be detected in the autonomous driving scenario includes: obtaining the scene image collected by the camera device or the perception device in the autonomous driving scenario, segmenting the scene image, segmenting each object in the scene image to obtain a plurality of segmented images, each segmented image includes an object to be recognized, and each segmented image is used as an image to be detected.

[0077] S102. Perform hierarchical feature extraction on the image to be detected to obtain features of the image to be detected at different levels.

[0078] In one embodiment, performing hierarchical feature extraction on the image to be detected to obtain features of the image to be detected at different levels includes:

[0079] Input the image to be detected into the pre-trained neural network model, perform hierarchical feature extraction on the image to be detected through the neural network model to obtain initial features of the image to be detected at different levels; the initial features are multi-dimensional features; convert the initial features at different levels into one-dimensional vectors through the hierarchical feature extractor to obtain features of the image to be detected at different levels.

[0080] Among them, the initial features of the image to be detected obtained by the neural network model can be expressed as where l ∈ [1, L] represents the l-th layer of the neural network, H l 、W l and C l respectively represent the height, width and channels of the feature map.

[0081] The hierarchical feature extractor can be simply expressed as:

[0082] F l (X) = φ(f l (x)) (1)

[0083] Among them, φ(·) is an operation to expand the features along the horizontal axis, represents the features obtained from the hierarchical feature extractor, where M l = H l W l C l , and this hierarchical feature extractor converts the extracted multi-dimensional features into one-dimensional vectors.

[0084] The neural network model is a multi-layer network structure. The input of the first layer is the image to be detected, and the output of the first layer is a kind of tensor. The input of the second layer is the output of the first layer. Similarly, the output of the second layer is also a tensor vector. The input of the third layer is the output of the second layer, and so on. In this embodiment, the tensor vectors output from different layers can be selected as the features of the image to be detected at different levels required by the present invention. However, the features at different levels should include deep features and shallow features. For example, if the neural network model has a total of 20 layers, the features of the layers such as 1, 2, and 3 at the front are shallow, and the features of the last 18, 19, and 20 layers are deep. Of course, the intermediate layers can also be used. We can extract them in the form of 1, 4, 7, 11, etc. The extraction method can be changed according to the requirements of the task, or the features output from all layers can be directly used as the features of the image to be detected at different levels.

[0085] In this embodiment, the features of the image to be detected at different levels include deep features and shallow features. The receptive field of the deep network is larger, and its semantic information representation ability is strong; the receptive field of the shallow network is smaller, the geometric detail information is rich, but the semantic information is weak. There are many geometric details in the shallow features, that is, there are many details of the covariate shift in the domain of the style background. In the present invention, shallow features are combined with deep features, so as to increase the understanding of the details in this geometric background domain, and thus make up for the characteristics of insensitivity to covariates.

[0086] S103, project the features of the image to be detected at different levels onto the orthogonal basis of the corresponding level to obtain the features after dimensionality reduction processing at each level.

[0087] The features at different levels can be regarded as deep features and shallow features. There is a large amount of redundancy between the features at different levels. An adaptive dimensionality reduction strategy is used to perform dimensionality reduction processing on the features at different levels to reduce feature redundancy and improve computational efficiency; the adaptive dimensionality reduction strategy includes: calculating the subspace of the projection of the features of each layer of the image to be detected, and then projecting the sample features onto the orthogonal basis of the subspace, which ensures the orthogonality between the features, thereby reducing redundancy and improving the computational efficiency.

[0088] Optionally, the construction method of the orthogonal basis for each level includes: obtaining training samples within the distribution data from the autonomous driving system, the open-source autonomous driving scenario dataset, and the scenario target dataset; performing hierarchical feature extraction on the training samples to obtain the features of the training samples at different levels; determining the projection matrix of the training samples at different levels according to the features of the training samples at different levels; for the projection matrix of any level, performing eigenvalue decomposition on the projection matrix to obtain an orthogonal matrix, and determining the orthogonal basis of the training samples at the level according to the orthogonal matrix.

[0089] Collect image data from the autonomous driving system, label its target attribute tags, and establish a detailed database by combining the open-source autonomous driving scenario dataset and the scenario target dataset.

[0090] Use the symbols X and Y = {1, 2, …, c} (where c is the number of known in-distribution classes (the classes include animals, plants, vehicles, people, aircraft, electrical appliances, traffic lights, etc.)) to represent the input space and the label space. Use D to represent the real world. At the same time, the input that is in the same distribution as the samples used for pre-training the model is N is the total number of input samples D id and Y' represents the label of the out-of-distribution, and the symbol D ood represents the out-of-distribution samples with the label Y'; the samples D id within the distribution data are used as training samples.

[0091] Use the pre-trained neural network model to obtain the initial features of the input image x ∈ D id at different levels, and then use the hierarchical feature extractor to convert the initial features at different levels into one-dimensional vectors to obtain the features of the training samples at different levels. The specific implementation method is the same as the specific method for obtaining the features of the image to be detected at different levels in the above embodiment, and this embodiment will not be elaborated here.

[0092] Define the features of the training samples at different levels as F l =(F l (x 1 ), F l (x 2 ), …, F l (x i ), …, F l (X N )] T , where X i ∈ D id and i ∈ [1, N], and F l (x i ) are the features from the hierarchical feature extractor.

[0093] The projection matrix is:

[0094]

[0095] where, represents the projection matrix of the l-th layer, and F l (x i ) represents the features of the i-th training sample x i ; Var(F l (x i )) = F l (x i)·F l (x i ) represents feature F l (x i ) is the variance of Cov(F l (x i ), F l (x v )) = F l (x i )·F l (x v ) are the covariances of the features of two training samples x i and x v ;

[0096] Perform eigen - decomposition on the projection matrix to obtain an orthogonal matrix, including: To make the covariance zero and maximize the variance, eigen - decompose the projection matrix as:

[0097]

[0098] where Q l is the orthogonal matrix, is the inverse of Q l , Λ is the eigenvalue matrix, the eigenvalues in Λ are arranged in descending order, and

[0099] In an exemplary embodiment, determine the orthogonal basis of the training samples at the hierarchy according to the orthogonal matrix, including: Determine the dimension K of the hierarchical dimensionality reduction according to the features of the training samples at the hierarchy l ; Determine the columns from the 1st column to the K l + 1st column in the orthogonal matrix as the orthogonal basis.

[0100] Optionally, let the columns from the 1st column to the K l + 1st column of the orthogonal matrix Q l in formula (3) be the new matrix where the dimension K l of the dimensionality reduction can be calculated from the feature information of each layer in the formula, and the calculation method of the dimension K l of the dimensionality reduction is:

[0101]

[0102] where A l and A L respectively represent the averages of the features of the training samples at the l - th layer and the L - th layer (the last layer), N represents the number of training samples, F l (x j ) and F L (x jrespectively represent the training sample x j features at the l-th layer and the L-th layer, M L represents the dimension of the L-th layer; F l (x i ) and F L (x i ) respectively represent the training sample x i features at the l-th layer and the L-th layer.

[0103] Optionally, the calculation method of the features after dimensionality reduction at each level is:

[0104]

[0105] where F' l represents the feature after dimensionality reduction at the l-th layer, F' l = [F' l (x 1 ), F' l (x 2 ), …, F' l (x N )] T , F l represents the feature of the image to be detected at the l-th layer, P l represents the orthogonal basis of the l-th layer.

[0106] S104. Input the features after dimensionality reduction at each level into the cross-layer score aggregator, calculate the score of the image to be detected through the cross-layer score aggregator, and compare the score of the image to be detected with the preset score threshold to determine the abnormal distribution detection result of the image to be detected.

[0107] Dimensionality reduction is performed on the features of the training samples at different levels through formula (7) to obtain the feature space F' of the training samples after dimensionality reduction l .

[0108] The cross-layer score aggregator is a module for comprehensively processing features at different levels; the cross-layer score aggregator specifically includes:

[0109] (1) Normalize the features of the training samples after dimensionality reduction:

[0110]

[0111] where the symbol ||·|| 2 represents the Euclidean distance, and denote the normalized features of the training samples as

[0112] (2) Calculate the Euclidean distance between the features of the training sample set:

[0113] D l (x i ) v =‖Z l (x i ) - z l (x v )|| 2 (9)

[0114] where z l (x i ), v ≠ i.

[0115] (3) Rearrange {D l (x i ) 1 , D l (x i ) 2 , …, D l (x i ) N} in ascending order to get {D l (x i ) (1) , D l (x i ) (2) , …, D l (x i ) (N)}.

[0116] (4) Design the score of the distribution sample by integrating the feature distances of each layer as follows:

[0117]

[0118] where is the p-th shortest distance between the training sample x i and other training samples.

[0119] (5) Calculate the threshold score through the following formula:

[0120]

[0121] where 1 {·} is the indicator function, and the preset score threshold λ needs to be adjusted according to a certain ratio to correctly classify most of the normal data within the same distribution (for example, accuracy: 95%).

[0122] Continuously adjust p and λ, and based on the preset prediction accuracy, find the optimal p and λ, and then perform distribution anomaly detection through the optimal p and λ.

[0123] After obtaining the features of each layer of the training samples after projection dimensionality reduction, directly mixing them will lead to confusion in feature representation. Therefore, the Euclidean distance is used to unify these representation vectors, and the scores of the distribution samples are designed using the distances between the training sample features. Finally, the threshold is set through the indicator function to ensure that more than 95% of the training set samples can be correctly classified.

[0124] Therefore, in an exemplary embodiment, calculating the score of the image to be detected through a cross-layer score aggregator includes: for any level, normalizing the features after dimensionality reduction at that level to obtain normalized features; calculating the Euclidean distance between the normalized features and the normalized features of the historical detected images; the historical detected images are the images for which abnormal distribution detection was performed before the image to be detected; obtaining the p-th shortest distance among the Euclidean distances; and determining the ratio of the sum of the p-th shortest distances among the Euclidean distances at all levels to the total number of all levels as the score of the image to be detected. Among them, the specific implementation manner of this embodiment is the same as the specific description of the above embodiment, and this embodiment will not be elaborated here.

[0125] Among them, the historical detected images can be the images for which abnormal distribution detection was performed between the images to be detected.

[0126] Optionally, comparing the score of the image to be detected with a preset score threshold to determine the abnormal distribution detection result of the image to be detected includes: if the score of the image to be detected is greater than the preset score threshold, determining that the abnormal distribution detection result of the image to be detected is that the image to be detected is data with an abnormal distribution; if the score of the image to be detected is less than or equal to the preset score threshold, determining that the abnormal distribution detection result of the image to be detected is that the image to be detected is data with a normal distribution.

[0127] In an exemplary embodiment, if the abnormal distribution detection result of the image to be detected is data with an abnormal distribution, upload the image to be detected to the server, and give an alarm reminder on the display screen interface of the main control console of the autonomous driving vehicle, and emit a prompt sound to indicate that there is abnormal distribution data in the current autonomous driving scenario.

[0128] As Figure 2 and Figure 3 shown, Figure 2 is a schematic diagram of an alarm on the display screen interface of the main control console of an autonomous driving vehicle; Figure 3 is a structural diagram of a voice alarm module. The voice alarm module includes a power supply input unit, a relay output unit, a relay status indicator light, an external microphone interface, an on-board microphone, a speaker interface, a voice upgrade port, a power light, and a speaker. The speaker can be an 8R 0.5 speaker.

[0129] In an exemplary embodiment, as Figure 4 shown, Figure 4It is a flowchart for training the network architecture of an anomaly distribution detection method based on an autonomous driving scenario; first, obtain a training data set in the same distribution, extract the deep and shallow features of the training data set through a pre-trained neural network model, calculate the orthogonal basis of the projection subspace according to the deep and shallow features of the training data set, project the deep and shallow features of the training data set through the orthogonal basis, then calculate the scores of the distribution samples according to the projected features, and then calculate the preset score threshold.

[0130] As Figure 5 shown, Figure 5 It is a flowchart for the actual application of an anomaly distribution detection method based on an autonomous driving scenario; obtain data in the real world, extract the deep and shallow features of the data in the real world through a pre-trained neural network model, project the deep and shallow features onto a subspace, calculate the scores of the data according to the projected features, compare the scores with the preset score threshold, and determine whether the data is an anomaly distribution sample. If the data is an anomaly distribution sample, the display interface of the main console of the autonomous driving vehicle will alarm and emit an alarm sound.

[0131] The present invention is an anomaly distribution detection method based on an autonomous driving scenario. By annotating the target attribute labels of the image data collected in the autonomous driving system and combining the open-source autonomous driving scenario data set and the scenario target data set, a detailed database is constructed to form learning samples. Then, the method uses a pre-trained deep learning model to perform hierarchical feature extraction on the image data to obtain deep and shallow features. Then, an adaptive dimensionality reduction strategy is used to project the features onto an orthogonal basis for dimensionality reduction processing. Then, a cross-layer score aggregator is used to calculate the Euclidean distance between the sample features, and the distance information of each layer is combined to calculate the scores of the distribution samples, and a preset threshold is set. Finally, based on the preset threshold score, the system will make an anomaly distribution detection decision to determine whether the test sample is an anomaly distribution sample. Once an anomaly distribution sample is detected, the system will perform corresponding alarm processing according to the detection result.

[0132] In view of the defect that the abnormal distribution detection method in the existing autonomous driving system fails to fully focus on scene details and utilize attribute information to improve the detection accuracy, in the method proposed by the present invention, through a custom network structure, the recognition of covariate shift in the distribution shift of the autonomous driving scene is increased, and at the same time, the correct rate of abnormal distribution detection is improved. By using an elaborate scene database, the reliability and scalability of the system are increased. The present invention not only makes up for the characteristic that the out-of-abnormal-distribution detection is insensitive to the covariate shift of the domain, but also improves the detection ability of the autonomous driving system for unknown situations, enhances the safety and robustness of the autonomous driving system, and provides a solid technical foundation for realizing more advanced autonomous driving functions. This provides important technical support for the work of realizing the safety monitoring, real-time abnormal processing and intelligent transportation system optimization of the autonomous driving system. The method of the present invention is an important basis for improving the decision-making ability of the autonomous driving system in a complex traffic environment, ensuring driving safety and optimizing traffic flow.

[0133] To verify the reliability of the method provided by the present invention, a comparative experiment is carried out on the ImageNet-1K benchmark: this benchmark uses the ImageNet-1K dataset as the training set, and uses SSB-hard, ImageNet-21K, NINCO, iNaturalist, Textures, OpenImage-O as test samples in the abnormal distribution. The experimental result analysis respectively uses the indexes AUROC and FPR@95 for abnormal distribution detection to evaluate, and the higher the AUROC and the lower the FPR@95, the better the performance.

[0134] Among them, the comparative methods include MSP, ODIN, Energy, KLM, VIM and other methods. The test results on the ImageNet-1K dataset are shown in Table 1.

[0135] Table 1

[0136]

[0137] The experimental results show that the present invention, through a custom network structure, considering the covariate shift between distributions, and using a large-scale scene dataset, realizes the effective recognition of abnormal distribution samples while improving the accuracy of the original algorithm, and provides an important basis for realizing the safe travel and intelligent analysis of autonomous driving.

[0138] When applying the abnormal distribution detection method based on the autonomous driving scene provided by the present invention, it is not necessary to execute according to Figure 1 the order of the steps shown. The specific execution order of each step can be determined according to needs, and the present invention does not make any restrictions on this.

[0139] The above is the method for detecting abnormal distribution based on the autonomous driving scenario provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding device for detecting abnormal distribution based on the autonomous driving scenario, as Figure 6 shown.

[0140] Figure 6 FIG. is a schematic diagram of a device for detecting abnormal distribution based on the autonomous driving scenario provided by the present invention. The device 600 includes:

[0141] An acquisition module 601, configured to acquire an image to be detected in the autonomous driving scenario;

[0142] An extraction module 602, configured to perform hierarchical feature extraction on the image to be detected to obtain features of the image to be detected at different levels;

[0143] A projection module 603, configured to project the features of the image to be detected at different levels onto the orthogonal basis corresponding to each level to obtain the features after dimensionality reduction processing at each level;

[0144] A calculation module 604, configured to input the features after dimensionality reduction processing at each level into a cross-layer score aggregator, and calculate the score of the image to be detected through the cross-layer score aggregator; the cross-layer score aggregator is a module for comprehensively processing features at different levels;

[0145] A comparison module 605, configured to compare the score of the image to be detected with a preset score threshold to determine the detection result of the abnormal distribution of the image to be detected.

[0146] For the specific limitations on the device for detecting abnormal distribution based on the autonomous driving scenario, reference can be made to the limitations on the method for detecting abnormal distribution based on the autonomous driving scenario in the above text, which will not be elaborated here. Each module in the above device for detecting abnormal distribution based on the autonomous driving scenario can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0147] The present invention also provides a computer-readable storage medium, which stores a computer program, and the computer program can be used to execute the above Figure 1 provided method for detecting abnormal distribution based on the autonomous driving scenario.

[0148] The present invention also provides Figure 7 the structural schematic diagram of the computer device shown in FIG., as Figure 7As shown, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to implement the above Figure 1 abnormal distribution detection method based on the autonomous driving scenario provided.

[0149] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods. Among them, any reference to a memory, storage, database, or other medium used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0150] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope recorded by the present invention.

Claims

1. A method for detecting abnormal distribution based on an autonomous driving scenario, characterized in that: include: Obtain the image to be detected in the autonomous driving scenario; Performing hierarchical feature extraction on the image to be detected to obtain features of the image to be detected at different levels; Projecting the features of the image to be detected at different levels onto the orthogonal basis of the corresponding level to obtain the features of each level after dimensionality reduction processing; Input the features after dimensionality reduction processing at each level into a cross-layer score aggregator, and calculate the score of the image to be detected by the cross-layer score aggregator; the cross-layer score aggregator is a module that comprehensively processes features at different levels; The score of the image to be detected is compared with a preset score threshold to determine an abnormal distribution detection result of the image to be detected.

2. The method according to claim 1, characterized in that The step of performing hierarchical feature extraction on the image to be detected to obtain features of the image to be detected at different levels includes: Inputting the image to be detected into a pre-trained neural network model, performing hierarchical feature extraction on the image to be detected through the neural network model, and obtaining initial features of the image to be detected at different levels; the initial features are multi-dimensional features; The initial features at different levels are converted into one-dimensional vectors by a hierarchical feature extractor to obtain the features of the image to be detected at different levels.

3. The method according to claim 1, characterized in that The calculation method of the features after the dimensionality reduction processing at each level is: Among them, F' l represents the features after the dimensionality reduction process at the lth layer, F l represents the features of the lth layer, P l represents the orthogonal basis of the lth level.

4. The method according to claim 3, characterized in that: The construction methods of the orthogonal bases at each level include: Obtain training samples within the distribution data from the autonomous driving system, open source autonomous driving scenario datasets, and scenario target datasets; Performing hierarchical feature extraction on the training samples to obtain features of the training samples at different levels; Determining projection matrices of the training samples at different levels according to the features of the training samples at different levels; For the projection matrix at any level, the projection matrix is ​​subjected to eigendecomposition to obtain an orthogonal matrix, and the orthogonal basis of the training samples at the level is determined based on the orthogonal matrix.

5. The method according to claim 4, characterized in that The projection matrix is: in, represents the projection matrix of the lth layer, F l (x i ) represents the i-th training sample x i Characteristics of Var(F l (x i ))=F l (x i )·F l (x i ) represents the feature F l (x i ), Cov(F l (x i ),F l (x v ))=F l (x i )·F l (x v ) are two training samples x i and x v The covariance of the features.

6. The method according to claim 4, characterized in that Determining the orthogonal basis of the training samples at the level according to the orthogonal matrix includes: According to the characteristics of the training sample at the level, the dimension K of the level reduction is determined l ; The first to the Kth columns of the orthogonal matrix l The +1 column is determined as the orthogonal basis.

7. The method according to claim 6, characterized in that The dimension K of the dimensionality reduction l The calculation method is: Among them, A l and A L Respectively represent the average values ​​of the features of the training samples in the lth layer and the Lth layer (the last layer), N represents the number of training samples, and F l (x j ) and F L (x j ) represent the training samples x j In the features of the lth and Lth layers, M L represents the dimension of the Lth layer; F l (x i ) and F L (x i ) represent the training samples x i Features at layer l and layer L.

8. The method according to claim 1, characterized in that The calculating the score of the image to be detected by a cross-layer score aggregator includes: For any level, normalize the features after dimensionality reduction processing of the level to obtain normalized features; Calculating the Euclidean distance between the normalized feature and the normalized feature of a historical detection image; the historical detection image is an image that was detected for abnormal distribution before the image to be detected; Get the pth shortest distance in the Euclidean distance; The ratio of the sum of the p-th shortest distances in the Euclidean distances at all levels to the total number of all levels is determined as the score of the image to be detected.

9. The method according to claim 1, characterized in that: The step of comparing the score of the image to be detected with a preset score threshold to determine the abnormal distribution detection result of the image to be detected includes: If the score of the image to be detected is greater than the preset score threshold, determining the abnormal distribution detection result of the image to be detected as data that the image to be detected is abnormally distributed; If the score of the image to be detected is less than or equal to the preset score threshold, it is determined that the abnormal distribution detection result of the image to be detected is data indicating that the image to be detected is normally distributed.

10. The method according to claim 1, characterized in that The method further comprises: If the abnormal distribution detection result of the image to be detected is abnormal distribution data, the image to be detected is uploaded to the server, and an alarm reminder is issued on the display screen interface of the main console of the autonomous driving vehicle.

Images