Test-oriented classification method for limit scenarios of expected functionality of vehicle-mounted vision sensors

Abstract

The application provides a limit scene classification method for expected functions of a vehicle-mounted visual sensor, and the method comprises the following steps: acquiring driving scene data of a vehicle and an image of a driving scene captured by a vehicle-mounted visual sensor; determining an objective image quality evaluation score according to parameters of the image of the driving scene; extracting features of the driving scene data by using a multilayer perception machine to obtain corresponding feature vectors; performing feature fusion and splicing on the feature vectors corresponding to the driving scene data, the feature vectors corresponding to the objective image quality evaluation score and the feature vectors of the image of the driving scene to obtain fused features; and inputting the fused features into a limit scene recognition network to output a safety category of the vehicle-mounted data sensor under a current driving condition. The scheme can reduce the dependence of an intelligent driving vehicle environment perception system on a vehicle-mounted visual sensor, improve the expected function safety of the vehicle-mounted visual sensor and avoid traffic accidents.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving expected function safety technology, and specifically relates to an extreme scenario classification method for testing the expected functions of onboard vision sensors. Background Technology

[0002] Autonomous driving technology is a research hotspot that integrates multiple disciplines such as traffic engineering, vehicle engineering, communication engineering, and computer science. It is an effective way to ensure traffic safety and improve traffic efficiency. Although autonomous driving technology is constantly developing, autonomous vehicles must undergo extensive testing in real and reliable traffic scenarios before being launched to the market to ensure the safety of traffic personnel. Therefore, testing technology is an indispensable and important part of the research and development and application of autonomous driving technology.

[0003] With the continuous development of autonomous driving testing technology and the increasing emphasis on the safety of its intended functions, testing the intended functions of autonomous vehicles has become a key research direction for researchers. Autonomous vehicles consist of three parts: environmental perception, decision-making, and control, with environmental perception being a crucial component. Onboard vision sensors, as vital sensors for environmental perception in autonomous vehicles, perform important tasks such as obstacle detection and tracking, providing accurate information support for decision-making and control, thereby ensuring traffic safety and improving traffic efficiency. Therefore, research on testing the intended functions of onboard vision sensors is of paramount importance.

[0004] Due to the complexity, timeliness, and dispersion of driving scenarios for autonomous vehicles, which require consideration of multiple factors such as weather, roads, and other road users, effectively testing the expected functional safety of onboard vision sensors and classifying driving scenarios according to their degree of hazard are urgent research issues for ensuring the safety of road users in autonomous vehicles. With the rise of neural network artificial intelligence technology, neural network technology has provided new ideas for identifying extreme scenarios of the expected functions of onboard vision sensors. Summary of the Invention

[0005] The purpose of the embodiments in this specification is to provide a method for classifying extreme scenarios for testing the expected functions of vehicle-mounted vision sensors.

[0006] To solve the above-mentioned technical problems, the embodiments of this application are implemented in the following ways:

[0007] This application provides a method for classifying extreme scenarios for testing the expected functionality of vehicle-mounted vision sensors, the method comprising:

[0008] Acquire vehicle driving scene data and images of the driving scene captured by the vehicle's vision sensors;

[0009] Based on the parameters of the images in the driving scenario, an objective image quality evaluation score is determined;

[0010] Features are extracted from driving scene data using a multilayer perceptron to obtain the corresponding feature vectors;

[0011] The feature vectors corresponding to driving scene data, the feature vectors corresponding to objective image quality evaluation scores, and the feature vectors of driving scene images are fused and concatenated to obtain fused features.

[0012] The fused features are input into the extreme scene recognition network, which outputs the safety category of the vehicle data sensor under the current driving conditions.

[0013] In one embodiment, determining an objective image quality assessment score based on image parameters includes:

[0014] Based on the image parameters, determine the image brightness, image sharpness, and image entropy value;

[0015] An objective image quality evaluation score is obtained by weighting image brightness, image sharpness, and image entropy.

[0016] In one embodiment, the driving scenario data includes at least: driving scenario data in the dimensions of weather, road, vehicle driving parameters, and traffic participants and static obstacles;

[0017] Features are extracted from driving scene data using a multilayer perceptron to obtain the corresponding feature vectors, including:

[0018] Six-dimensional weather data is input into a multilayer perceptron, which outputs a three-dimensional feature vector.

[0019] Four-dimensional road data is input into a multilayer perceptron, which outputs a two-dimensional feature vector.

[0020] Five-dimensional data on traffic participants and static obstacles are input into a multilayer perceptron, which outputs a two-dimensional feature vector.

[0021] The vehicle's driving parameters (3D data) are input into a multilayer perceptron, which outputs a 1D feature vector.

[0022] In one embodiment, the extreme scene recognition network employs an optimized, densely connected convolutional network.

[0023] In one embodiment, the optimized densely connected convolutional network includes a densely connected optimization module.

[0024] In one embodiment, the dense connection optimization module includes an input layer, a first normalization, a first activation function, a first convolutional layer, a first feature layer, a second normalization, a second activation function, a second convolutional layer, a second feature layer, a third normalization, a third activation function, a third convolutional layer, a third feature layer, and an output layer.

[0025] In one embodiment, the first activation function, the second activation function, and the third activation function are SiLU activation functions.

[0026] In one embodiment, the optimized densely connected convolutional network includes a transition module.

[0027] In one embodiment, the transition module includes an input layer, a batch normalization layer, a 1×1 convolutional layer, a 2×2 average pooling layer, and an output layer.

[0028] As can be seen from the technical solutions provided in the embodiments of this specification above, this solution: after classifying and detecting the feature map, outputs the detection result and the detection is completed. When the detection result of the current vehicle vision sensor is a dangerous scene in the extreme scene recognition of the expected functional safety of the vehicle vision sensor, it can reduce the dependence of the intelligent driving vehicle environmental perception system on the vehicle vision sensor, improve the expected functional safety of the vehicle vision sensor, avoid traffic accidents, and make a certain contribution to the safety of the human-vehicle-road-cloud collaboration in the future Internet of Vehicles. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A flowchart illustrating the extreme scenario classification method for testing the expected functions of vehicle-mounted vision sensors provided in this application;

[0031] Figure 2 A diagram of the weather dimension components provided for this application;

[0032] Figure 3 The road dimension component diagram provided for this application;

[0033] Figure 4 A diagram illustrating the dimensional components of traffic participants and static obstacles provided for this application;

[0034] Figure 5 A schematic diagram of the dense connection optimization module provided in this application;

[0035] Figure 6 A schematic diagram of the structure of the transition module provided in this application;

[0036] Figure 7 This is another flowchart illustrating the extreme scenario classification method for testing the expected functions of vehicle-mounted vision sensors provided in this application. Detailed Implementation

[0037] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0038] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0039] Various modifications and variations can be made to the specific embodiments described in this application without departing from the scope or spirit of this application, as will be apparent to those skilled in the art. Other embodiments derived from this application will be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0040] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0041] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0042] Reference Figure 1 It illustrates a flowchart of an extreme scenario classification method for testing the expected functions of an onboard vision sensor, as provided in the embodiments of this application.

[0043] like Figure 1 As shown, the extreme scenario classification method for testing the expected function of vehicle-mounted vision sensors may include:

[0044] S110: Acquire vehicle driving scene data and images of the driving scene captured by the vehicle's vision sensor.

[0045] Specifically, the vehicle's driving scenario data includes at least four dimensions: weather, road, vehicle driving parameters, and traffic participants and static obstacles. Each dimension is independent of the others.

[0046] The weather dimension refers to the different types of weather conditions required for autonomous vehicles to operate, including weather types and climate parameters. Weather types include sunny, cloudy, overcast, snowy, and rainy weather, while climate parameters include temperature, humidity, visibility, and air pollution index. For specific classification methods, see [link to relevant documentation]. Figure 2 .

[0047] The road dimension refers to the dimension in which autonomous vehicles drive on different roads. It constructs road dimension scenarios from three aspects: road surface conditions, road surface types (or road surface categories), and road conditions. Road surface conditions primarily define road surface conditions, including structured and unstructured roads. Both structured and unstructured road levels include four levels: smooth, potholes, uphill, and downhill. Road surface types primarily define the types of road surfaces within the driving scenario, including different levels such as asphalt pavement, concrete pavement, flooded pavement, icy pavement, and speed bump pavement. Road conditions mainly cover dynamic road users within the scenario, including different road conditions such as highway intersections, straight roads, S-curves, crossroads, roundabouts, T-junctions, and tunnels. For a detailed explanation of the road dimension division method, see [link to relevant documentation]. Figure 3 .

[0048] The traffic participant and static obstacle dimensions are important targets for vehicle-mounted perception sensors in driving scenarios. These include traffic participants such as cars, trucks, commercial vehicles, vans, and two-wheeled vehicles, as well as static obstacles. For each category of traffic participant, information includes distance from the vehicle, speed, and acceleration. For static obstacles, the primary information is their distance from the vehicle. Each category of traffic participant and static obstacle is independent of the others. For a detailed explanation of the traffic participant and static obstacle dimension classification, see [link to relevant documentation]. Figure 4 .

[0049] The vehicle's own parameter dimension (i.e., the vehicle's own driving parameter dimension) is the main vehicle parameter element that constitutes the image data collected by the actual vehicle vision sensor. It includes the vehicle's lateral and longitudinal speeds, acceleration, and the installation height of the vehicle vision sensor.

[0050] The driving scene image captured by the vehicle vision sensor has a dimension of W*H*3, where W and H represent the length and width of the image, and 3 represents the three dimensions of the image: R, G, and B.

[0051] S120. Determine the objective image quality evaluation score based on the parameters of the driving scene image, including: determining the image brightness, image sharpness, and image entropy value based on the image parameters;

[0052] An objective image quality evaluation score is obtained by weighting image brightness, image sharpness, and image entropy.

[0053] Image brightness is one of the references for evaluating image quality, and its calculation method is shown in formula (1.1).

[0054]

[0055] Where M and N represent the horizontal and vertical resolutions of the image (which can be simply referred to as horizontal and vertical resolutions), R(i,j), G(i,j), and B(i,j) represent the R, G, and B values ​​of each pixel, respectively, and Y represents the image brightness.

[0056] Image sharpness is assessed using the Tenengrad gradient function as one of the image quality evaluation references. The gradient of the image at (x,y) is obtained from formula (1.2), and the Tenengrad value of the image is finally obtained from formula (1.3). The calculation method is shown in formulas (1.2)-(1.4):

[0057]

[0058]

[0059]

[0060] in and These represent g in formula (1.4) respectively. x The convolution of the kernel over pixels and g y The convolution kernel performs convolution on pixels, where M and N represent the horizontal and vertical resolutions of the image, respectively, and Ten is the Tenengrad value.

[0061] Entropy is used as one of the references for image quality evaluation. The calculation method is shown in formulas (1.5) and (1.6).

[0062] P ij = f(i,j) / MN(1.5)

[0063]

[0064] Where i represents the gray value of a pixel, j represents the average gray value of neighboring pixels, f(i,j) represents the frequency of occurrence of the feature pair i,j, M and N represent the horizontal and vertical resolutions of the image, and H represents the image entropy value.

[0065] The brightness, sharpness, and entropy values ​​of the dataset are normalized according to formula (1.7).

[0066]

[0067] Where, m min For the minimum value, m max It is the maximum value;

[0068] The objective combined image quality assessment score OA is:

[0069] OA = w oY Y N +w oTen Ten N +w oH H N (1.8)

[0070] Among them, w oY w is the weight for image brightness. oTen The weights for the image's Tenengrad values, w oH The weights are the image entropy values.

[0071] Understandably, when the extreme scene recognition network is in training mode, the aforementioned image brightness, image sharpness, and image entropy values ​​can be directly obtained from the extreme scene dataset of the vehicle vision sensor's expected functions. However, when the extreme scene recognition network is in non-training mode, the aforementioned image brightness, image sharpness, and image entropy values ​​are calculated using the methods described above.

[0072] S130. Extract features from driving scene data using a multilayer perceptron to obtain the corresponding feature vector, including:

[0073] Six-dimensional weather data is input into a multilayer perceptron, which outputs a three-dimensional feature vector.

[0074] Four-dimensional road data is input into a multilayer perceptron, which outputs a two-dimensional feature vector.

[0075] Five-dimensional data on traffic participants and static obstacles are input into a multilayer perceptron, which outputs a two-dimensional feature vector.

[0076] The vehicle's driving parameters (3D data) are input into a multilayer perceptron, which outputs a 1D feature vector.

[0077] Understandably, the driving scenario data for each of the above dimensions uses its own corresponding multi-layer perceptron.

[0078] S140. The feature vectors corresponding to the driving scene data, the feature vectors corresponding to the objective image quality evaluation scores, and the feature vectors of the driving scene images are fused and spliced ​​to obtain fused features.

[0079] Specifically, the 3D feature vector obtained from the weather dimension after passing through the MLP layer, the 2D feature vector from the road information dimension, the 2D feature vector from the traffic participants and static obstacles dimension, the 1D feature vector from the vehicle's own parameters dimension, the 1D feature vector from the objective image quality evaluation score, and the W*H*3 image of the driving scene captured by the onboard vision sensor are concatenated in the third dimension to perform feature fusion operation, outputting a W*H*12-dimensional fused feature that includes weather information, road information, traffic participants and static obstacles information, vehicle's own parameters information, image information, and objective image quality evaluation score information.

[0080] S150 inputs the fused features into the extreme scene recognition network and outputs the safety category of the vehicle data sensor under the current driving conditions.

[0081] Extreme scene recognition networks can employ optimized DenseNet (Dense Convolutional Network).

[0082] To reduce the amount of data, the number of Dense Block modules (dense connection optimization modules) in the DenseNet network structure was adjusted and modified to 2, 4, 8, and 4. See the detailed structure below. Figure 5 .

[0083] To reduce computational cost and improve detection speed, the Dense Block in DenseNet was simplified by proposing the L-Dense Block module structure. This simplifies the feature concatenation operation between layers in the previous Dense Block module. See [link to specific structure] for details. Figure 6 Specifically, the dense connection optimization module includes an input layer, a first normalization layer, a first activation function, a first convolutional layer, a first feature layer, a second normalization layer, a second activation function, a second convolutional layer, a second feature layer, a third normalization layer, a third activation function, a third convolutional layer, a third feature layer, and an output layer.

[0084] To compensate for the decrease in accuracy caused by reducing computational cost, the activation function in DenseNet is optimized by replacing the ReLU function with the SiLU activation function. The SiLU activation function formula is as follows:

[0085]

[0086] Optimized densely connected convolutional networks include transition layers. For example... Figure 7As shown, the transition module includes an input layer, a batch normalization layer, a 1×1 convolutional layer, a 2×2 average pooling layer, and an output layer.

[0087] The safety categories of onboard data sensors under the current driving conditions can include safe scenarios, relatively safe scenarios, and dangerous scenarios.

[0088] Figure 7 The diagram shown is another flowchart illustrating the extreme scenario classification method for testing the expected functionality of vehicle-mounted vision sensors according to this application. Figure 7 As shown, weather, road, surrounding vehicle, and vehicle information, along with RGB images, are acquired. The 6-dimensional weather data is input into a multilayer perceptron (MLP), outputting a 3-dimensional feature vector. The 4-dimensional road data is input into the MLP, outputting a 2-dimensional feature vector. The 5-dimensional traffic participant and static obstacle data (i.e., surrounding vehicle information) is input into the MLP, outputting a 2-dimensional feature vector. The 3-dimensional vehicle driving parameter data (i.e., vehicle information) is input into the MLP, outputting a 1-dimensional feature vector. The 3-dimensional feature vector obtained from the weather dimension, the 2-dimensional feature vector from the road information dimension, the 2-dimensional feature vector from the traffic participant and static obstacle dimension, the 1-dimensional feature vector from the vehicle's own vehicle parameters dimension, and the 1-dimensional feature vector from the objective image quality evaluation score are then combined with a W*H*3 image of the driving scene captured by the onboard vision sensor. Through a feature linking layer, a concatenation feature fusion operation is performed on the third dimension, outputting a W*H*12-dimensional fused feature that includes weather information, road information, traffic participant and static obstacle information, vehicle's own vehicle parameter information, image information, and objective image quality evaluation score information. Then, the feature input is fused into the extreme scene recognition network (which sequentially passes through a 7×7 convolutional layer, a max pooling layer, two dense connection optimization modules, a transition module, four dense connection optimization modules, a transition module, eight dense connection optimization modules, a transition module, four dense connection optimization modules, a 7×7 average pooling layer, a linear connection layer, and a normalized exponential function) to obtain the sensor's expected functional safety scene classification result.

[0089] The extreme scene classification method for testing the expected functions of vehicle-mounted vision sensors provided in this application classifies and detects feature maps, outputs detection results, and completes the detection. When the extreme scene identification and detection results of the current vehicle-mounted vision sensor's expected functional safety are dangerous scenes, it can reduce the dependence of the intelligent driving vehicle's environmental perception system on the vehicle-mounted vision sensor, improve the expected functional safety of the vehicle-mounted vision sensor, avoid traffic accidents, and make a certain contribution to the safety of human-vehicle-road-cloud collaboration in the future Internet of Vehicles.

[0090] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0091] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A method for classifying extreme scenarios to test the expected functionality of vehicle-mounted vision sensors, characterized in that, The method includes: Acquire vehicle driving scene data and images of the driving scene captured by the vehicle's vision sensors; An objective image quality evaluation score is determined based on the parameters of the image of the driving scenario. Features are extracted from the driving scene data using a multilayer perceptron to obtain the corresponding feature vector; The feature vectors corresponding to the driving scene data, the feature vectors corresponding to the objective image quality evaluation scores, and the feature vectors of the driving scene images are fused and concatenated to obtain fused features. The fused features are input into an extreme scene recognition network, which outputs the safety category of the vehicle-mounted data sensor under the current driving condition. The extreme scene recognition network employs an optimized densely connected convolutional network. The optimized densely connected convolutional network includes a dense connection optimization module. The dense connection optimization module includes an input layer, a first normalization layer, a first activation function, a first convolutional layer, a first feature layer, a second normalization layer, a second activation function, a second convolutional layer, a second feature layer, a third normalization layer, a third activation function, a third convolutional layer, a third feature layer, and an output layer. The first activation function, the second activation function, and the third activation function employ the SiLU activation function. The optimized densely connected convolutional network includes a transition module. The transition module includes an input layer, a batch normalization layer, a 1×1 convolutional layer, a 2×2 average pooling layer, and an output layer.

2. The method according to claim 1, characterized in that, The step of determining an objective image quality evaluation score based on the parameters of the image includes: Based on the parameters of the image, determine the image brightness, image sharpness, and image entropy value; The objective image quality evaluation score is obtained by weighting the image brightness, the image sharpness, and the image entropy value.

3. The method according to claim 1, characterized in that, The driving scenario data includes at least the following dimensions: weather, road, vehicle driving parameters, and traffic participants and static obstacles. The step of extracting features from the driving scene data using a multilayer perceptron to obtain the corresponding feature vector includes: The 6-dimensional weather data is input into the multilayer perceptron, which outputs a 3-dimensional feature vector. The 4-dimensional data of the road dimension is input into the multilayer perceptron, and the output is a 2-dimensional feature vector. The 5-dimensional data of traffic participants and static obstacles are input into the multilayer perceptron, which outputs a 2-dimensional feature vector. The three-dimensional data of the vehicle's driving parameters are input into the multilayer perceptron, which outputs a one-dimensional feature vector.

Images