Method for constructing a functional test scenario dataset for a vehicle-mounted vision sensor

Abstract

The application provides a vehicle-mounted visual sensor expected function test scene dataset construction method, which comprises the following steps: acquiring driving scene data of a vehicle; acquiring original information of images collected by a vehicle-mounted visual sensor on a driving scene; determining subjective and objective image information according to the quality of the collected images; determining the category of the expected function safety driving scene of the vehicle-mounted sensor according to the subjective and objective image information; and taking the driving scene data, the original information, the subjective and objective image information and the category of the expected function safety driving scene of the vehicle-mounted sensor of a plurality of images as a dataset of the expected function driving scene of the vehicle-mounted visual sensor. The scheme can provide a theoretical method for the construction of the expected function limit dataset of the automatic driving vehicle-mounted visual sensor.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, and specifically relates to a method for constructing a dataset of test scenarios for the expected functions of vehicle-mounted vision sensors. Background Technology

[0002] Autonomous driving technology is a hot research topic at the intersection of academic fields such as transportation, automotive, computer science, and communication control technology. It is an effective way to solve traffic congestion and reduce traffic accident rates. Autonomous driving consists of three parts: environmental perception, path planning, and decision control. The environmental perception module is the "eyes" of autonomous vehicles and an important module for the safe driving of intelligent connected vehicles. In the process of research and development to production and application of intelligent connected vehicles, the testing and evaluation of environmental perception functions are necessary to ensure that intelligent vehicles accurately understand the external environment and drive safely.

[0003] While autonomous driving technology is developing rapidly both domestically and internationally, some anticipated functional safety issues have become prominent. Compared with the development of autonomous driving technology, anticipated functional safety is still in its infancy. There is a lack of systematic research on sensor failure, data security, and driving road conditions, which means that the external risks of various systems in autonomous vehicles have not been effectively addressed. With the introduction of the anticipated functional safety standard ISO PAS 21448, the anticipated functional safety of autonomous driving has not only received much attention from many researchers, but has also pointed the way for the anticipated functional testing of onboard vision sensors for autonomous vehicles.

[0004] Vehicle-mounted vision sensors are a key infrastructure for the environmental perception system of autonomous vehicles. Identifying the categories of safe driving scenarios for the expected functions of vehicle-mounted sensors, quantifying the dimensions of vehicle driving scenario data, and using artificial intelligence technology to construct driving scenario datasets for testing the expected functions of vehicle-mounted vision sensors are powerful methods to improve the testing efficiency of environmental perception capabilities and effective means to promote the development of expected functional safety. Summary of the Invention

[0005] The purpose of the embodiments in this specification is to provide a method for constructing a dataset of test scenarios for the expected functions of vehicle-mounted vision sensors.

[0006] To solve the above-mentioned technical problems, the embodiments of this application are implemented in the following ways:

[0007] This application provides a method for constructing a test scenario dataset for the expected functions of an onboard vision sensor, the method comprising:

[0008] Acquire vehicle driving scenario data;

[0009] Acquire raw information from images captured by the vehicle's vision sensors of the driving scene;

[0010] Determine subjective and objective image information based on the quality of the acquired images;

[0011] Based on subjective and objective image information, determine the category of the expected safe driving scenario for the vehicle sensors;

[0012] The dataset of driving scene data from several images, raw information, subjective and objective image information, and categories of safe driving scenarios for the expected functions of vehicle-mounted sensors are used as the dataset of driving scenarios for the expected functions of vehicle-mounted vision sensors.

[0013] In one embodiment, the vehicle's driving scenario data includes at least the following dimensions: weather, road, vehicle driving parameters, and traffic participants and static obstacles.

[0014] In one embodiment, the original information of the image includes at least: aperture size, ISO, shutter speed, and image resolution.

[0015] In one embodiment, the subjective and objective image information includes image brightness;

[0016] Based on the quality of the acquired images, determine the subjective and objective image information, including:

[0017] If we obtain the horizontal resolution M, vertical resolution N, and R, G, B values ​​of each pixel in the image, then the image brightness Y is:

[0018]

[0019] Where R(i,j), G(i,j), and B(i,j) represent the R, G, and B values ​​of each pixel, respectively.

[0020] In one embodiment, the subjective and objective image information includes the image Tenengrad value;

[0021] Based on the quality of the acquired images, determine the subjective and objective image information, including:

[0022]

[0023]

[0024]

[0025] in, and They represent g respectively x The convolution of the kernel over pixels and g y The convolution kernel performs a convolution on each pixel; Ten is the Tenengrad value.

[0026] In one embodiment, the subjective and objective image information includes image entropy values;

[0027] Based on the quality of the acquired images, determine the subjective and objective image information, including:

[0028] If we obtain the gray value i of a pixel, the average gray value j of its neighboring pixels, and the frequency of occurrence of the feature pairs i,j, then the image entropy value H is:

[0029]

[0030] Among them, P ij = f(i,j) / MN.

[0031] In one embodiment, the subjective and objective image information includes a subjective score MOS value, which is the average of multiple image quality evaluation scores.

[0032] In one embodiment, the subjective and objective image information includes an objective combined image quality evaluation score.

[0033] Based on the quality of the acquired images, determine the subjective and objective image information, including:

[0034] Normalize the image brightness, image Tenengrad value, image entropy value, and subjective score MOS value:

[0035]

[0036] Where, m min For the minimum value, m max It is the maximum value;

[0037] The objective combined image quality assessment score OA is:

[0038] OA = w oY Y N +w oTen Ten N +w oH H N

[0039] Among them, w oY w is the weight for image brightness. oTen The weights for the image's Tenengrad values, w oH The weights are the image entropy values.

[0040] In one embodiment, the subjective and objective image information includes a combined subjective and objective image quality assessment score (SOA):

[0041] SOA = w o OA+w s MOS N

[0042] Among them, w o To objectively incorporate the weights of the image quality assessment score OA, w s The weight of the subjective score MOS value.

[0043] In one embodiment, the subjective and objective image information includes a combined subjective and objective image quality assessment score (SOA).

[0044] Based on subjective and objective image information, determine the categories of expected safe driving scenarios for vehicle-mounted sensors, including:

[0045] If 0 ≤ SOA ≤ 0.3, then the expected functional safe driving scenario for the vehicle sensor is a dangerous scenario.

[0046] If 0.3 < SOA ≤ 0.7, then the expected functional safety driving scenario for the vehicle sensor is a relatively safe scenario.

[0047] If 0.7 < SOA ≤ 1.0, then the expected function of the vehicle sensor is a safe driving scenario.

[0048] As can be seen from the technical solutions provided in the embodiments of this specification above, this solution provides a theoretical method for constructing the expected functional limit dataset of autonomous vehicle-mounted visual sensors. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart illustrating the method for constructing the test scenario dataset for the expected functions of the vehicle-mounted vision sensor provided in this application;

[0051] Figure 2 A diagram of the weather dimension components provided for this application;

[0052] Figure 3 The road dimension component diagram provided for this application;

[0053] Figure 4 A diagram illustrating the dimensional components of traffic participants and static obstacles provided for this application;

[0054] Figure 5 This is a multi-dimensional hierarchical framework diagram of the driving scenario for autonomous vehicles provided in this application. Detailed Implementation

[0055] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0056] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0057] Various modifications and variations can be made to the specific embodiments described in this application without departing from the scope or spirit of this application, as will be apparent to those skilled in the art. Other embodiments derived from this application will be obvious to those skilled in the art. This application specification and embodiments are merely exemplary.

[0058] The terms “include,” “including,” “have,” “contain,” etc., used in this article are all open-ended terms, meaning that they include but are not limited to.

[0059] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0060] Reference Figure 1 This document illustrates a flowchart of a method for constructing a test scenario dataset for the expected functions of vehicle-mounted vision sensors, applicable to the embodiments of this application. This method can fill the current gap in the field of testing the safety limits of expected functions of autonomous vehicle-mounted vision sensors.

[0061] like Figure 1 As shown, the method for constructing a test scenario dataset for the expected functions of an onboard vision sensor may include:

[0062] S110. Obtain vehicle driving scenario data. This data must include at least the following dimensions: weather, road conditions, vehicle driving parameters, and traffic participants and static obstacles.

[0063] Specifically, this application establishes a multi-dimensional hierarchical framework for autonomous vehicle driving scenarios. The overall system structure is divided into four dimensions: weather, road, vehicle driving parameters, and traffic participants and static obstacles. Each dimension is independent of the others. It is understood that this application only provides methodological guidance; in practical applications, different dimensions of the driving scenario framework can be added as needed to facilitate the addition or deletion of constituent elements.

[0064] Each dimension contains various elements. By combining the elements within each dimension, we can construct extreme scenarios for vehicle vision sensors with different expected functional safety levels. These scenarios can be used to provide extreme scenario detection and early warning for the expected functional safety of vehicles under test at different levels of autonomous driving.

[0065] The first dimension is the weather dimension, which is the driving scenario for different types of weather to be constructed for autonomous vehicles. This includes weather types and climate parameters. Weather types include sunny, cloudy, overcast, snowy, and rainy weather, among other driving scenario weather types. Climate parameters include driving environment climate parameters such as temperature, humidity, visibility, and air pollution index. For a detailed breakdown, see [link to relevant documentation]. Figure 2 .

[0066] In this dimension, weather type and various parameters in climate parameters are combined to form a weather-dimensional driving scenario. For example, if the weather type is sunny, the climate parameters are cool temperature, dry humidity, good visibility, good air pollution index, and wind force level 2, then a weather-dimensional driving scenario is formed.

[0067] The second dimension is the road dimension, which represents the driving experience of autonomous vehicles on different roads. The road dimension scenario is constructed from three aspects: road surface conditions, road surface type (or road surface category), and road conditions. Road surface conditions primarily define road surface conditions, including structured and unstructured roads. Both structured and unstructured road levels include four sub-levels: smooth, potholed, uphill, and downhill. Road surface type primarily defines the types of road surfaces within the driving scenario, including different sub-levels such as asphalt pavement, concrete pavement, wading pavement, icy pavement, and speed bump pavement. Road conditions primarily cover dynamic road users within the scenario, including different road conditions such as highway intersections, straight roads, S-curves, crossroads, roundabouts, T-junctions, and tunnels. For a detailed explanation of the road dimension's classification method, see [link to relevant documentation]. Figure 3 .

[0068] In this dimension, road surface conditions, road surface types, and various road dimension parameters combine to form a road dimension driving scenario. For example, a road surface condition of uphill on a structured road, a road surface classification of asphalt pavement, and a road condition of an intersection constitute one road dimension driving scenario.

[0069] The third dimension is the traffic participant and static obstacle dimension, which is an important target for vehicle-mounted perception sensors in driving scenarios. It includes traffic participants such as cars, trucks, commercial vehicles, vans, and two-wheeled vehicles, as well as static obstacles. For each category of traffic participant, information includes distance from the vehicle, speed, and acceleration. For static obstacles, the information mainly includes their distance from the vehicle. Each category of traffic participant and static obstacle is independent of the others. For a detailed explanation of the traffic participant and static obstacle dimension division, see [link to relevant documentation]. Figure 4 .

[0070] Within this dimension, one type of traffic participant or static obstacle is selected as the object of perception for the vehicle-mounted vision sensor. For example, a car that is 0-10m away from the vehicle, moving at a speed of 10km / h and with an acceleration of 0m / s².

[0071] The fourth dimension is the vehicle's own parameter dimension (i.e., the vehicle's own driving parameter dimension), which consists of the main vehicle parameter elements that constitute the image data collected by the actual vehicle vision sensor. These include the vehicle's lateral and longitudinal speeds, acceleration, and the installation height of the vehicle vision sensor.

[0072] In this dimension, the vehicle's own parameters are formed by the combination of three parameters: driving speed, driving acceleration, and the height of the onboard vision sensor. For example, the vehicle's own parameters in the driving scenario are a driving speed of 20 km / h, a driving acceleration of 0 m / s², and an onboard vision sensor installation height of 1.2 m.

[0073] Multi-dimensional hierarchical framework for autonomous vehicle driving scenarios, such as Figure 5 As shown, the four driving scenarios only list the main elements of the scenario. In actual application, the required elements of the scenario can be added or removed as needed.

[0074] S120. Acquire the raw information of the image captured by the vehicle vision sensor of the driving scene; wherein the raw information of the image includes at least: aperture size, ISO, shutter speed, and image resolution.

[0075] Specifically, based on the four-dimensional driving scenario construction mentioned above, the onboard vision sensor collects image data of the driving scenario. Because the environmental perception algorithm of autonomous driving based on vision sensors is mainly for detecting and perceiving traffic participants and static obstacles, the expected functional limit scenario dataset of the vision sensor also focuses on the raw information of images of traffic participants and static obstacles at different distances, speeds, and accelerations. The raw information of the images collected by the default vision sensor includes four parameters: aperture size, ISO sensitivity, shutter speed, and image resolution.

[0076] S130. Based on the quality of the acquired images, determine the subjective and objective image information, which includes at least the image brightness, image Tenengrad value, image entropy value, subjective score MOS value (or simply MOS value), weight of subjective score MOS value, weight of image brightness, weight of image Tenengrad value, weight of image entropy value, objective combined image quality evaluation score OA value (or simply OA value), and subjective and objective combined image quality evaluation score SOA value (or simply SOA value).

[0077] Specifically, a no-reference objective image quality assessment is performed on the collected data:

[0078] To reflect the impact of illumination on the visual sensor, image brightness is used as one of the references for image quality evaluation, and its calculation method is shown in formula (1.1).

[0079]

[0080] Where M and N represent the horizontal and vertical resolutions of the image (which can be simply referred to as horizontal and vertical resolutions), R(i,j), G(i,j), and B(i,j) represent the R, G, and B values ​​of each pixel, respectively, and Y represents the image brightness.

[0081] To reflect the sharpness of the acquired images, the Tenengrad gradient function is used as one of the references for image quality evaluation. The gradient of the image at (x,y) is obtained by formula (1.2), and finally the Tenengrad value of the image is obtained by formula (1.3). The calculation method is shown in formulas (1.2)-(1.4):

[0082]

[0083]

[0084]

[0085] in and These represent g in formula (1.4) respectively. x The convolution of the kernel over pixels and g y The convolution kernel performs convolution on pixels, where M and N represent the horizontal and vertical resolutions of the image, respectively, and Ten is the Tenengrad value.

[0086] To reflect the degree of pixel disorder in the acquired image and to distinguish between various weather conditions such as rain and snow, entropy value is selected as one of the references for image quality evaluation. The calculation method is shown in formulas (1.5) and (1.6).

[0087] P ij= f(i,j) / MN(1.5)

[0088]

[0089] Where i represents the gray value of a pixel, j represents the average gray value of neighboring pixels, f(i,j) represents the frequency of occurrence of the feature pair i,j, M and N represent the horizontal and vertical resolutions of the image, and H represents the image entropy value.

[0090] The image quality of the collected driving scene images was evaluated by multiple different manual observation methods. The average score (MOS) was obtained by averaging the image quality evaluation scores. The MOS value ranged from 0 to 9, with a higher value indicating better image quality.

[0091] The dataset brightness, image sharpness (i.e., image Tenengrad value), image entropy value, and subjective score MOS value are normalized according to formula (1.7).

[0092]

[0093] Where, m min For the minimum value, m max It is the maximum value;

[0094] The objective combined image quality assessment score OA is: (1.8)

[0096] OA = w oY Y N +w oTen Ten N +w oH H N

[0097] Among them, w oY w is the weight for image brightness. oTen The weights for the image's Tenengrad values, w oH The weights are the image entropy values.

[0098] Image Quality Assessment (SOA) scores, combining subjective and objective metrics:

[0099] SOA = w o OA+w s MOS N (1.9)

[0100] Among them, w o To objectively incorporate the weights of the image quality assessment score OA, w s The weight of the subjective score MOS value.

[0101] S140. Based on subjective and objective image information, determine the category of the expected functional safety driving scenario for the vehicle sensors, including:

[0102] If 0 ≤ SOA ≤ 0.3, then the expected functional safe driving scenario for the vehicle sensor is a dangerous scenario.

[0103] If 0.3 < SOA ≤ 0.7, then the expected functional safety driving scenario for the vehicle sensor is a relatively safe scenario.

[0104] If 0.7 < SOA ≤ 1.0, then the expected function of the vehicle sensor is a safe driving scenario.

[0105] Specifically, based on a combination of subjective and objective factors and the Image Quality Assessment (SOA) score, the expected functions of automotive vision sensors are categorized for safe driving scenarios, as shown in the table below.

[0106] SOA score threshold Intended Functional Safety Driving Scenarios for Vehicle Vision Sensors 0 ≤ SOA ≤ 0.3 Dangerous scenarios 0.3 < SOA ≤ 0.7 Safer scenarios 0.7 < SOA ≤ 1.0 security scenarios

[0107] S150. The driving scene data of several images, the original information, the subjective and objective image information, and the categories of the expected functions of the vehicle sensor for safe driving scenarios are used as the dataset of the expected functions of the vehicle vision sensor for driving scenarios.

[0108] Specifically, the information for each frame of the vehicle driving scene image dataset is summarized in a JSON file. The dictionary encoding order is as follows: weather type, climate parameters (temperature, humidity, visibility, air pollution index, wind force), road surface condition, road surface category, road conditions, type of traffic participant or static obstacle (distance, speed, acceleration), vehicle speed, acceleration, original image information (aperture size, ISO, shutter speed, image resolution), subjective and objective image information (image brightness, image Tenengrad value, image entropy value, MOS value, subjective score MOS weight, image brightness weight, image entropy weight, OA value, SOA value), and the expected function of the vehicle vision sensor for safe driving scenarios. These are then written into a label file and placed in the label folder. The collected image data is placed in the image folder. The dataset data is randomly shuffled, with 80% of the data used as the training set and 20% as the test set.

[0109] The method for constructing a test scenario dataset for the expected functions of vehicle-mounted visual sensors provided in this application decomposes the test scenario into four independent dimensions: weather, road, vehicle driving parameters, and traffic participants and static obstacles. Different levels of elements within each dimension are cross-combined to form different driving scenarios for the vehicle. Images of the driving scenarios are acquired by the vehicle-mounted visual sensor, and subjective and objective evaluations are given based on the image quality. The expected functions of the vehicle-mounted sensor are classified, and the expected functions driving scenario dataset of the vehicle-mounted visual sensor is constructed. The dataset construction method provided in this application provides a theoretical method for constructing the expected functions limit dataset of autonomous driving vehicle-mounted visual sensors.

[0110] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0111] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A method for constructing a dataset of expected functional test scenarios for a vehicle-mounted vision sensor, characterized in that, The method comprises: acquiring driving scene data of a vehicle; acquiring original information of an image collected by a vehicle-mounted visual sensor on a driving scene; determining subjective and objective image information according to the quality of the collected image, wherein the subjective and objective image information comprises image brightness; the determination of the subjective and objective image information according to the quality of the collected image comprises: Obtain the lateral resolution M, longitudinal resolution N and R, G, B value of each pixel point of the image, then the image brightness Y is: wherein, , , respectively represent the R, G, B values of each pixel point; the subjective and objective image information comprises an image Tenengrad value; the determination of the subjective and objective image information according to the quality of the collected image comprises: wherein, with respectively represent convolution of the convolution kernel on the pixel point and convolution of the convolution kernel on the pixel point; M, N represent horizontal and vertical resolutions of the image respectively; is the Tenengrad value; the subjective and objective image information comprises an image entropy value; the determination of the subjective and objective image information according to the quality of the collected image comprises: The gray value of the pixel is obtained i The average gray value of the neighborhood pixels j And the feature binary tuple i , j The frequency of occurrence, and the image entropy value H Is: wherein, ; representative feature pair i , j frequency of occurrence The subjective and objective image information includes a subjective score MOS The subjective score MOS The subjective score value is an average of a plurality of image quality evaluation scores. the subjective and objective image information comprises an objective combined image quality evaluation score value; the determination of the subjective and objective image information according to the quality of the collected image comprises: normalizing the image brightness, the image Tenengrad value, the image entropy value, the subjective score MOS values wherein is a minimum value, is a maximum value; The objective combined image quality evaluation score value OA Is: wherein, is a weight for image brightness, is a weight for image Tenengrad value, is a weight for image entropy value; The subjective and objective image information includes a subjective and objective combined image quality evaluation score value SOA : wherein is the objective combined image quality evaluation score value OA is the weight of the objective image quality evaluation score value, is the subjective score MOS is the weight of the subjective score value; The subjective and objective image information includes a subjective and objective combined image quality evaluation score value SOA ; determining the category of the expected functional safety driving scene of the vehicle-mounted sensor according to the subjective and objective image information, comprising: if 0≤SOA≤0.3, the expected functional safety driving scene of the vehicle-mounted sensor is a dangerous scene; if 0.3 if 0.7 determining the category of the expected functional safety driving scene of the vehicle-mounted sensor according to the subjective and objective image information; using the driving scene data, the original information, the subjective and objective image information and the category of the expected functional safety driving scene of the vehicle-mounted sensor as a data set of the expected functional driving scene of the vehicle-mounted visual sensor.

2. The method of claim 1, wherein, The driving scene data of the vehicle at least comprises driving scene data of weather, road, self-vehicle driving parameters, and traffic participants and static obstacles.

3. The method of claim 1, wherein, The original information of the image at least comprises aperture size, light sensitivity, shutter time, and image resolution.

Images