Intelligent automobile multi-path camera-in-the-loop test method based on video injection

Abstract

The application provides a kind of intelligent automobile multi-path camera in-loop test method based on video injection, including test object parameter selection, test scene design, edge scene extraction, test result evaluation, the application faces four-way camera data, and virtual simulation image data is injected into the ECU to be tested by video injection board card and fault injection board card for testing, while the application has unlimited rich problems for the parameters affecting camera test results, an edge test scene search method based on improved greedy algorithm is proposed, the edge test scene library is accelerated to be constructed under the premise of meeting the parameter coverage rate of 100%, and the edge test scene library constructed is evaluated, and the scene parameter influence evaluation, target detection algorithm static evaluation and dynamic evaluation method are output.

Description

Technical Field

[0001] This invention relates to a method for testing cameras in the loop in intelligent vehicles, and more particularly to a method for testing multiple cameras in the loop in intelligent vehicles based on video injection. Background Technology

[0002] Camera sensors, as one of the most commonly used hardware devices in intelligent vehicles, have become the preferred perception system hardware for automakers due to their lower cost compared to LiDAR and ultrasonic radar, and their richer feature capture capabilities. Furthermore, as the functions of intelligent vehicles become increasingly rich and complex, the number of cameras required is also gradually increasing. However, compared to LiDAR, millimeter-wave radar, and ultrasonic radar, cameras are passive sensors, significantly affected by external factors in the scene, such as complex lighting conditions and complex weather conditions like rain, fog, and snowfall. To address these complex weather conditions, intelligent vehicles equipped with camera sensors require extensive testing before being deployed in real vehicles to meet safety requirements. However, using traditional real-vehicle mileage-based testing methods would require significant time and financial resources. One effective solution is to conduct testing based on simulation scenarios. However, pure simulation testing deviates somewhat from real-vehicle testing. Therefore, hardware-in-the-loop testing can compensate for the high cost, poor repeatability, and low safety of real-vehicle testing, while also addressing the poor realism issues of pure virtual simulation testing. Hardware-in-the-loop testing (HIL) refers to embedding hardware entities into a virtual simulation testing environment and using the responses of real hardware to test intelligent vehicle systems. Camera-in-the-loop testing (BIB) is a testing method that embeds camera entities into a virtual simulation testing environment. BIB methods include projection-based BIB and injection-based BIB. Projection-based BIB is more severely affected by external light, while injection-based BIB suffers from problems such as an infinite number of parameters affecting camera test results, low parameter coverage, slow speed of building edge test scenario libraries, and lack of evaluation of edge test scenario libraries. Summary of the Invention

[0003] To address the aforementioned technical problems, this invention provides a method for loop-in testing of multi-channel cameras in intelligent vehicles based on video injection, comprising the following steps:

[0004] (I) Selection of Test Object Parameters

[0005] First, select the parameters of the test object, that is, the parameters of the test camera. The parameters of the test object mainly include two aspects: camera installation parameters and camera performance parameters. Camera installation parameters include installation position and installation angle, and camera performance parameters include camera focal length and camera sensor size.

[0006] The parameters of the test object mainly include two aspects: camera installation parameters and camera performance parameters. Camera installation parameters include installation position and installation angle, while camera performance parameters include camera focal length and camera sensor size.

[0007] The camera's installation position refers to the translation coordinate x, y, and z axes, with the vehicle's center of gravity as the origin, the forward direction of the vehicle as the x-axis, the leftward direction as the y-axis, and the vertical upward direction as the z-axis. a y a , z a ), where the subscript 'a' represents the corresponding camera number; the camera's mounting angle refers to the rotation angle (σ) around the x, y, and z axes, respectively. a ,β a γ a The camera focal length f refers to the lens focal length, which is the distance from the rear principal point of the lens to the focal point, measured in millimeters; the camera sensor size (X... a Y a () refers to the physical size of the CMOS image sensor to be simulated, in millimeters.

[0008] (II) Test Scenario Design

[0009] The test scenario parameters of this invention include the vehicle's motion state, weather conditions, and target object conditions;

[0010] (1) The motion state of the vehicle mentioned includes the vehicle speed v vut ;

[0011] (2) The meteorological categories mentioned include lighting conditions and complex weather: lighting conditions mainly describe the position of the sun source in the scene, and are described by azimuth angle α and polar angle θ; complex weather parameters include precipitation weather, fog weather and snowfall weather;

[0012] The meteorological data for rainfall use rainfall intensity μ and raindrop diameter D. rain Describe it;

[0013]

[0014] The above formula is used to set the corresponding rainfall scene parameters in the virtual simulation software;

[0015] The aforementioned fog weather utilizes visibility D fog Describe it;

[0016] The snowfall meteorology described uses the snowflake diameter D snow Describe it;

[0017] (3) The target object class includes the target object category T, the target object running direction Dri, and the target moving speed v. tar and collision location P;

[0018] The target object category T includes pedestrians and vehicles; the target object's direction of travel Dri includes, for pedestrians, distal, proximal, and longitudinal; for vehicles, it includes traveling in the same direction as the vehicle or crossing it; the target's speed v... tar For pedestrians, a speed of 5 to 8 km / h is used; for vehicles, a speed of 20 km / h and 50 km / h are used, accompanied by -2 to -6 m / s. 2 The braking deceleration; the collision position P refers to the position where the vehicle under test and the target object collide in front of the vehicle under test at their own speeds. For pedestrians, P is 25% to 75%, and for vehicles, P is -50% to 50%.

[0019] Each test scenario is described by an array of scenario parameters, specifically expressed as: Π=[v vut , α, θ, μ, a, D fog D snow T, Dri, v tar P].

[0020] (III) Edge Scene Extraction

[0021] This invention will test edge scenarios ∏ tar It is defined as a scenario in which there exists at least one test scenario parameter that satisfies the condition that changes by a certain threshold, thereby causing a significant change in the behavior or decision of the object under test. The threshold for the change is set according to the test accuracy and test requirements.

[0022] Edge testing scenario ∏ tar The screening steps are as follows:

[0023] Step 1: Initialize the parameters of the camera under test and the values ​​of discontinuous variables in the test scene;

[0024] Step 2: Chaos possesses a certain degree of randomness and initial value sensitivity, enabling the algorithm to converge faster. This invention utilizes the Sine chaotic mapping to generate a random chaotic sequence to initialize the parameters of the continuous test scenario in the test scenario ∏. The specific expression is as follows:

[0025] x i+1 =δsin(πx) i (2)

[0026] In the formula, δ is the system parameter, δ∈[0,1], and chaos occurs when δ∈(0.87,0.93)∪(0.95,1); x i x i+1It belongs to the iterative sequence value; x0∈(0,1); since the output of this chaotic mapping is a value in (0,1), it needs to be denormalized. Specifically, the maximum value Ω of the scene parameter is taken. min and minimum value Ω max Initialize the test scenario parameter Ω using denormalization:

[0027] Ω=Ω min +(Ω max -Ω min )*x i , i∈[0,100] (3)

[0028] Step 3: Test the scene using a video injection camera-in-the-loop testing platform. If the test result fails, invert the operation of each step as follows:

[0029] To quickly determine the edge test scenario search space ∑, this invention is based on reverse learning to rapidly explore the test scenario search space. For any continuous test scenario parameter Ω, there exists a corresponding reverse number. Specific reverse numbers The method for obtaining this result is shown in the following formula:

[0030]

[0031] However, this invention does not perform reverse learning on all test scenario parameters. When the initial test result is pass, the test scenario parameters that might cause the test result to fail should be automatically selected for reverse learning, and the test should be performed using the reverse-learned test scenario parameters. If the test result is still pass after performing reverse learning on all scenario parameters that might lead to different results, then it is necessary to return to the second step and select a new set of initial values ​​for the scenario parameters to enter the loop. If the test object fails the test scenario after the scenario parameters are updated, all Ω values ​​that change the test result and their inverse numbers are recorded. And The scenario constitutes a reverse testing scenario. Will The corresponding ∏ is determined as the edge test scenario search space ∑; ∑ may be a space of different dimensions, depending on the number of Ω that cause the test object to fail due to reverse learning. If other parameters are included that cause the test results to change, then ∑ increases the dimension of the corresponding number of parameters.

[0032] Step 4: The above steps have determined the search space ∑ for the edge scene, in order to determine the specific edge test scene ∏ tarThe search needs to be performed within ∑. This invention proposes a test scenario parameter selection method based on a greedy learning algorithm. The method uniformly samples the test scenario parameters within the corresponding parameter range to generate a set of combined test cases. The greedy algorithm is implemented through in-parameter-order expansion, and Π... tar The search process is viewed as a test case set whose scenario parameters are progressively expanded. First, a test case set with 100% coverage is generated for a small number of test scenario parameters. Then, new scenario parameters are gradually added, while the original test case set is expanded and modified to cover the newly added factors and all their related combinations. The operation steps include:

[0033] (1) Select any two factors to generate a combined test case set, which contains all possible combinations of the two factors, and the combinations constitute all the current paired sets;

[0034] (2) Expand in the horizontal direction, that is, add another factor and select a new value from it to ensure that the most scenario factors are paired value combinations.

[0035] (3) If there are still uncovered pairs of combinations in the horizontal expansion, then expand in the vertical direction to generate a new set of test cases;

[0036] Step 5: The stopping conditions for the test include: the number of iterations meets the test requirements or the output Π. tar If the number of test cases meets the requirements, the loop will exit once either condition is met, and the final edge test scenario library will be output.

[0037] (iv) Evaluation of test results

[0038] (1) Evaluation of the impact of scene parameters

[0039] During the test, the collision accident is considered to occur when the collision detector or target object in the virtual simulation software overlaps with the bounding box of the main vehicle. After the test scenario parameters are designed, the system under test is tested in the video injection camera loop test platform.

[0040] To evaluate the influence of different scenario parameters on the collision experiment results, this invention uses chi-square analysis to analyze the experimental results and then evaluate the influence of a single scenario parameter on the experimental results.

[0041] Chi-square analysis is a method used to compare whether there is a correlation between two or more groups of categorical variables; it is also known as the chi-square test. First, it assumes that the variables are independent and uncorrelated, obtaining a set of ideal data, defined as the null hypothesis, where the values ​​of each group are called the expected frequencies T. i The actual values ​​of each group are called the observation frequency A. iThe chi-square method is as follows:

[0042]

[0043] Chi-square analysis reflects the magnitude of the difference between expected and observed frequencies. A larger chi-square value indicates a weaker null hypothesis, meaning a stronger correlation exists among the selected groups of variables, suggesting that scene factors have a significant impact on the collision results. To quantify this correlation, a chi-square distribution function is introduced:

[0044]

[0045] Where Γ is the Gamma distribution function; n is the degrees of freedom of the chi-square distribution, for data with feature dimension (c*d):

[0046] n=(a-1)×(b-1) (7)

[0047] When applying chi-square analysis, the method of comparing the critical values ​​of the chi-square distribution is usually adopted.

[0048] Chi-square analysis can reveal the relationship between a single scene parameter and the collision result. However, to evaluate the impact of multiple scene parameters on the collision result, this invention combines weather category factors with other scene factors and uses a two-way ANOVA to analyze the interaction effect on the collision situation. Here, the interaction effect refers to the influence of one independent variable on the dependent variable across different values ​​of the other independent variable. The two-way ANOVA is a statistical analysis method used to analyze whether different levels of two factors have a significant impact on the result and whether there is an interaction effect between the two factors. Assuming that there are r and s level levels for two scene parameters A and B respectively, and t experiments are repeated for each level combination, the experimental result x is obtained. ijk Then, calculate the total sum of squared deviations (SSAB), the sum of squared deviations of errors (SSE), and the degrees of freedom (d) in sequence. f The mean square and MS values, the test value F, and finally the values ​​are compared with the critical values ​​of the F-distribution table to determine the degree of influence.

[0049] The sum of squared deviations reflects the discreteness of the interaction effect or random error between scene parameters A and B, and is calculated as follows:

[0050]

[0051]

[0052] in,

[0053]

[0054]

[0055]

[0056]

[0057] The degrees of freedom include the degrees of freedom df of the total effect after the interaction of A and B. AB Degrees of freedom d of the error fE :

[0058] df AB =(r-1)(s-1) (14)

[0059] MSE = SSE / df E =SSE / rs(t-1) (15)

[0060] The final test value F is obtained as follows:

[0061] F = MSAB / MSE (16)

[0062] The F-distribution critical value table (α = 0.1) is used to evaluate two scene elements, namely, the test value F under the corresponding degrees of freedom when the confidence level of the interaction between scene elements A and B having an impact on the result is 90%.

[0063] (2) Static evaluation of target detection algorithm

[0064] Target recognition is a commonly used function of cameras. This invention describes the evaluation index corresponding to this function and uses specific values ​​as the basis for updating scene parameters.

[0065] For object recognition, the camera output object classification results are categorized into four types: TruePositives (TP), where positive samples are correctly identified as positive samples; True Negatives (TN), where negative samples are correctly identified as negative samples; False Positives (FP), where negative samples are incorrectly identified as positive samples; and False Negatives (FN), where positive samples are incorrectly identified as negative samples. TP, TN, and FN are classified using the Intersection over Union (IOU), which is the ratio of the overlap area between the bounding box output by the target recognition algorithm and the minimum bounding box (GT) of the true target to the area of ​​their union. The number of detection boxes with an IOU greater than 0.5 is denoted as TP; the number of detection boxes with an IOU less than or equal to 0.5, or redundant detection boxes that detect the same GT, is defined as FP; the detection accuracy Pre can be expressed by the following formula:

[0066]

[0067] Recall (Rel) can be expressed as the proportion of correctly identified positive samples out of the total number of positive samples:

[0068]

[0069] When the confidence score of each bounding box is used as a threshold to determine whether a sample is positive or negative, the detection precision and recall will be equal to the number of samples, allowing us to plot a Precision-Recall (PR) curve. Ideally, if a network can identify every positive sample with a confidence score of 1 and every negative sample with a confidence score of 0, then the PR curve will only pass through the points (0,1) and (1,1), where the area enclosed by the PR curve and the coordinate axis is 1. In other cases, the area enclosed by the PR curve and the coordinate axis should be greater than 0. The area enclosed by the PR curve and the coordinate axis represents the performance of the object detection network, i.e., mAP, which is expressed as:

[0070]

[0071] (3) Dynamic evaluation of target detection algorithms

[0072] This invention proposes an initial detection distance d min Detection accuracy d with minimum safe distance p As a dynamic evaluation index for object detection algorithms.

[0073] The initial detection distance refers to the actual distance between the target object and the current location when the target detection algorithm first accurately identifies the target object during the execution of a test case. The initial detection distance is d. min A larger value indicates that the target detection algorithm detects the target object earlier, which is more conducive to the intelligent vehicle system performing control operations as early as possible. Minimum safe distance detection accuracy d p This refers to the absolute value of the difference between the distance output by the ranging module of the target detection algorithm and the actual distance when the actual distance between the target object and the workshop is the minimum safe distance at the current speed, during the execution of a test case. The smaller the minimum safe distance detection accuracy, the closer the distance estimated by the ranging module is to the actual distance between the target object and the workshop, which is more conducive to the intelligent vehicle braking according to the preset trajectory.

[0074] The beneficial effects of this invention are:

[0075] This invention provides a video injection-based multi-camera-in-the-loop testing method for intelligent vehicles. Targeting data from four cameras, it injects virtual simulation image data into the ECU under test via a video injection board and a fault injection board. Furthermore, addressing the challenge of an infinitely rich set of parameters affecting camera test results, this invention proposes an edge test scene search method based on an improved greedy algorithm. This method accelerates the construction of an edge test scene library while ensuring 100% parameter coverage. The constructed edge test scene library is then evaluated, outputting scene parameter impact evaluation, static evaluation of the target detection algorithm, and dynamic evaluation methods. Attached Figure Description

[0076] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0077] Figure 2 This is a hardware schematic diagram of the video injection camera-in-the-loop test platform of the present invention;

[0078] Figure 3 This is a schematic diagram of the hardware connection method of the present invention;

[0079] Figure 4 This is a schematic diagram illustrating the generation of random chaotic sequences according to the present invention. Detailed Implementation

[0080] like Figure 1 As shown, this invention provides a video injection-based multi-camera loop testing method for intelligent vehicles. The hardware composition of the video injection camera loop testing platform is as follows: Figure 2 As shown, the system includes a host computer, the ECU under test (DUT) hardware, a video injection board, and a fault injection board. Test platform initialization includes two aspects: parameter selection for the camera under test and initialization parameter selection for the test scenario. The host computer is used for virtual simulation scenario construction and vehicle dynamics model deployment; the DUT hardware deploys the camera's perception algorithms, including target detection and target tracking algorithms; the video injection board converts the four-channel camera image data output from the host computer from the HDMI interface to the GMSL interface; the fault injection board performs noise simulation on the four-channel image data, simulating both global and local noise; the specific hardware connection methods are as follows... Figure 3As shown, the host computer first outputs four-channel vehicle video signals rendered by virtual simulation scene software from the HDMI interface on the host side. The HDMI video signal format is then converted to GMSL video format by the Camera Injection Module (CIM) and output to the fault injection board for video display. Global noise and color blocks are simulated to test the recognition and tracking performance of the vehicle camera under conditions of poor contact or lens damage. Afterwards, the video data is output from the serializer and input to the external anti-serializer of the ECU under test via the GMSL interface for testing and verification of the target recognition and tracking algorithm in the ECU. After target recognition and tracking, the ECU returns the target recognition class, accuracy, and the four-point coordinate parameters corresponding to the BoundingBox as dictionary values ​​to the ADAS model in the ECU. The dictionary keys are the corresponding timestamps. The virtual vehicle in the host computer receives the results from the ECU and outputs control signals through the vehicle dynamics model to ensure real-time position updates of the vehicle in the virtual simulation scene.

[0081] This invention includes the following steps:

[0082] (I) Selection of Test Object Parameters

[0083] First, select the parameters of the test object, namely the parameters of the test camera. Before each test, the camera parameters need to be set according to the test requirements and the camera under test is initialized on the camera-in-the-loop test platform. The camera parameters remain unchanged during scene updates after selection.

[0084] The parameters of the test object mainly include two aspects: camera installation parameters and camera performance parameters. Camera installation parameters include installation position and installation angle, while camera performance parameters include camera focal length and camera sensor size.

[0085] The camera's installation position refers to the translation coordinates with the vehicle's center of gravity as the origin, the forward direction of the vehicle as the x-axis, the leftward direction as the y-axis, and the vertical upward direction as the z-axis (x...). a ,y a ,z a The subscript 'a' represents the corresponding camera number. This invention is tested on a smart car equipped with four cameras, therefore a = 1, 2, 3, 4; the camera mounting angle refers to the rotation angle (σ) around the x, y, and z axes respectively. a ,β a ,γ a ), a = 1, 2, 3, 4; the camera focal length f refers to the lens focal length, that is, the distance from the rear principal point of the lens to the focal point, in millimeters; the camera sensor size (X a ,Y a) refers to the physical size of the CMOS image sensor to be simulated, in millimeters, where a = 1, 2, 3, 4.

[0086] (II) Test Scenario Design

[0087] The test scenario selected in this invention is designed based on the EURO-NCAP-AEB-2023 test procedure. Since this test procedure is designed for closed-site testing, while virtual simulation testing can expand the boundaries of scenario parameters at no cost, this invention selects a parameter space with a larger parameter range to test the algorithm under test when designing the test scenario.

[0088] Table 1 Scene Parameter Table

[0089]

[0090]

[0091] As shown in Table 1, the intelligent vehicle camera sensor is a passive sensor, which is significantly affected by the external environment. There are many factors that affect the camera sensor in the scene. The scene parameters of this invention include the vehicle's motion state, weather conditions, and target objects.

[0092] (1) The motion state of the vehicle mentioned includes the vehicle speed; the speed v of the vehicle in this invention vut The value ranges from 10 to 80 km / h;

[0093] (2) The meteorological categories mentioned include lighting conditions and complex weather: lighting conditions mainly describe the position of the sun source in the scene, and are described by azimuth angle α and polar angle θ; complex weather parameters include precipitation weather, fog weather and snowfall weather;

[0094] The meteorological data for rainfall use rainfall intensity μ (unit: mm / h) and raindrop diameter D. rain (Unit: mm) is used for description;

[0095]

[0096] The above formula can be used to set the corresponding rainfall scene parameters in virtual simulation software;

[0097] The aforementioned fog weather utilizes visibility D fog (Unit: m) is used for description;

[0098] The snowfall meteorology described above utilizes the snowflake diameter D snow (Unit: millimeters) is used for description.

[0099] (3) The target object class includes the target object category, target object running direction, target motion speed, and collision position;

[0100] The target object category T includes pedestrians or vehicles being tested; the target object's direction of travel Dri includes, for pedestrians, distal, proximal, and longitudinal; for vehicles, it includes traveling in the same direction as the vehicle or crossing it; the target's speed v tar For pedestrians, a speed of 5 to 8 km / h is used; for vehicles, a speed of 20 km / h and 50 km / h are used, accompanied by -2 to -6 m / s. 2 The braking deceleration; the collision position P refers to the position where the vehicle under test and the target object collide in front of the vehicle under test at their own speeds. For pedestrians, P is 25% to 75%, and for vehicles, P is -50% to 50%.

[0101] Each test scenario is described by an array of scenario parameters, specifically expressed as: Π=[v vut ,α,θ,μ,a,D fog D snow ,T,Dri,v tar ,P].

[0102] (III) Edge Scene Extraction

[0103] Except for the target object category T and the target object's running direction Dri, all other test scene parameters are continuous variables. Using a traversal testing method would require significant computational resources. This invention addresses this by using the edge testing scene Π. tar It is defined as a scenario in which there exists at least one test scenario parameter that satisfies the condition that a change in this parameter at a certain threshold will cause a significant change in the behavior or decision of the object under test. The threshold for the change is set according to the test accuracy and test requirements.

[0104] Edge testing scenario Π tar The screening steps are as follows:

[0105] Step 1: Initialize the parameters of the camera under test and the values ​​of discontinuous variables in the test scene;

[0106] Step 2: Chaos exhibits a certain degree of randomness and sensitivity to initial conditions, such as... Figure 4 As shown, this invention enables the algorithm to have a faster convergence speed. It utilizes a Sine chaotic mapping to generate a random chaotic sequence to initialize the parameters of the continuous test scenario in test scenario Π. The specific expression is as follows:

[0107] x i+1 =δsin(πx) i (2)

[0108] In the formula, δ is the system parameter, δ∈[0,1], and chaos occurs when δ∈(0.87,0.93)∪(0.95,1); x i ,x i+1 This belongs to the iterative sequence value; x0∈(0,1), and in this embodiment, the number of experiments i is chosen to be 100. Since the output of this chaotic mapping is a value in (0,1), it needs to be denormalized. Specifically, the maximum value Ω of the scene parameter is taken. min and minimum value Ω max Initialize the test scenario parameter Ω using denormalization:

[0109] Ω=Ω min +(Ω max -Ω min )*x i ,i∈[0,100] (3)

[0110] Step 3: Test the scenario using a video-injected camera-in-the-loop testing platform;

[0111] Table 2. Edge Testing Scene Selection Method

[0112]

[0113] Table 2 shows how to search for edge scenes when the test result is passed, and how to perform the reverse operation at each step when the test result is failed, as follows:

[0114] To quickly determine the edge test scenario search space Σ, this invention is based on reverse learning to rapidly explore the test scenario search space. For any continuous test scenario parameter Ω, there exists a corresponding reverse number. Specific reverse numbers The method for obtaining this result is shown in the following formula:

[0115]

[0116] However, this invention does not perform reverse learning on all test scenario parameters. When the initial test result is pass, the test scenario parameters that might cause the test result to fail should be automatically selected for reverse learning, and the test should be performed using the reverse-learned test scenario parameters. If the test result is still pass after performing reverse learning on all scenario parameters that might lead to different results, then it is necessary to return to the second step and select a new set of initial values ​​for the scenario parameters to enter the loop. If the test object fails the test scenario after the scenario parameters are updated, all Ω values ​​that change the test result and their inverse numbers are recorded. And The scenario constitutes a reverse testing scenario. Will And its corresponding Π is determined as the edge test scenario search space Σ; Σ may be a space of different dimensions, depending on the number of Ω that cause the test object to fail due to back learning, for example, when v vut If the initial value is 30km / h and the test result is passed, then choosing its reverse value of 60km / h will result in a failed test. Therefore, v will be... vut In the search space Σ within [30,60], which serves as the edge testing scenario, if only v vut If the test results change due to reverse learning, then Σ is a one-dimensional space. If other parameters are included that cause the test results to change, then Σ increases in dimension by the number of parameters.

[0117] Step 4: The above steps have determined the search space Σ for the edge scene, in order to determine the specific edge test scene Π tar The search needs to be performed within Σ. This invention proposes a test scenario parameter selection method based on a greedy learning algorithm, which uniformly samples the test scenario parameters within the corresponding parameter range to generate a set of combined test cases. This invention implements the greedy algorithm through an in-parameter-order expansion method, which expands Π... tar The search process is viewed as a test case set whose scenario parameters are gradually expanded. First, a test case set with 100% coverage is generated for a small number of test scenario parameters. Then, new scenario parameters are gradually added, while the original test case set is expanded and modified to cover the newly added factors and all their related combinations.

[0118] Assuming the dimension of Σ is 3, taking the three-dimensional test scene parameters in Σ as an example, let them be A, B, and C, and assume they have 2, 2, and 3 value ranges respectively. The method proposed in this paper is used for combined testing. If the value range is increased, the method proposed in this invention can also be used for testing. The basic operation steps are as follows:

[0119] (1) Select any two factors to generate a set of combined test cases, which includes all possible combinations of values ​​for the two factors. These combinations form the current set of all pairs:

[0120] Table 3. Paired sets of two factors

[0121]

[0122]

[0123] (2) Expand in the horizontal direction, that is, add another factor and select a new value from it to ensure that the most scenario factors are paired value combinations.

[0124] As shown in Table 4, another factor C is added; three test cases are used to cover all pairs of combinations, as shown in Table 5, where (A1, C2), (A2, C1), (A2, C3), (B1, C3), (B2, C1), and (B2, C2) are the uncovered pairs of combinations; the missing value in Table 4 is not determined, so C1, C2, and C3 are substituted to determine the coverage of the corresponding test cases with the uncovered parts in Table 5. It can be seen that C1 covers two test cases, while C2 and C3 each cover one test case, so the missing value is C1.

[0125] Table 4 Horizontal Expansion

[0126] A B C <![CDATA[A1]]> <![CDATA[B1]]> <![CDATA[C1]]> <![CDATA[A2]]> <![CDATA[B1]]> <![CDATA[C2]]> <![CDATA[A1]]> <![CDATA[B2]]> <![CDATA[C3]]> <![CDATA[A2]]> <![CDATA[B2]]>

[0127] Table 5: Combinations Not Covered

[0128]

[0129] (3) If there are still uncovered pairs in the horizontal expansion, then expand in the vertical direction. Four test cases are obtained by filling the gaps in Table 4. At this time, Table 5 is updated. The uncovered pairs are shown in Table 6. These four pairs are expanded vertically to obtain the results in Table 7.

[0130] Table 6. Pair combinations still uncovered after horizontal expansion

[0131]

[0132]

[0133] Table 7 shows the vertical combinations after pairing.

[0134]

[0135] After the above steps, the final set of test cases obtained is as follows:

[0136] Table 8 Final Test Case Set

[0137] A B C 1 <![CDATA[A1]]> <![CDATA[B1]]> <![CDATA[C1]]> 2 <![CDATA[A2]]> <![CDATA[B1]]> <![CDATA[C2]]> 3 <![CDATA[A1]]> <![CDATA[B2]]> <![CDATA[C3]]> 4 <![CDATA[A2]]> <![CDATA[B2]]> <![CDATA[C1]]> 5 <![CDATA[A1]]> <![CDATA[B2]]> <![CDATA[C2]]> 6 <![CDATA[A2]]> <![CDATA[B1]]> <![CDATA[C3]]>

[0138] This method ensures full test case coverage while enabling rapid searching of Π. tar This leads to the construction of an edge testing scenario library.

[0139] Step 5: The stopping condition for the test is determined by two factors: firstly, the number of iterations meets the test requirements; secondly, the output Π... tar If the number of elements meets the test requirements, the loop will exit once either condition is met, and the final edge test scenario library will be output.

[0140] (iv) Evaluation of test results

[0141] (1) Evaluation of the impact of scene parameters

[0142] During the test, an overlap between the collision detector or target object in the virtual simulation software and the bounding box of the main vehicle is considered a collision accident. After designing the test scenario parameters, the system under test is tested in a video-injected camera-in-the-loop test platform. To evaluate the influence of different scenario parameters on the collision test results, this invention uses chi-square analysis to analyze the experimental results and thus evaluate the influence of a single scenario parameter on the experimental results. Chi-square analysis is a method used to compare whether there is a correlation between two or more groups of categorical variables, also known as the chi-square test. First, it is assumed that the variables are independent and uncorrelated, resulting in a set of ideal data defined as the null hypothesis, where the values ​​of each group are called the expected frequencies T. i The actual values ​​of each group are called the observation frequency A. i The chi-square method is as follows:

[0143]

[0144] Chi-square analysis reflects the magnitude of the difference between expected and observed frequencies. A larger chi-square value indicates a weaker null hypothesis, meaning a stronger correlation exists among the selected groups of variables, suggesting that scene factors have a significant impact on collision results. To quantify this correlation, a chi-square distribution function is introduced:

[0145]

[0146] Where Γ is the Gamma distribution function; n is the degrees of freedom of the chi-square distribution, for data with feature dimension (c*d):

[0147] n=(a-1)×(b-1) (7)

[0148] When applying chi-square analysis, the method of comparing the critical values ​​of the chi-square distribution is usually adopted.

[0149] Chi-square analysis can reveal the relationship between a single scene parameter and the collision result. However, to evaluate the impact of multiple scene parameters on the collision result, this invention combines weather category factors with other scene factors and uses a two-way ANOVA to analyze the interaction effect on the collision situation. Here, the interaction effect refers to the influence of one independent variable on the dependent variable across different values ​​of the other independent variable. The two-way ANOVA is a statistical analysis method used to analyze whether different levels of two factors have a significant impact on the result and whether there is an interaction effect between the two factors. Assuming that there are r and s level levels for two scene parameters A and B respectively, and t experiments are repeated for each level combination, the experimental result x is obtained.ijk Then, calculate the total sum of squared deviations (SSAB), the sum of squared deviations of errors (SSE), and the degrees of freedom (d) in sequence. f The mean square and MS values, the test value F, and finally the values ​​are compared with the critical values ​​of the F-distribution table to determine the degree of influence.

[0150] The sum of squared deviations reflects the discreteness of the interaction effect or random error between scene parameters A and B, and is calculated as follows:

[0151]

[0152]

[0153] in,

[0154]

[0155]

[0156]

[0157]

[0158] The degrees of freedom include the degrees of freedom df of the total effect after the interaction of A and B. AB Degrees of freedom d of the error fE :

[0159] df AB =(r-1)(s-1) (14)

[0160] MSE = SSE / df E =SSE / rs(t-1) (15)

[0161] The final test value F is obtained as follows:

[0162] F = MSAB / MSE (16)

[0163] The F-distribution critical value table (α = 0.1) is used to evaluate the two scene elements, that is, when the confidence level of the interaction between scene elements A and B having an impact on the result is 90%, the test value F is the size of the corresponding degrees of freedom.

[0164] (2) Static evaluation of target detection algorithm

[0165] The evaluation of collision results is crucial for determining the direction of scene parameter updates and constructing an edge scene library. However, in practical applications, evaluation metrics for cameras should not be limited to collision results alone. While collision results can indicate the "quality" of an algorithm to some extent, this evaluation metric clearly cannot provide guidance or reference for regression testing and algorithm improvement. Target recognition is a commonly used function of cameras. This invention describes the corresponding evaluation metrics for this function and uses specific values ​​as the basis for scene parameter updates.

[0166] For object recognition, the camera output object classification results are categorized into four types: TruePositives (TP), where positive samples are correctly identified as positive samples; True Negatives (TN), where negative samples are correctly identified as negative samples; False Positives (FP), where negative samples are incorrectly identified as positive samples; and False Negatives (FN), where positive samples are incorrectly identified as negative samples. TP, TN, and FN are classified using the Intersection over Union (IOU), which is the ratio of the overlap area between the bounding box output by the target recognition algorithm and the minimum bounding box (GT) of the true target to the area of ​​their union. The number of detection boxes with an IOU greater than 0.5 is denoted as TP; the number of detection boxes with an IOU less than or equal to 0.5, or redundant detection boxes that detect the same GT, is defined as FP; the detection accuracy Pre can be expressed by the following formula:

[0167]

[0168] Recall (Rel) can be expressed as the proportion of correctly identified positive samples out of the total number of positive samples:

[0169]

[0170] When the confidence score of each bounding box is used as a threshold to determine whether a sample is positive or negative, the detection precision and recall will be equal to the number of samples, allowing us to plot a Precision-Recall (PR) curve. Ideally, if a network can identify every positive sample with a confidence score of 1 and every negative sample with a confidence score of 0, then the PR curve will only pass through the points (0,1) and (1,1), where the area enclosed by the PR curve and the coordinate axis is 1. In other cases, the area enclosed by the PR curve and the coordinate axis should be greater than 0. The area enclosed by the PR curve and the coordinate axis represents the performance of the object detection network, i.e., mAP, which is expressed as:

[0171]

[0172] (3) Dynamic evaluation of target detection algorithms

[0173] Static metrics can evaluate the basic performance of an algorithm under test. When the algorithm under test works in conjunction with vehicle dynamics models and control algorithms on a test platform, dynamic metrics need to be developed based on the functions of the intelligent vehicle to evaluate the algorithm's performance in specific test scenarios. This invention proposes an initial detection distance d. min Detection accuracy d with minimum safe distance p As a dynamic evaluation index for object detection algorithms.

[0174] The initial detection distance refers to the actual distance between the target object and the current location when the target detection algorithm first accurately identifies the target object during the execution of a test case. The initial detection distance is d. min A larger value indicates that the target detection algorithm detects the target object earlier, which is more conducive to the intelligent vehicle system performing control operations as early as possible. Minimum safe distance detection accuracy d p This refers to the absolute value of the difference between the distance output by the ranging module of the target detection algorithm and the actual distance when the actual distance between the target object and the workshop is the minimum safe distance at the current speed, during the execution of a test case. The smaller the minimum safe distance detection accuracy, the closer the distance estimated by the ranging module is to the actual distance between the target object and the workshop, which is more conducive to the intelligent vehicle braking according to the preset trajectory.

Claims

1. A method for loop-in-the-loop testing of multi-channel cameras in intelligent vehicles based on video injection, characterized in that: Includes the following steps: (a) Selection of test object parameters First, select the parameters of the test object, that is, the parameters of the test camera. The parameters of the test object mainly include two aspects: camera installation parameters and camera performance parameters. Camera installation parameters include installation position and installation angle, and camera performance parameters include camera focal length and camera sensor size. (II) Test Scenario Design The test scenario parameters include the vehicle's motion state, weather conditions, and target object types; (1) The motion state of the vehicle mentioned includes the vehicle speed. ; (2) The meteorological categories mentioned include lighting conditions and complex weather: lighting conditions mainly describe the position of the sun's light source in the scene, using azimuth angles. With polar angle Describe the complex meteorological parameters, including precipitation, fog, and snowfall. The rainfall intensity mentioned above is used in meteorological terms. and raindrop diameter Describe it; The above formula is used to set the corresponding rainfall scene parameters in the virtual simulation software; The aforementioned fog weather utilizes visibility Describe it; The snowfall meteorology described uses snowflake diameter Describe it; (3) The target object class includes the target object category. Target object's direction of movement Target speed and collision location ; Each test scenario is described by an array of scenario parameters. Specifically, it can be expressed as follows: ; (III) Edge Scene Extraction Edge testing scenarios Defined as at least one test scenario parameter in the scenario, such that a change in this parameter to a certain threshold will cause a significant change in the behavior or decision of the object under test. The threshold for the change is set according to the test accuracy and test requirements. Edge testing scenarios The screening steps are as follows: Step 1: Initialize the parameters of the camera under test and the values ​​of discontinuous variables in the test scene; Step 2: Generate random chaotic sequences using Sine chaotic mapping for the test scenario. The parameters for the continuous test scenario are initialized using the following expression: In the formula, These are system parameters. ,when Chaos may occur at times; It belongs to the iterative sequence value; Because the output of this chaotic mapping is The value is such that it needs to be denormalized, specifically by taking the minimum value of the scene parameter. and maximum value Initialize test scenario parameters through denormalization : Step 3: Test the scene using a video injection camera-in-the-loop testing platform. If the test result fails, invert the operation of each step as follows: Based on reverse learning, the test scenario search space can be quickly explored for any continuous test scenario parameters. There are corresponding reverse numbers for each. Specific reverse numbers The method for obtaining this result is shown in the following formula: When the initial test result is passed, the test scenario parameters that may cause the test result to fail should be automatically selected for reverse learning, and the test should be performed using the reverse-learned test scenario parameters; if the test result is still passed after all scenario parameters that may cause different results have been reverse-learned, then it is necessary to return to the second step and select a new set of initial values ​​for scenario parameters to enter the loop. If the test object fails the test scenario after the scenario parameters are updated, record all changes to the test results. and its reverse number and will The scenario constitutes a reverse testing scenario. ,Will and its corresponding Determined as the search space for edge testing scenarios ; It could be a space of different dimensions, depending on whether the test object fails due to back learning. The number of items, if other parameters cause the test results to change, then Increase the dimension corresponding to the number of parameters; Step 4: Using a test scenario parameter selection method based on a greedy learning algorithm, test scenario parameters are uniformly sampled within the corresponding parameter range to generate a set of combined test cases. The greedy algorithm is implemented through a factor-by-factor expansion approach. The search process is viewed as a test case set whose scenario parameters are gradually expanded. First, a test case set with 100% coverage is generated for a small number of test scenario parameters. Then, new scenario parameters are gradually added, while the original test case set is expanded and modified to cover the newly added factors and all related combinations. Step 5: The stopping conditions for the test include: the number of iterations meets the test requirements, and the output... If the number of test cases meets the requirements, the loop will break out and the final edge test scenario library will be output. (iv) Evaluation of test results (1) Evaluation of the impact of scene parameters Chi-square analysis was used to analyze the experimental results and evaluate the influence of a single scene parameter on the experimental results; the relationship between the single scene parameter and the collision results was obtained through chi-square analysis of the test results. The interaction effect of weather category factors combined with other scene factors was analyzed using a two-way ANOVA method. The interaction effect refers to the influence of one independent variable on the dependent variable on the other independent variable at different values. (2) Static evaluation of target detection algorithm Target recognition is a commonly used function of cameras. The evaluation index corresponding to this function is described, and specific values ​​are used as the basis for updating scene parameters. (3) Dynamic evaluation of target detection algorithm Using the first detection distance Detection accuracy with minimum safe distance As a dynamic evaluation index for object detection algorithms; First-detection distance refers to the actual distance between the target object and the work area when the target detection algorithm first accurately identifies the target object ahead during the execution of a test case; first-detection distance A larger value indicates that the target detection algorithm detects the target object earlier, which is more conducive to the intelligent vehicle system performing control operations as early as possible; minimum safe distance detection accuracy This refers to the absolute value of the difference between the distance output by the ranging module of the target detection algorithm and the actual distance when the actual distance between the target object and the workshop is the minimum safe distance at the current speed during the execution of a test case. The smaller the minimum safe distance detection accuracy, the closer the distance estimated by the ranging module is to the actual distance between the target object and the workshop, which is more conducive to the intelligent vehicle braking according to the preset trajectory.

2. The method for loop-in testing of multi-channel cameras in intelligent vehicles based on video injection according to claim 1, characterized in that: In step (1) of selecting test object parameters, the parameters of the test object mainly include two aspects, namely camera installation parameters and camera performance parameters; camera installation parameters include installation position and installation angle, and camera performance parameters include camera focal length and camera sensor size; The camera's installation position refers to the translation coordinate system with the vehicle's center of gravity as the origin, the forward direction of the vehicle as the x-axis, the leftward direction as the y-axis, and the vertical upward direction as the z-axis. The subscript 'a' represents the corresponding camera number; the camera's mounting angle refers to the rotation angle around the x, y, and z axes, respectively. Camera focal length This refers to the focal length of the lens, which is the distance from the rear principal point of the lens to the focal point, measured in millimeters; the size of the camera's image sensor. This refers to the physical size of the CMOS image sensor that needs to be simulated, measured in millimeters.

3. The method for loop-in-the-loop testing of multi-channel cameras in intelligent vehicles based on video injection according to claim 1, characterized in that: In step (two) of the test scenario design, the target object category is... This includes pedestrians and vehicles; the direction of movement of the target object. For pedestrians, this includes distal, proximal, and longitudinal travel; for vehicles, it includes vehicles traveling in the same direction as or intersecting with the vehicle; the target speed of movement... For pedestrians, a speed of 5 to 8 km / h is used; for vehicles, a speed of 20 km / h and 50 km / h are used simultaneously. Braking deceleration; the collision position This refers to the point where the vehicle under test and the target object, both traveling at their own speeds, ultimately collide in front of the vehicle under test. For a pedestrian... Values For vehicles, Values .

4. The method for loop-in testing of multi-channel cameras in intelligent vehicles based on video injection according to claim 1, characterized in that: In step (three) edge scene extraction, the fourth step includes the following steps: (1) Select any two factors to generate a set of combined test cases, which includes all possible combinations of the two factors. These combinations constitute all the current set of pairs. (2) Expand in the horizontal direction, that is, add another factor and select a new value from it to ensure that the most scenario factors are paired value combinations; (3) If there are still uncovered pairs of combinations in the horizontal expansion, then expand in the vertical direction to generate a new set of test cases.

5. The method for loop-in testing of multi-channel cameras in intelligent vehicles based on video injection according to claim 1, characterized in that: In step (iv) of the test result evaluation, the chi-square analysis is a method used to compare whether there is a correlation between two or more groups of categorical variables, also known as the chi-square test. First, it is assumed that the variables are independent and uncorrelated, and a set of ideal data is obtained, which is defined as the null hypothesis. The values ​​of each group are called the expected frequencies. The actual values ​​of each group are called the observation frequencies. The chi-square method is as follows: Chi-square analysis reflects the magnitude of the difference between expected and observed frequencies. A larger chi-square value indicates a less valid null hypothesis, meaning a stronger correlation exists among the selected groups of variables, suggesting that scene factors have a significant impact on collision results. To quantify this correlation, a chi-square distribution function is introduced: in, is the Gamma distribution function; n is the chi-square distribution's degrees of freedom for data with feature dimension (c*d): When applying chi-square analysis, the method of comparing the critical values ​​of the chi-square distribution is usually adopted. The two-way ANOVA method assumes that there are r and s level levels for parameters A and B in two scenarios, respectively, and that the experiment is repeated t times for each level combination to obtain the experimental results. Then, calculate the total sum of squared deviations (SSAB), the sum of squared deviations (SSE), and the degrees of freedom in sequence. The mean square and MS, the test value F, are then compared with the critical value of the F-distribution table to determine the degree of influence. The sum of squared deviations reflects the discreteness of the interaction effect or random error between scene parameters A and B, and is calculated as follows: in, Degrees of freedom include the degrees of freedom of the total effect after the interaction of A and B. Degrees of freedom of sum and error : The final test value F is obtained as follows: The F-distribution critical value table is used to evaluate two scene elements, namely, the test value F under the corresponding degrees of freedom when the confidence level of the interaction between scene elements A and B having an impact on the result is 90%.

6. The method for loop-in testing of multi-channel cameras in intelligent vehicles based on video injection according to claim 1, characterized in that: In step (iv) of the test result evaluation, the target recognition output by the camera has four types of target recognition classification results: True Positives (TP), that is, positive samples are correctly identified as positive samples; True Negatives (TN), that is, negative samples are correctly identified as negative samples; False Positives (FP), that is, negative samples are incorrectly identified as positive samples; and False Negatives (FN), that is, positive samples are incorrectly identified as negative samples. TP, TN, and FN are classified by the value of IOU. IOU refers to the ratio of the overlap area between the bounding box output by the target recognition algorithm and the minimum bounding box GT of the real target to the area of ​​their union. The number of detection boxes with IOU greater than 0.5 is denoted as TP; the number of detection boxes with IOU <= 0.5 or redundant detection boxes that detect the same GT is defined as FP; the detection accuracy Pre is expressed by the following formula: Recall (Rel) can be expressed as the proportion of correctly identified positive samples out of the total number of positive samples: When the confidence level of each bounding box is used as a threshold to determine whether a sample is positive or negative, the detection precision and recall will be equal to the number of samples, allowing the plotting of a Precision-Recall (PR) curve. Ideally, if a network can identify each positive sample with a confidence level of 1 and each negative sample with a confidence level of 0, then the PR curve will only pass through the points (0,1) and (1,1), where the area enclosed by the PR curve and the coordinate axis is 1. In other cases, the area enclosed by the PR curve and the coordinate axis should be greater than 0. The area enclosed by the PR curve and the coordinate axis represents the performance of the object detection network, i.e., mAP, which is expressed as: 。

Images