An automatic driving image processing evaluation method, related equipment, medium and product

By building a collection of test scenarios and models, the ISP processing effect in the autonomous driving system is evaluated, which solves the ISP evaluation problem and improves the system's safety and identification accuracy.

CN114743076BActive Publication Date: 2024-11-08TSINGHUA UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210429094.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-22
Publication Date
2024-11-08
Estimated Expiration
2042-04-22

AI Technical Summary

Technical Problem

The prior art lacks effective methods to evaluate the image signal processing effect of image signal processors (ISPs) in autonomous driving systems, making it difficult to guarantee the accuracy and safety of system identification.

Method used

By building multiple test scenarios and corresponding test models, ISP controls to process data of each test scenario, obtains processing results and enters the test model for evaluation, and finally objectively evaluates the ISP based on the evaluation results to ensure its reliability and security .

Benefits of technology

It provides an objective and accurate evaluation index to ensure the image processing effect of the ISP, improves the safety and identification accuracy of the autonomous driving system, and ensures that the system meets the qualified standards before it is put into use.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114743076B_ABST
    Figure CN114743076B_ABST
Patent Text Reader

Abstract

The present application discloses an autonomous driving image processing evaluation method, related equipment, medium and product. The method is applied to an autonomous driving device, which includes an ISP; the method includes: controlling the ISP to process data sets of m test scenes respectively, and obtaining m processing results corresponding to the m test scenes one by one; obtaining m test result sets corresponding to the m test scenes one by one, wherein each test model set includes at least one test model, each test result set includes at least one first test result, and one test model outputs a first test result; according to the m test result sets, determining m second test results corresponding to the m test scenes one by one; and evaluating the ISP according to the m second test results. The present application provides evaluation indicators for evaluating the ISP. In this way, the autonomous driving equipment will be put into use only after the ISP evaluation is passed, thereby improving the safety of autonomous driving.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of autonomous driving, and in particular to an autonomous driving image processing evaluation method, related equipment, media and products. Background Art

[0002] Autonomous driving technology is an important technology in the current motor vehicle field and is also the research direction that major manufacturers are focusing on.

[0003] The perception module of the autonomous driving system is the "eye" of the autonomous driving system. Whether it can work stably and reliably directly affects road traffic safety. The original image signals collected by the camera and lidar sensor equipped with the autonomous driving system need to be processed by the image signal processor (ISP) before they can be input into the perception module for image algorithm recognition and prediction.

[0004] ISP plays an extraordinary role as the pre-processing link of image signals. A good ISP can greatly improve the accuracy of system recognition and thus ensure the safe operation of the system. However, there is currently no relevant solution for how to evaluate the image signal processing effect of ISP. Summary of the invention

[0005] The embodiments of the present application provide an autonomous driving image processing evaluation method, related equipment, media, and products, which can propose solutions for the evaluation of the image signal processing effect of ISP and improve the safety of autonomous driving.

[0006] In a first aspect, an embodiment of the present application provides an autonomous driving image processing evaluation method, which is applied to an autonomous driving device, wherein the autonomous driving device includes an image signal processor ISP; the method includes:

[0007] Control the ISP to process the data sets of m test scenarios respectively to obtain m processing results corresponding to the m test scenarios one by one, where m is a positive integer;

[0008] The m processing results are respectively input into m test model sets corresponding to the m test scenarios one by one,

[0009] Obtain m test result sets corresponding one-to-one to the m test scenarios, wherein each test model set includes at least one test model, each of the test result sets includes at least one first test result, and one test model outputs one first test result;

[0010] Determine, according to the m test result sets, m second test results corresponding one-to-one to the m test scenarios;

[0011] The ISP is evaluated according to the m second test results.

[0012] In a second aspect, the present application provides an automatic driving image processing and evaluation device.

[0013] The automatic driving image processing and evaluation device comprises an image signal processor ISP; the device comprises:

[0014] A control module, used to control the ISP to process the data sets of m test scenarios respectively, to obtain m processing results corresponding to the m test scenarios one by one, where m is a positive integer;

[0015] The first acquisition module is used to input the m processing results into m test model sets corresponding to the m test scenarios one by one,

[0016] Obtain m test result sets corresponding one-to-one to the m test scenarios, wherein each test model set includes at least one test model, each of the test result sets includes at least one first test result, and one test model outputs one first test result;

[0017] A first determining module, configured to determine, according to the m test result sets, m second test results corresponding one-to-one to the m test scenarios;

[0018] An evaluation module is used to evaluate the ISP according to the m second test results.

[0019] In a third aspect, an embodiment of the present application provides an autonomous driving image processing and evaluation device, the device comprising: a processor and a memory storing program instructions; when the processor executes the program instructions, it implements the autonomous driving image processing and evaluation method as described in the first aspect.

[0020] In a fourth aspect, an embodiment of the present application provides a storage medium, wherein program instructions are stored on the readable storage medium, and when the program instructions are executed by a processor, the autonomous driving image processing and evaluation method as described in the first aspect is implemented.

[0021] In a fifth aspect, an embodiment of the present application provides a program product, characterized in that when the instructions in the program product are executed by a processor of an electronic device, the electronic device executes the autonomous driving image processing and evaluation method as described in the first aspect.

[0022] In an embodiment of the present application, m test scenarios for autonomous driving and m test model sets corresponding to the m test scenarios can be constructed. For each test scenario, the ISP of the autonomous driving device can be controlled to process its data set to obtain the processing results of the ISP for each test scenario; then, the processing results of the ISP for each test scenario are respectively input into the test model set corresponding to each test scenario to obtain the test result set of each test scenario; based on the test result set of each test scenario, the second test result of each test scenario is obtained, that is, m second test results. Then, the ISP is evaluated based on the m second test results. It can be seen that the embodiment of the present application provides an evaluation index for evaluating the ISP, that is, the second test result. In this way, if the ISP evaluation fails, the ISP can continue to be debugged until the ISP evaluation passes before the autonomous driving equipment is put into use, thereby improving the safety of autonomous driving. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 This is one of the flow charts of the autonomous driving image processing evaluation method provided in the embodiment of the present application;

[0025] Figure 2a It is an architecture diagram of image processing provided by an embodiment of the present application;

[0026] Figure 2b This is the second flowchart of the autonomous driving image processing evaluation method provided in the embodiment of the present application;

[0027] Figure 3 is a structural diagram of an autonomous driving image processing and evaluation device provided in an embodiment of the present application;

[0028] Figure 4 It is a structural diagram of the autonomous driving image processing evaluation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.

[0030] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0031] For ease of understanding, some contents involved in the embodiments of the present application are described below:

[0032] Cars equipped with automatic driving systems can greatly alleviate driver fatigue and reduce the occurrence of traffic accidents.

[0033] Compared with other unmanned systems (such as unmanned warehouses and unmanned supermarkets), the natural environment involved in the autonomous driving system is more changeable and the social environment is more complex. Complex scenes composed of multiple factors such as weather, traffic, pedestrians, vehicles, lane lines, road signs, traffic lights, etc. may pose challenges to the safety performance of autonomous driving.

[0034] The perception module of autonomous driving is the "eye" of the autonomous driving system. Whether it can work stably and reliably directly affects road traffic safety. Autonomous driving is usually equipped with multiple high-definition cameras and lidar sensors. The data extracted by these two types of hardware often need to be processed by ISP before being input into the perception module. The functions of ISP are relatively complex, and include 3A (automatic white balance, auto focus, automatic exposure), sharpening, denoising, color space conversion, color enhancement, skin color enhancement and other algorithms. An original image needs to be processed by ISP before it can be input into the perception module for image algorithm recognition and prediction.

[0035] At present, there are still a lot of special cases (corner cases) that are difficult to identify and judge in the perception module of the autonomous driving system. ISP plays an extraordinary role as the pre-processing link of the image signal. A good ISP image signal processing mechanism can greatly improve the accuracy of system recognition and thus ensure the safe operation of the system. However, the objective indicators for evaluating the image signal processing effect of ISP have not yet been established, and there are still a lot of subjective evaluations.

[0036] Based on this, an embodiment of the present application provides an autonomous driving image processing evaluation method, which can establish a set of objective indicators that can accurately and objectively reflect the image signal processing effect.

[0037] The following is a detailed description of the autonomous driving image processing evaluation method provided by the embodiment of the present application through some embodiments and their application scenarios in combination with the accompanying drawings.

[0038] The autonomous driving image processing evaluation method of the embodiment of the present application can be applied to autonomous driving equipment, such as autonomous driving vehicles, autonomous driving aircraft or drones, etc. The autonomous driving equipment includes an ISP. The autonomous driving image processing evaluation method of the embodiment of the present application is intended to evaluate the image processing effect of the ISP in the autonomous driving equipment.

[0039] See also Figure 1 , Figure 1 This is one of the flow charts of the autonomous driving image processing evaluation method provided in the embodiment of the present application. Figure 1 As shown, the autonomous driving image processing evaluation method of the embodiment of the present application may include the following steps:

[0040] Step 101: Control the ISP to process the data sets of m test scenarios respectively to obtain m processing results corresponding to the m test scenarios one by one, where m is a positive integer.

[0041] In specific implementation, the autonomous driving device can obtain the test scene set Fig = {Im1, Im2…Im m}, where Im i represents the data set of test scenario i. Afterwards, the ISP can be controlled to process the data set of each test scenario to obtain the processing result of the ISP for each test scenario. For easy understanding, please refer to Figure 2a .exist Figure 2a In the figure, the m test scenes are: scene 1, scene 2, ... scene m-1 and scene m; the m processing results are: scene 1', scene 2', ... scene m-1' and scene m'.

[0042] In practical applications, the test scenario set may be obtained by construction or by reception, which may be determined based on actual conditions and is not limited in this regard in the embodiments of the present application.

[0043] For ease of understanding, the following example uses the construction and acquisition method to illustrate:

[0044] Step 1: Determine the m test scenarios.

[0045] In specific implementation, the m test scenarios can be pre-determined based on the driving environment of the autonomous driving device (natural driving environment and / or social driving environment), such as: the m test scenarios can include at least one of the following: normal weather test scenarios, extreme weather (such as rain, fog, snow, etc.) test scenarios, intersection test scenarios, traffic light intersection test scenarios, sidewalk intersection test scenarios, etc.

[0046] Step 2: Obtain the data set for each test scenario.

[0047] In a specific implementation, the data set of each test scenario can be obtained from a training sample database. In the training sample database, the data set of each test scenario can be stored in categories. Furthermore, the training sample database can also store the annotation results of the data set of each test scenario.

[0048] Step 102: input the m processing results into m test model sets corresponding one-to-one to the m test scenarios respectively, and obtain m test result sets corresponding one-to-one to the m test scenarios, wherein each test model set includes at least one test model, each of the test result sets includes at least one first test result, and one test model outputs one first test result.

[0049] It can be understood that each of the m test model sets includes at least one test model, and the number of test models included in different test model sets can be the same or different. In addition, different test model sets can include the same test model. Figure 2a In the example, the test model set corresponding to scenario 1 may include: model a, model b and model c; the test model combination corresponding to scenario m may include: model b, model f and model g.

[0050] During specific implementation, the processing result corresponding to a certain test scenario is input into: each test model in the test model set corresponding to the test scenario, each test model outputs a first test result respectively, and a test result set corresponding to the test scenario is obtained.

[0051] For ease of understanding, the following is an example of the i-th test scenario among the m test scenarios. Assume that the test model set corresponding to the i-th test scenario includes n test models, i is a positive integer less than or equal to m, and n is a positive integer.

[0052] The processing result corresponding to the i-th test scenario is input into the n test models to obtain n first test results corresponding to the i-th test scenario. It can be understood that the test result set corresponding to the i-th test scenario includes the n first test results.

[0053] The input of each of the n test models includes: the processing result corresponding to the i-th test scenario. Furthermore, the input of each of the test models in the test model set corresponding to the i-th test scenario may also include: the annotation result of the data set of the i-th test scenario. The output of each of the n test models is: the first test result corresponding to the i-th test scenario. In practical applications, the test results can be expressed as a test pass rate, a test pass degree or a test value, which can be determined according to actual conditions, and the embodiments of the present application do not limit this.

[0054] In the embodiment of the present application, each test model corresponding to a test scenario can evaluate the reliability of the ISP processing the data of the test scenario from different dimensions. Therefore, it can be understood that each of the n first test results can characterize the reliability of the ISP processing the data of the i-th test scenario from one dimension.

[0055] Step 103: Determine, according to the m test result sets, m second test results corresponding one-to-one to the m test scenarios.

[0056] In order to determine the reliability of the ISP processing the data of the i-th test scenario as a whole, the second test result corresponding to the i-th test scenario can be determined based on the n first test results. That is, the second test result corresponding to the i-th test scenario can characterize the reliability of the ISP processing the data of the i-th test scenario as a whole.

[0057] In a specific implementation, in one embodiment, the average test result of the n first test results may be determined as the second test result corresponding to the i-th test scenario.

[0058] In another implementation, the n first test results may be weighted averaged to obtain the second test result corresponding to the i-th test scenario. Figure 2a As shown, the m second test results are represented as: weighted result 1, ... weighted result m.

[0059] Step 103: Evaluate the ISP according to the m second test results.

[0060] As can be seen from the above content, the second test result corresponding to each test scenario can be used to characterize the reliability of the ISP in processing the data of the test scenario. Therefore, the autonomous driving device can evaluate the processing reliability of the ISP based on the m second test results.

[0061] In one implementation, a threshold may be set for each test scenario. The threshold for each test scenario is: the minimum value used to characterize that the ISP processes the data of the test scenario as qualified. When the second test result of a test scenario is greater than or equal to the threshold of the test scenario, it indicates that the ISP processes the data of the test scenario reliably or qualified, otherwise it indicates that the ISP processes the data of the test scenario unreliably or unqualified. In this implementation, when there are preset second test results among the m second test results that are less than the corresponding threshold, the evaluation result of the ISP may be set to: unqualified; otherwise, the evaluation result of the ISP may be set to: qualified.

[0062] In another implementation manner, the evaluating the ISP according to the m second test results may include:

[0063] Determine a target test result according to the m second test results;

[0064] In the case where the target test result is verified to be passed, indication information is output, where the indication information is used to indicate that the ISP is qualified.

[0065] In this embodiment, the target test result can be determined based on the m second test results, and the target test result can be used to characterize the reliability of the ISP. A threshold is set for the target test result, and the threshold is the minimum value used to characterize the ISP as qualified. In this embodiment, when the target test result is greater than or equal to the threshold, the evaluation result of the ISP can be set to: qualified; otherwise, the evaluation result of the ISP can be set to: unqualified. In this way, compared with the above embodiment, the operation of ISP verification can be simplified, thereby reducing the operating burden of the autonomous driving equipment.

[0066] If the evaluation result of the ISP is qualified, the autonomous driving device can be put into use. If the evaluation result of the ISP is unqualified, the ISP can be adjusted or replaced until the evaluation result of the ISP of the autonomous driving device is qualified, and the autonomous driving device is put into use, so that the safety of autonomous driving can be improved.

[0067] The autonomous driving image processing evaluation method of this embodiment can construct m test scenes for autonomous driving, and m test model sets corresponding to the m test scenes. For each test scene, the ISP of the autonomous driving device can be controlled to process its data set to obtain the processing results of the ISP for each test scene; then, the processing results of the ISP for each test scene are respectively input into the test model set corresponding to each test scene to obtain the test result set of each test scene; based on the test result set of each test scene, the second test result of each test scene is obtained, that is, m second test results. Then, the ISP is evaluated based on the m second test results. It can be seen that the embodiment of the present application provides an evaluation index for evaluating the ISP, that is, the second test result. In this way, if the ISP evaluation fails, the ISP can continue to be debugged until the ISP evaluation passes before the autonomous driving device is put into use, thereby improving the safety of autonomous driving.

[0068] In some embodiments of the present application, the evaluating the ISP according to the m second test results may include:

[0069] The ISP is evaluated according to the m second test results and the m weight values ​​corresponding one-to-one to the m test scenarios.

[0070] In this embodiment, taking into account the different degrees of influence of different test scenarios on the perception module of the autonomous driving device, a weight value can be set for each test scenario to evaluate the ISP based on the second test result and weight value corresponding to each test scenario, thereby improving the reliability of the ISP evaluation.

[0071] In specific implementation, the evaluation score P of the ISP may be calculated using the following formula:

[0072]

[0073] in, represents the weight value corresponding to the i-th test scenario, P i In other words, the m weight values ​​can be used to perform weighted average calculation on the m second test results to obtain the evaluation score P of the ISP.

[0074] In some embodiments of the present application, the test model set corresponding to the i-th test scenario includes n test models, the test model set corresponding to the i-th test scenario includes n test models, and the test result set corresponding to the i-th test scenario includes n first test results, wherein the n first test results are outputs of the n test models, i is a positive integer less than or equal to m, and n is a positive integer;

[0075] The determining, according to the m test result sets, m second test results corresponding one-to-one to the m test scenarios comprises:

[0076] According to the n first test results and the n weight values ​​corresponding one-to-one to the n test models, a second test result corresponding to the i-th test model is determined.

[0077] In this embodiment, it can be seen from the foregoing content that different test models corresponding to a test scenario evaluate the reliability of the ISP processing the test data from different dimensions, and different dimensions have different degrees of influence on the reliability of the ISP processing the test data. Therefore, a weight value can be set for each test model to determine the second test result corresponding to the i-th test model based on the n first test results and the n weight values ​​corresponding one-to-one to the n test models, thereby improving the reliability of the ISP evaluation.

[0078] For ease of understanding, the i-th test scenario is used as an example.

[0079] In one implementation, the n weight values ​​may be used to perform a weighted average calculation on the n first test results to obtain a second test result P corresponding to the i-th test model. i .

[0080] In another implementation, the weight set corresponding to the test model set corresponding to the i-th test scenario is The device automatically obtains the weight set of the test model. Afterwards, the following formula can be used to Perform normalization operation and get

[0081]

[0082] Afterwards, using the normalized Perform weighted average calculation on the n first test results to obtain a second test result P corresponding to the i-th test model. i . In this way, the calculation speed can be improved.

[0083] In the embodiments of the present application, the method for determining the weight value is not limited, for example, it can be set by human experience, or determined by class hierarchy analysis method. The specific method can be determined according to actual needs, and the embodiments of the present application do not limit this.

[0084] In some embodiments, the method may further include:

[0085] Obtain a judgment matrix corresponding to q objects, wherein the judgment matrix includes q×q elements, the element located in the rth row and the jth column represents the importance of object r relative to object j, r and j are both positive integers less than or equal to q; the q objects are the m test scenarios, or, n test models;

[0086] According to the judgment matrix, a consistency check is performed on the q objects to obtain a consistency check result;

[0087] When the consistency check result is verified to be passed, q weight values ​​corresponding to the q objects are determined according to the judgment matrix.

[0088] In specific implementation, the Delphi method can be used to construct a judgment matrix corresponding to q objects. The judgment matrix includes q×q elements, that is, the size of the judgment matrix is ​​q×q. The judgment matrix can be expressed as:

[0089]

[0090] Among them, c rj Represents the importance of object r relative to object j, and satisfies

[0091] After that, the q objects are tested once. Specifically, a perfect judgment matrix C needs to satisfy And the rows of the matrix are linearly correlated, and the columns of the matrix are linearly correlated. Define the consistency index where λ max is the maximum eigenvalue of the judgment matrix.

[0092] When the consistency check is performed on the q objects, the maximum eigenvalue in the judgment matrix may be obtained, and then the consistency index values ​​CI of the q objects are calculated.

[0093] In one implementation, the consistency index value CI may be directly used as the consistency test result.

[0094] In another implementation, a random consistency index RI is further introduced. The value of the random consistency index RI is related to the value of q, as shown in Table 1.

[0095] Table 1: Relationship between RI and q

[0096] q 1 2 3 4 5 6 7 8 9 10 11 RI 0 0 0.58 0.9 1.12 1.24 1.32 1.41 1.45 1.49 1.51

[0097] In this embodiment, the consistency ratio can be defined as The consistency ratio is taken as the consistency test result.

[0098] A threshold value for comparison with the consistency check result may be set, and the threshold value is the minimum value used to indicate that the consistency check has passed. If the consistency check result is greater than or equal to the threshold value, the consistency check result is determined to have passed verification, otherwise the consistency check result is determined to have failed verification.

[0099] When the consistency check result is verified to be passed, q weight values ​​corresponding to the q objects are determined according to the judgment matrix.

[0100] If the consistency check result fails to be verified, the judgment matrix corresponding to the q objects can be updated and the above steps can be re-executed until the consistency check result passes the verification, and then the q weight values ​​corresponding to the q objects are determined.

[0101] In this way, the weight values ​​corresponding to the test scenarios and the weight values ​​corresponding to the test models can be made more objective, thereby improving the objectivity of ISP evaluation.

[0102] In an optional implementation manner, determining, according to the judgment matrix, q weight values ​​corresponding one-to-one to the q objects may include:

[0103] Normalizing the elements of each column of the judgment matrix respectively;

[0104] The weight value corresponding to the object r is determined according to the r-th row elements of the judgment matrix after normalization.

[0105] In specific implementation, the arithmetic mean value of the r-th row elements of the normalized judgment matrix may be calculated to obtain the weight value corresponding to the object r, but the invention is not limited thereto.

[0106] In one implementation, the weight value corresponding to the object r can be calculated by the following formula:

[0107]

[0108] In the above manner, the elements of each column of the judgment matrix are first normalized, and then the weight value corresponding to the object r is determined according to the rth row element of the judgment matrix after the normalization. In this way, the situation where the sum of the elements of each column of the judgment matrix is ​​not 1 can be avoided, thereby improving the reliability of the calculation of the weight value of each object.

[0109] Of course, in other implementations, the arithmetic mean value of the elements in the rth row of the judgment matrix may be directly calculated to obtain the weight value corresponding to the object r, but the invention is not limited thereto.

[0110] It should be noted that the various optional implementation modes introduced in the embodiments of the present application can be implemented in combination with each other or separately if they do not conflict with each other, and the embodiments of the present application are not limited to this.

[0111] For ease of understanding, the following examples are provided:

[0112] In order to effectively, objectively and quantitatively evaluate the image signal processing effect of the perception module of the autonomous driving system, this application designs an evaluation index for the image signal processing algorithm of the perception module of the autonomous driving system. Specifically, the index design proposed in this application can make full use of the test model to conduct a comprehensive image signal processing effect evaluation.

[0113] In order to achieve the above objectives, the specific technical solutions adopted in this application are as follows:

[0114] A design of image signal processing evaluation index for perception module of autonomous driving system includes the following steps:

[0115] Step 1: Construct a test scenario set Fig = {Im1,Im2…Im m In order to objectively and quantitatively evaluate the image signal processing effect, multiple test scene sets can be prepared. The test scenes can be selected from the training sample database, and all selected scene sets are classified and stored. i They all represent a data set of a type of scene.

[0116] Step 2: Construct the test model set Model = {M1, M2, ... M n}. Currently, the image data of the perception module will be called by multiple deep learning models. For example, an image of a traffic light at an intersection will be called by the traffic light recognition model M. i ,Lane line detection model M j ,Vehicle tracking model M k For each of the tested scene sets, a test model set is constructed. Model i It contains all deep learning models of calling scenario i, which is a subset of the entire model set of the perception module.

[0117] Step 3: Design weight indicators for test scenarios Considering that the test scenarios are not unique and different scenarios have different impacts on the autonomous driving perception module, it is necessary to design weight indicators for different test scenarios. This application can use the class hierarchy analysis method to design the scene weights.

[0118] Step 4: Design weight indicators for testing models Considering that there are multiple models that call test scenarios, and different test scenarios call different models. This application designs weights for all models at the beginning of the design, and can be directly referenced when calling the model. This application can use the class hierarchy analysis method to design the model weights.

[0119] Step 5: Use the image signal processing algorithm to process scene i and calculate the test pass rate P for the test scene i i .

[0120] Step 6: Input all tested scenes into the image signal processing in sequence and calculate the overall test pass rate: If the passing degree P ≥ P t If the image signal processing passes the verification, otherwise it fails. t It is the model pass rate threshold, which needs to be given in advance by experts according to the system characteristics.

[0121] The beneficial effects of the embodiments of the present application include: for the image signal processing scenario of the autonomous driving perception module, the test models and test scenarios used are all existing data models, which can greatly reduce the manpower, material resources and time loss of re-development of the test model and re-collection of the test scene. At the same time, the present application adopts a two-class hierarchical analysis weight determination method, and designs weights for the test scenario and the test model respectively. The present application also normalizes the test model weights to ensure the rationality of the results. The index evaluation method designed in the present application can objectively and quantitatively evaluate the effect of the image signal processing algorithm. Compared with the prior art, the present application can more effectively evaluate the image signal processing algorithm, and the data has both subjective feelings and objective effects. The weight index can be reused, the calculation speed is fast, the evaluation effect is good, and it has good practicality and superiority. It is an efficient and practical evaluation index design for the image processing algorithm of the perception module of the autonomous driving system.

[0122] The autonomous driving image processing evaluation method provided in this application can accurately and objectively evaluate the effect of the image processing algorithm, and provide technical support for the stable and reliable operation of the perception module of the autonomous driving system. Figure 2b To assist understanding:

[0123] The autonomous driving image processing evaluation method of the embodiment of the present application may include:

[0124] Step 1: Build a test scenario collection, which includes the following steps:

[0125] Step 1.1: Determine the total number of all scene categories as m.

[0126] Step 1.2: According to the total number of scene categories determined, the scene database is divided into scene data to form a scene set Fig = {Im1, Im2…Imm}, where Im i Represents the set of scenes of category i.

[0127] Step 2: Build a test model set, which includes the following steps:

[0128] Step 2.1: For the i-th test scene set Im i , filter calls Im i Model Construct the i-th test model set Model i .

[0129] Step 2.2: Build Model for all test scenario sets i , and finally form Model = {Model i ,i=1,2,…,m}.

[0130] Step 3: Design weight indicators for test scenarios Taking into account that the test scenarios are not unique and different scenarios have different degrees of impact on the autonomous driving perception module, it is necessary to design weight indicators for different test scenarios.

[0131] Step 3.1: Conduct Delphi expert survey method to obtain the judgment matrix A of the test scenario = {a ij}. The size of matrix A is m×m.

[0132]

[0133] Element a in matrix A ij represents the importance of scene i relative to scene j, and satisfies

[0134] Step 3.2: Perform consistency check on the test scenario. Specifically, a perfect judgment matrix A needs to satisfy And the rows of the matrix are linearly correlated, and the columns of the matrix are linearly correlated. Define the consistency index where λ max is the maximum eigenvalue of the judgment matrix, and m is the number of influencing factors.

[0135] Step 3.3: Determine whether the consistency test has passed. CI = 0, the matrix has complete consistency, CI → 0 has a relatively satisfactory consistency, and the larger the CI, the more serious the inconsistency. In order to make an objective and quantitative judgment, the random consistency index RI is introduced. Definition of consistency ratio If CR<0.1, the inconsistency of A is considered to be within the allowable range and continue with step 3.4; otherwise, return to step 3.1.

[0136] Step 3.4: Calculate the test scenario weight. For the i-th scenario, the weight calculation formula is given as

[0137]

[0138] This formula normalizes each column of the judgment matrix, and then calculates the arithmetic mean of the normalized vectors.

[0139] Step 4: Design weight indicators for testing models Considering that there are multiple models that will call test scenarios, and different test scenarios call different models. This application designs weights for all models at the beginning of the design, and can call them directly when calling models, avoiding multiple calculations. The calculation method performed in step 4 is similar to that in step 3.

[0140] Step 4.1: Conduct Delphi expert survey to obtain the judgment matrix B of the test model = {b ij}. The matrix B has size n×n.

[0141]

[0142] The element b in matrix B ij Represents the importance of model i relative to model j, and satisfies

[0143] Step 4.2: Perform consistency check on the test scenario. Specifically, a perfect judgment matrix B needs to satisfy And the rows of the matrix are linearly correlated, and the columns of the matrix are linearly correlated. Define the consistency index where λ max is the maximum eigenvalue of the judgment matrix, and n is the number of influencing factors.

[0144] Step 4.3: Determine whether the consistency test has passed. CI = 0, the matrix has complete consistency, CI → 0 has a relatively satisfactory consistency, and the larger the CI, the more serious the inconsistency. In order to make an objective and quantitative judgment, the random consistency index RI is introduced. Definition of consistency ratio If CR<0.1, the inconsistency of A is considered to be within the allowable range and continue with step 4.4; otherwise, return to step 4.1.

[0145] Step 4.4: Calculate the test model weight. For the i-th scenario, the weight calculation formula is given as

[0146]

[0147] This formula normalizes each column of the judgment matrix, and then calculates the arithmetic mean of the normalized vectors.

[0148] Step 5: Use the image signal processing algorithm to process scene i and calculate the test pass rate P for the test scene i i .

[0149] Step 5.1: Extract scene i and process it using image processing algorithms

[0150] Step 5.2: Extract the test model weight index set involved in scenario i right Perform normalization operation and get It can be obtained by the following calculation expression

[0151]

[0152] Step 5.3: Calculate the test pass rate of the test scene i after image processing, O ij is the test pass rate of model j under scenario i input, O ij ∈[0,1].

[0153]

[0154] Step 6: Input all tested scenes into the image signal processing in sequence, and calculate the overall test pass rate as:

[0155]

[0156] If the passing degree P ≥ P t If the image signal processing passes the verification, otherwise it fails. t is the model pass rate threshold, which is given in advance by experts based on system characteristics.

[0157] Based on the autonomous driving image processing evaluation method provided in the above embodiment, the present application also provides a specific implementation of the autonomous driving image processing evaluation device. Please refer to the following embodiment.

[0158] See also Figure 3 , the autonomous driving image processing evaluation device provided in the embodiment of the present application may include:

[0159] The control module 301 is used to control the ISP to process the data sets of m test scenarios respectively, and obtain m processing results corresponding to the m test scenarios one by one, where m is a positive integer;

[0160] The first acquisition module 302 is used to input the m processing results into m test model sets corresponding to the m test scenarios one by one,

[0161] Obtain m test result sets corresponding one-to-one to the m test scenarios, wherein each test model set includes at least one test model, each of the test result sets includes at least one first test result, and one test model outputs one first test result;

[0162] A first determining module 303 is used to determine m second test results corresponding to the m test scenarios one by one according to the m test result sets;

[0163] The evaluation module 304 is used to evaluate the ISP according to the m second test results.

[0164] In some embodiments, the evaluation module 303 is specifically used to:

[0165] The ISP is evaluated according to the m second test results and the m weight values ​​corresponding one-to-one to the m test scenarios.

[0166] In some embodiments, the test model set corresponding to the i-th test scenario includes n test models, and the test result set corresponding to the i-th test scenario includes n first test results, wherein the n first test results are outputs of the n test models, i is a positive integer less than or equal to m, and n is a positive integer;

[0167] The first determining module 303 includes:

[0168] The first determining unit is used to determine a second test result corresponding to the i-th test model according to the n first test results and n weight values ​​corresponding one-to-one to the n test models.

[0169] In some embodiments, the apparatus further comprises:

[0170] A second acquisition module is used to acquire a judgment matrix corresponding to q objects, wherein the judgment matrix includes q×q elements, the element located in the rth row and the jth column represents the importance of object r relative to object j, and r and j are both positive integers less than or equal to q; the q objects are the m test scenarios, or the n test models;

[0171] A verification module, used to perform consistency verification on the q objects according to the judgment matrix to obtain a consistency verification result;

[0172] The second determination module is used to determine q weight values ​​corresponding to the q objects one by one according to the judgment matrix when the consistency check result is verified to be passed.

[0173] In some embodiments, the second determining module includes:

[0174] A normalization unit, used for normalizing the elements of each column of the judgment matrix respectively;

[0175] The second determining unit is used to determine the weight value corresponding to the object r according to the r-th row elements of the judgment matrix after normalization.

[0176] In some embodiments, the evaluation module 303 includes:

[0177] A third determining unit, configured to determine a target test result according to the m second test results;

[0178] The output unit is used to output indication information when the target test result is verified to be passed, wherein the indication information is used to indicate that the ISP is qualified.

[0179] The automatic driving image processing and evaluation device provided in the embodiment of the present application can achieve Figure 1 The various processes implemented by the autonomous driving image processing and evaluation device in the method embodiment will not be described here to avoid repetition.

[0180] Figure 4 The hardware structure diagram of the image processing evaluation provided by the embodiment of the present application is shown.

[0181] The autonomous driving image processing and evaluation device may include a processor 401 and a memory 402 storing program instructions.

[0182] Specifically, the processor 401 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0183] Memory 402 may include a large capacity memory for data or instructions. By way of example and not limitation, memory 402 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, memory 402 may include a removable or non-removable (or fixed) medium. In appropriate cases, memory 402 may be inside or outside of an integrated gateway disaster recovery device. In a specific embodiment, memory 402 is a non-volatile solid-state memory.

[0184] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage medium device, an optical storage medium device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Therefore, generally, the memory includes one or more tangible (non-transitory) readable storage media (e.g., memory devices) encoded with software including executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of the present application.

[0185] The processor 401 implements any one of the autonomous driving image processing evaluation methods in the above-mentioned embodiments by reading and executing program instructions stored in the memory 402.

[0186] In one example, the autonomous driving image processing evaluation device may further include a communication interface 403 and a bus 410. Figure 4 As shown, the processor 401, the memory 402, and the communication interface 403 are connected via a bus 410 and communicate with each other.

[0187] The communication interface 403 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0188] The bus 410 includes hardware, software or both, coupling the components of the autonomous driving image processing evaluation device to each other. By way of example and not limitation, the bus may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industrial standard architecture (EISA) bus, a front-end bus (FSB), a hypertransport (HT) interconnect, an industrial standard architecture (ISA) bus, an infinite bandwidth interconnect, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus or other suitable bus or a combination of two or more of these. Where appropriate, the bus 410 may include one or more buses. Although the embodiments of the present application describe and illustrate a specific bus, the present application considers any suitable bus or interconnect.

[0189] In addition, in combination with the autonomous driving image processing evaluation method in the above embodiment, the embodiment of the present application can provide a storage medium for implementation. The storage medium stores program instructions; when the program instructions are executed by the processor, any of the autonomous driving image processing evaluation methods in the above embodiment is implemented.

[0190] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.

[0191] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a network such as the Internet, an intranet, etc.

[0192] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.

[0193] The present application is described above with reference to the flowchart and / or block diagram of the method, device (system) and program product according to the embodiment of the present application.It should be understood that each square block in the flowchart and / or block diagram and the combination of each square block in the flowchart and / or block diagram can be realized by program instructions.These program instructions can be provided to the processor of general, special, or other programmable data processing device, to produce a kind of machine, so that these instructions executed by the processor of other programmable data processing device enable the realization of the function / action specified in one or more square blocks of the flowchart and / or block diagram.Such processor can be but not limited to general processor, special processor, special application processor or field programmable logic circuit.It can also be understood that each square block in the block diagram and / or flowchart and the combination of square blocks in the block diagram and / or flowchart can also be realized by the dedicated hardware that performs the specified function or action, or can be realized by the combination of dedicated hardware and instruction.

[0194] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.

Claims

1. An automatic driving image processing evaluation method, applied to an automatic driving device, characterized in that: The autonomous driving device includes an image signal processor ISP; the method includes: Control the ISP to process the data sets of m test scenarios respectively to obtain m processing results corresponding to the m test scenarios one by one, where m is a positive integer; Input the m processing results into m test model sets corresponding to the m test scenarios one by one, respectively, to obtain m test result sets corresponding to the m test scenarios one by one, wherein each test model set includes at least one test model, each test result set includes at least one first test result, and one test model outputs one first test result; Determine, according to the m test result sets, m second test results corresponding one-to-one to the m test scenarios; Evaluating the ISP according to the m second test results; The test model set corresponding to the i-th test scenario includes n test models, and the test result set corresponding to the i-th test scenario includes n first test results, wherein the n first test results are outputs of the n test models, i is a positive integer less than or equal to m, and n is a positive integer; The determining, according to the m test result sets, m second test results corresponding one-to-one to the m test scenarios comprises: According to the n first test results and the n weight values ​​corresponding one-to-one to the n test models, a second test result corresponding to the i-th test scenario is determined.

2. The method according to claim 1, characterized in that The evaluating the ISP according to the m second test results includes: The ISP is evaluated according to the m second test results and the m weight values ​​corresponding one-to-one to the m test scenarios.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Obtain a judgment matrix corresponding to q objects, wherein the judgment matrix includes q×q elements, the element located in the rth row and the jth column represents the importance of object r relative to object j, r and j are both positive integers less than or equal to q; the q objects are the m test scenarios, or, n test models; According to the judgment matrix, a consistency check is performed on the q objects to obtain a consistency check result; When the consistency check result is verified to be passed, q weight values ​​corresponding to the q objects are determined according to the judgment matrix.

4. The method according to claim 3, characterized in that Determining q weight values ​​corresponding to the q objects one by one according to the judgment matrix includes: Normalizing the elements of each column of the judgment matrix respectively; The weight value corresponding to the object r is determined according to the r-th row elements of the judgment matrix after normalization.

5. The method according to claim 1, characterized in that The evaluating the ISP according to the m second test results includes: Determine a target test result according to the m second test results; In the case where the target test result is verified to be passed, indication information is output, where the indication information is used to indicate that the ISP is qualified.

6. An automatic driving image processing evaluation device, characterized in that: The automatic driving image processing and evaluation device comprises an image signal processor ISP; the device comprises: A control module, used to control the ISP to process the data sets of m test scenarios respectively, to obtain m processing results corresponding to the m test scenarios one by one, where m is a positive integer; A first acquisition module is used to input the m processing results into m test model sets corresponding to the m test scenarios one by one, respectively, to obtain m test result sets corresponding to the m test scenarios one by one, wherein each test model set includes at least one test model, each of the test result sets includes at least one first test result, and one test model outputs one first test result; A first determining module, configured to determine, according to the m test result sets, m second test results corresponding one-to-one to the m test scenarios; An evaluation module, configured to evaluate the ISP according to the m second test results; The test model set corresponding to the i-th test scenario includes n test models, and the test result set corresponding to the i-th test scenario includes n first test results, wherein the n first test results are outputs of the n test models, i is a positive integer less than or equal to m, and n is a positive integer; The first determination module is specifically used to determine the second test result corresponding to the i-th test scenario based on the n first test results and the n weight values ​​corresponding one-to-one to the n test models.

7. An automatic driving image processing evaluation device, characterized in that: The device includes: a processor and a memory storing program instructions; when the processor executes the program instructions, the autonomous driving image processing and evaluation method as described in any one of claims 1 to 5 is implemented.

8. A readable storage medium, characterized in that: The readable storage medium stores program instructions, and when the program instructions are executed by the processor, the autonomous driving image processing and evaluation method as described in any one of claims 1 to 5 is implemented.

9. A program product, characterized in that When the instructions in the program product are executed by a processor of an electronic device, the electronic device executes the autonomous driving image processing and evaluation method as described in any one of claims 1 to 5.