Evaluation methods, devices, electronic equipment, and storage media for target detection models

Abstract

This disclosure relates to an evaluation method, apparatus, electronic device, and storage medium for an object detection model, belonging to the field of artificial intelligence technology. The method includes: acquiring at least one target scene parameter of a target scene use case; acquiring parameter values ​​of the target scene parameters; running the target scene use case based on the parameter values ​​of the target scene parameters, and acquiring at least one scene image and a baseline annotation of the scene image during the execution of the target scene use case; predicting objects in the scene image using the object detection model running in the target scene use case, obtaining predicted annotations for the scene image; and calculating the evaluation result of the object detection model based on the baseline annotation and predicted annotations corresponding to at least one scene image. This disclosure can simulate real-world scenarios based on scene parameters and scene use cases, and then use the scene images of the simulated scenario to evaluate the object detection model. It incorporates the constraints of real-world conditions, making it more practical and resulting in more accurate model evaluation results.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to an evaluation method, apparatus, electronic device and storage medium for a target detection model. Background Technology

[0002] An object detection model is a model based on the geometric and statistical features of an object. By evaluating the object detection model, its performance can be determined.

[0003] In related technologies, the evaluation of object detection models often employs two methods: adversarial examples and formal verification. Both methods essentially test the robustness of the model within a data sample space. The difference lies in their approach to exploring this space: adversarial examples aim to force the model into errors to obtain an upper bound on robustness, while formal verification ensures the model's correctness to obtain a lower bound on robustness.

[0004] While the two methods described above can evaluate target detection models, both methods rely on perturbing or modifying the data in a high-dimensional data space, such as by adding noise. This lack of constraints related to real-world conditions greatly limits their application in practical problems, and the evaluation results are often not accurate enough. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this disclosure provides an evaluation method, apparatus, electronic device and storage medium for target detection models.

[0006] According to a first aspect of the present disclosure, an evaluation method for a target detection model is provided, comprising:

[0007] Obtain at least one target scenario parameter of the target scenario use case, wherein the target scenario parameter refers to a scenario parameter whose parameter value is adjustable in the target scenario use case;

[0008] Obtain the parameter values ​​of the target scene parameters;

[0009] Run the target scenario use case according to the parameter values ​​of the target scenario parameters, and obtain at least one scene image and the baseline annotation of the scene image during the execution of the target scenario use case. The annotation is used to identify objects in the scene image.

[0010] The predicted labels of the scene images are obtained by predicting objects in the scene images using an object detection model that runs in the target scene use case.

[0011] The evaluation result of the target detection model is calculated based on the baseline annotation and prediction annotation corresponding to at least one of the scene images. The evaluation result is used to indicate the prediction accuracy performance of the target detection model.

[0012] Optionally, after calculating the evaluation result of the target detection model based on the baseline annotation and predicted annotation corresponding to at least one of the scene images, the method further includes: in response to the evaluation result meeting a preset condition, ending the evaluation of the target detection model, wherein the preset condition is a boundary detection termination condition; in response to the evaluation result not meeting the preset condition, adjusting the parameter value of at least one of the target scene parameters, and re-executing the step of running the target scene use case based on the parameter value of the target scene parameters until the evaluation result meets the preset condition.

[0013] Optionally, adjusting the parameter value of at least one of the target scene parameters includes: determining a first scene parameter among the target scene parameters that needs parameter adjustment; obtaining a first parameter value of the first scene parameter, wherein the first parameter value is the parameter value of the first scene parameter when the evaluation result of the target detection model is calculated for the Nth time, and N is an integer greater than or equal to 2; determining to accept the first parameter value in response to a second evaluation result indicating that the model performance is lower than the model performance indicated by a first evaluation result; wherein the second evaluation result is the evaluation result of the target detection model calculated for the Nth time, and the first evaluation result is the evaluation result of the target detection model calculated for the (N-1)th time; calculating the probability that the first parameter value is accepted in response to a second evaluation result indicating that the model performance is higher than the model performance indicated by the first evaluation result, and determining whether to accept the first parameter value based on the probability; adjusting the parameter value of the first scene parameter based on the first parameter value in response to accepting the first parameter value; and adjusting the parameter value of the first scene parameter based on the second parameter value in response to not accepting the first parameter value.

[0014] Optionally, calculating the probability that the first parameter value is accepted includes: using a preset probability calculation formula, calculating the probability that the first parameter value is accepted based on the first evaluation result, the second evaluation result, and the obtained current annealing temperature; the probability calculation formula is as follows:

[0015]

[0016] Wherein, P represents the probability that the first parameter value is accepted, e represents the natural index, and f k+1 This represents the second evaluation result, where f kThis represents the first evaluation result, where T represents the annealing temperature; the annealing temperature is calculated according to the following formula:

[0017] T(k) = rateT(k+1)

[0018] Wherein, T(k) represents the annealing temperature of the k-th iteration, T(k+1) represents the annealing temperature of the (k+1)-th iteration, rate represents the annealing rate, and rate∈(0,1).

[0019] Optionally, calculating the evaluation result of the target detection model based on the baseline annotation and predicted annotation corresponding to at least one of the scene images includes: for each scene image, determining the evaluation index value corresponding to the scene image based on the baseline annotation and predicted annotation corresponding to the scene image; wherein, the evaluation index value refers to the evaluation value of the scene image under the corresponding evaluation index, and the evaluation index includes at least one of the following: average precision across all classes (mAP), accuracy, and error rate; and calculating the evaluation result of the target detection model based on the evaluation index value corresponding to at least one of the scene images.

[0020] Optionally, the target scenario use case is an autonomous driving scenario use case; the step of running the target scenario use case according to the parameter values ​​of the target scenario parameters and obtaining at least one scenario image and the reference annotation of the scenario image during the execution of the target scenario use case includes: generating a target scenario containing a scenario map, objects, and a main vehicle based on the scenario configuration file of the target scenario use case and the parameter values ​​of the target scenario parameters; controlling the main vehicle to move along a preset route in the scenario map, and controlling the main vehicle to collect at least one scenario image during the movement; and adding the reference annotation to the corresponding scenario image for each scenario image.

[0021] Optionally, the target scene parameters include at least one of the following: weather parameters, object static attribute parameters, and object dynamic attribute parameters; wherein, the weather parameters include at least one of the following: rainfall, cloud cover, solar altitude angle, and solar azimuth angle; the object static attribute parameters include at least one of the following: size and color; and the object dynamic attribute parameters include at least one of the following: speed and orientation.

[0022] According to a second aspect of the present disclosure, an evaluation apparatus for a target detection model is provided, comprising:

[0023] The parameter acquisition module is used to acquire at least one target scenario parameter of the target scenario use case, wherein the target scenario parameter refers to the scenario parameter whose parameter value is adjustable in the target scenario use case;

[0024] The parameter value acquisition module is used to acquire the parameter values ​​of the target scene parameters;

[0025] The image acquisition module is used to run the target scene test case according to the parameter value of the target scene parameter, and to acquire at least one scene image and the baseline annotation of the scene image during the execution of the target scene test case, wherein the annotation is used to identify objects in the scene image;

[0026] The object detection module is used to predict objects in the scene image by using an object detection model running in the target scene use case, and to obtain the predicted annotation of the scene image.

[0027] The model evaluation module is used to calculate the evaluation result of the target detection model based on the baseline annotation and prediction annotation corresponding to at least one of the scene images. The evaluation result is used to indicate the prediction accuracy performance of the target detection model.

[0028] According to a third aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor and a memory storing computer program instructions; the processor, when executing the computer program instructions, implements an evaluation method for a target detection model provided in the first aspect of the present disclosure.

[0029] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided having computer program instructions stored thereon, which, when executed by a processor, implement the evaluation method for the target detection model provided in the first aspect of the present disclosure.

[0030] The technical solutions provided in this disclosure have at least the following beneficial effects:

[0031] The object detection model evaluation method disclosed herein can simulate real-world scenarios based on scene parameters and scenario use cases, thereby evaluating the object detection model using scene images in the simulated scenario. Compared with traditional evaluation methods such as adversarial examples and formal verification, it is more reasonable, more practical, and the model evaluation results are more accurate.

[0032] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0034] Figure 1 This is a flowchart illustrating an evaluation method for a target detection model according to an exemplary embodiment.

[0035] Figure 2 This is a flowchart illustrating a method for acquiring scene images and their reference annotations according to an exemplary embodiment.

[0036] Figure 3 This is a schematic diagram illustrating the encapsulation and working mechanism of an object detection model according to an exemplary embodiment.

[0037] Figure 4 This is a flowchart illustrating an evaluation method for an object detection model according to another exemplary embodiment.

[0038] Figure 5 This is a flowchart illustrating a method for adjusting parameter values ​​according to an exemplary embodiment.

[0039] Figure 6 This is an initial scene diagram and a model recognition result diagram shown in an exemplary embodiment during the first round of evaluation of the target detection model.

[0040] Figure 7 This is an initial scene diagram and a model recognition result diagram shown in an exemplary embodiment during the 50th round of evaluation of the target detection model.

[0041] Figure 8 This is an initial scene diagram and a model recognition result diagram shown in an exemplary embodiment when the target detection model is evaluated for the 100th round.

[0042] Figure 9 This is a block diagram illustrating an evaluation apparatus for a target detection model according to an exemplary embodiment.

[0043] Figure 10 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0044] The exemplary embodiments will now be described in detail with reference to the accompanying drawings.

[0045] It should be noted that the relevant embodiments and accompanying drawings are only for describing and illustrating exemplary embodiments provided by this disclosure, and not all embodiments of this disclosure, nor should this disclosure be understood to be limited to the relevant exemplary embodiments.

[0046] It should be noted that the terms "first," "second," etc., used in this disclosure are only used to distinguish different steps, devices, or modules. These terms do not represent any specific technical meaning, nor do they indicate any order or interdependence between them.

[0047] It should be noted that the term "at least one" as used in this disclosure is illustrative rather than restrictive. Unless otherwise expressly indicated in the context, it should be understood as "one or more".

[0048] It should be noted that the term "and / or" used in this disclosure is used to describe the relationship between related objects, and generally indicates that there are at least three relationships. For example, A and / or B can at least indicate: the existence of A alone, the existence of both A and B, and the existence of B alone.

[0049] It should be noted that the various steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Unless otherwise specified, the scope of this disclosure is not limited to the order in which the steps are described in the relevant embodiments.

[0050] It should be noted that all actions involving the acquisition of signals, information, or data in this disclosure are carried out in compliance with the relevant data protection laws and policies of the country where the location is situated, and with authorization from the owner of the relevant device.

[0051] Object detection models are models used to detect objects (such as machine learning models and deep learning models), typically used to identify objects in images. The object is a specified entity based on the specific needs; different objects can be designated as targets depending on the requirements.

[0052] Figure 1 This is a flowchart illustrating an evaluation method for a target detection model according to an exemplary embodiment, such as... Figure 1 As shown, the evaluation method for the target detection model includes the following steps S110 to S150.

[0053] S110: Obtain at least one target scenario parameter for the target scenario use case.

[0054] The target scenario use case is a scenario use case used to evaluate the target detection model to be evaluated. The target scenario parameters are scenario parameters whose values ​​can be adjusted within the target scenario use case. By adjusting the parameter values ​​of the target scenario parameters, the target scenario use case can present different scenario effects at runtime.

[0055] S120: Obtain the parameter values ​​of the target scene parameters.

[0056] The parameter values ​​of the target scene parameters are adjustable. In actual use, users or computer devices can set the parameter values ​​of the target scene parameters according to actual needs. In this way, when evaluating the target detection model, the parameter values ​​of the target scene parameters set by the user or computer device can be obtained.

[0057] S130: Run the target scenario test case according to the parameter values ​​of the target scenario parameters, and obtain at least one scenario image and the baseline annotation of the scenario image during the execution of the target scenario test case, wherein the annotation is used to identify objects in the scenario image.

[0058] Running target scene test cases based on the parameter values ​​of the target scene parameters generates a scene corresponding to those parameter values. By acquiring images from the generated scene, scene images can be obtained. The baseline annotation serves as a benchmark for evaluating the object detection model; it accurately identifies objects in the scene images, which are the objects the object detection model needs to recognize.

[0059] S140: By predicting objects in a scene image using an object detection model that runs in the target scene use case, the predicted annotation of the scene image is obtained.

[0060] An object detection model refers to the object detection model to be evaluated, used to identify objects in an image. After acquiring a scene image, it is input into the object detection model, which then predicts the objects in the input scene image and outputs the prediction results. The prediction results use annotations to identify the objects predicted by the object detection model. These annotations used to identify objects in the prediction results are called prediction annotations.

[0061] S150: Calculate the evaluation result of the object detection model based on the baseline and predicted annotations corresponding to at least one scene image.

[0062] Since baseline annotations accurately identify objects in a scene image, and prediction annotations identify objects predicted by the object detection model, the accuracy of the object predictions can be determined based on the baseline and prediction annotations corresponding to the scene image. This leads to the evaluation result of the object detection model. The evaluation result indicates the prediction accuracy performance of the object detection model, reflecting the degree of accuracy with which the model predicts objects in the scene image.

[0063] This embodiment provides an evaluation method for an object detection model. First, the parameter values ​​of the target scene parameters in the target scene use cases are obtained. Then, the target scene use cases are run based on the parameter values, and at least one scene image and its baseline annotation are obtained during the execution of the target scene use cases. Next, the object detection model is used to predict objects in the scene images, obtaining predicted annotations for the scene images. Finally, the evaluation result of the object detection model is determined based on the baseline annotations and predicted annotations corresponding to at least one scene image. According to this embodiment, a real-world scenario can be simulated based on scene parameters and scene use cases, thereby using scene images from the simulated scenario to evaluate the object detection model. Compared to traditional evaluation methods using adversarial examples and formal verification, the evaluation method of this disclosure incorporates the constraints of real-world conditions, making it more reasonable, more practical, and resulting in more accurate model evaluation results.

[0064] In some embodiments, step S110 may further include determining target scenario use cases. Optionally, determining target scenario use cases may include: pre-building a scenario use case library containing multiple pre-built scenario use cases; selecting scenario use cases from the scenario use case library that match the target detection model as target scenario use cases. Different scenario use cases may correspond to different application scenarios. When it is determined that the target detection model needs to be evaluated, the evaluator may select scenario use cases from the scenario use case library that match the target detection model to be evaluated according to actual needs, and then use the scenario use cases selected by the evaluator as target scenario use cases. For example, the evaluator may select scenario use cases that match the target detection model to be evaluated based on the objects detected by the target detection model and / or application scenarios. Optionally, a human-computer interaction platform that can access the scenario use case library may be set up, through which the evaluator can access the scenario use case library and thus select target scenario use cases.

[0065] By determining target scenario use cases in this way, since different target detection models may need to detect different targets, in order to evaluate different target detection models, evaluators can select appropriate scenario use cases to evaluate the target detection models according to actual evaluation needs, so that the evaluation of the target detection models is more in line with the evaluation requirements.

[0066] In some embodiments, step S110 may further include determining target scenario use cases. Optionally, determining target scenario use cases may include: pre-constructing corresponding scenario use cases for the target detection model. Based on this, when evaluating the target detection model, the pre-constructed scenario use cases corresponding to the target detection model can be directly determined as target scenario use cases. Optionally, corresponding scenario use cases can be pre-constructed according to the objects and / or application scenarios detected by the target detection model, and different default scenario use cases can be set for target detection models with different detection objects.

[0067] This method of determining target scenario use cases allows computer equipment to automatically acquire target scenario use cases based on the correspondence between the target detection model and the scenario use cases. This eliminates the need for evaluation personnel to perform any operations, thus improving evaluation efficiency.

[0068] In some embodiments, step S110 may further include determining target scenario use cases. Optionally, determining target scenario use cases may include: when it is determined that the target detection model needs to be evaluated, the evaluator may construct scenario use cases for evaluating the target detection model according to the evaluation requirements, and determine the scenario use cases constructed by the evaluator as target scenario use cases. The evaluator may construct scenario use cases based on the objects detected by the target detection model to be evaluated and / or application scenarios, etc. Optionally, the evaluator may use a simulation environment to construct scenario use cases.

[0069] This method identifies target scenario use cases and constructs them in real time based on the evaluation requirements of the target detection model to be evaluated, making the target scenario use cases more consistent with the evaluation requirements of the target detection model.

[0070] In some embodiments, when obtaining the target scene parameters of the target scene use case in step S110, at least one scene parameter that can adjust the scene effect presented by the target scene use case can be obtained as the target scene parameter. Optionally, a scene use case typically includes multiple scene parameters that can adjust the scene effect presented by the scene use case. The evaluator can select at least one scene parameter from these multiple scene parameters according to the actual evaluation needs. Thus, when obtaining at least one target scene parameter of the target scene use case in S110, the at least one scene parameter selected by the evaluator can be obtained as the target scene parameter according to the evaluator's selection operation.

[0071] This approach allows evaluators to independently select appropriate target scenario parameters based on actual needs, thereby ensuring that the obtained target scenario parameters better meet the actual evaluation requirements.

[0072] In some embodiments, when obtaining the target scene parameters of the target scene use case in step S110, at least one scene parameter that can adjust the scene effect presented by the target scene use case can be obtained as the target scene parameter. Optionally, for a pre-built target scene use case, the target scene parameters used for evaluating the target detection model can be pre-set in the target scene use case when constructing the target scene use case; based on this, when obtaining at least one target scene parameter of the target scene use case in S110, at least one pre-set target scene parameter used for evaluating the target detection model can be directly obtained.

[0073] In this way, computer devices can directly obtain target scene parameters based on pre-set parameters without subsequent manual operation, thereby improving evaluation efficiency.

[0074] In some embodiments, when obtaining the parameter values ​​corresponding to the target scene parameters in step S120, for each target scene parameter, the parameter range of the target scene parameter can be determined first, and then the computer device can automatically and randomly select a parameter value from the parameter range, or select the median, maximum, minimum, etc. of the parameter range as the parameter value of the target scene parameter. The parameter range of the target scene parameters can be preset or set in real time according to actual needs.

[0075] This method can automatically generate parameter values ​​for the target scene without manual intervention, thereby improving evaluation efficiency.

[0076] In some embodiments, when obtaining the parameter value corresponding to the target scene parameter in step S120, for each target scene parameter, the parameter range of the target scene parameter can be determined first, and then the evaluator can select a parameter value from the parameter range of the target scene parameter as the parameter value of the target scene parameter according to the actual situation.

[0077] This approach allows evaluators to select appropriate parameter values ​​based on actual needs, making the scenarios generated by running the tests based on these parameter values ​​more aligned with the evaluation requirements, and consequently, making the final evaluation results more in line with those requirements.

[0078] In some embodiments, step S130 can utilize a simulation testing platform to run the target scenario use case based on the parameter values ​​of the target scenario parameters. Since the object detection model is used to detect objects in an image, at least one scene image generated during the runtime of the target scenario use case needs to be obtained to evaluate the object detection model. The method of obtaining scene images may differ for different scenario use cases. The following explanation uses an example where the object detection model is used to detect vehicles (i.e., vehicles as objects) in an image, and the target scenario use case is an autonomous driving scenario use case, to illustrate the process in step S130 of running the target scenario use case based on the parameter values ​​of the target scenario parameters and obtaining at least one scene image during the runtime of the target scenario use case.

[0079] Optionally, such as Figure 2 As shown, step S130 may include the following steps S210 to S230.

[0080] S210: Based on the scenario configuration file of the target scenario use case, it generates a target scenario containing a scenario map, objects and the main vehicle by combining the parameter values ​​of the target scenario parameters.

[0081] The scenario configuration file for the target scenario use case is pre-set during the construction of the target scenario use case, and therefore can be obtained directly. The scenario configuration file may include configuration information such as map configuration information, dynamic object configuration information, and static object configuration information.

[0082] The target scene parameters may include at least one of the following: weather parameters, static object attribute parameters, and dynamic object attribute parameters. Weather parameters may include rainfall, cloud cover, solar altitude angle, solar azimuth angle, etc.; static object attribute parameters may include size, color, etc.; and dynamic object attribute parameters may include speed, orientation, etc.

[0083] Taking the target scene parameters, including weather parameters, object static attribute parameters, and object dynamic attribute parameters, as an example, when generating a target scene containing a scene map, objects, and a main vehicle based on the scene configuration file of the obtained target scene use cases and the parameter values ​​of the target scene parameters, the following steps can be taken: First, load the scene map according to the map configuration information; second, change the scene weather according to the corresponding parameter values ​​of the weather parameters; third, place objects in the scene map according to the configuration information of dynamic and static objects, as well as the corresponding parameter values ​​of the object's dynamic and static attribute parameters, and set dynamic parameters such as the speed and orientation of the dynamic objects; fourth, generate the main vehicle in the scene map and pass the main vehicle's corresponding route into the target scene.

[0084] S220: Controls the main vehicle to move along a preset route on the scene map, and controls the main vehicle to collect at least one scene image during the movement.

[0085] For example, the simulation test platform (such as Unreal Engine) running the target scenario test case can be set to synchronous mode, allowing the main vehicle to start moving according to the route in the input scenario. Based on each frame of the moving image, it is determined whether the main vehicle has reached the preset photo-taking area. After determining that the main vehicle has moved to the photo-taking area, scene images are collected using sensors attached to the main vehicle for image generation while the main vehicle is moving within the photo-taking area. Optionally, the main vehicle can generate one scene image every 20 frames while collecting scene images.

[0086] S230: For each scene image, add a baseline annotation to the corresponding scene image.

[0087] After acquiring at least one scene image, the simulation testing platform can automatically add baseline annotations to each scene image.

[0088] In some embodiments, the object detection model may be pre-built and stored. Figure 3 This is a schematic diagram illustrating the encapsulation and testing mechanism of the object detection model, as shown below. Figure 3As shown, an object detection model can be constructed first according to the model definition rules. The constructed object detection model is then uploaded and stored in an evaluation platform used for model evaluation. The storage address of the object detection model and the corresponding task information are written into a database. The task information is the evaluation task information corresponding to the object detection model. Based on this, in step S140, the storage address of the object detection model can be obtained from the database based on the task information. Then, the object detection model is called from the evaluation platform according to the storage address, and then used to detect scene images, thereby obtaining the predicted annotations of the scene images. Optionally, when storing the object detection model on the evaluation platform, the object detection model can be encapsulated according to preset encapsulation rules before being uploaded to the evaluation platform for storage. By encapsulating the model, it is converted into a format that the evaluation platform can call, ensuring that the evaluation platform can call the model.

[0089] In some embodiments, when calculating the evaluation result of the object detection model based on the baseline and predicted labels of at least one scene image in step S150, the evaluation index value corresponding to the scene image can be determined for each scene image based on the baseline and predicted labels corresponding to the scene image, and then the evaluation result of the object detection model can be calculated based on the evaluation index value corresponding to at least one scene image.

[0090] The evaluation index value refers to the evaluation value of the scene image under the corresponding evaluation index. The evaluation index can be set according to actual needs. For example, the mean average precision (mAP), accuracy, and error rate can be used as evaluation indexes.

[0091] Optionally, when calculating the evaluation result of the object detection model based on the evaluation index values ​​corresponding to multiple scene images, the average value of the evaluation index values ​​corresponding to the multiple scene images can be calculated, and the calculated average value can be used as the evaluation result of the object detection model. The evaluation result obtained in this way utilizes the evaluation index values ​​of multiple scene images, thus reflecting the features of multiple scene images and ensuring the accuracy of the evaluation result.

[0092] Optionally, when calculating the evaluation result of the object detection model based on the evaluation index values ​​corresponding to multiple scene images, the evaluation index values ​​corresponding to the multiple scene images can be weighted and the calculated value can be used as the evaluation result of the object detection model. Specifically, when weighting the evaluation index values ​​corresponding to multiple scene images, the weight corresponding to each scene image can be determined first, and the sum of the weights corresponding to multiple scene images can be 1; then, the evaluation index value corresponding to each scene image can be multiplied by its corresponding weight to obtain the weighted evaluation index value corresponding to each scene image; then, the sum of the weighted evaluation index values ​​corresponding to multiple scene images can be calculated, and the calculated sum can be used as the evaluation result of the object detection model. The weight corresponding to each scene image can be set by the evaluator. This method of determining the evaluation results first utilizes the evaluation index values ​​of multiple scene images, thus reflecting the characteristics of multiple scene images and ensuring the accuracy of the evaluation results. Second, the evaluators determine the weights of each scene image, allowing them to set appropriate weights according to actual needs. Based on this, the final evaluation results obtained according to the set weights can better meet the evaluation needs of the evaluators.

[0093] Figure 4 This is a flowchart of an evaluation method for a target detection model provided in another exemplary embodiment of this disclosure, such as... Figure 4 As shown, the evaluation method for the target detection model provided in this embodiment may include the following steps S410 to S470.

[0094] S410: Obtain at least one target scenario parameter of the target scenario use case.

[0095] S420: Obtain the parameter values ​​of the target scene parameters.

[0096] S430: Run the target scenario test case based on the parameter values ​​of the target scenario parameters, and obtain at least one scenario image and the baseline annotation of the scenario image during the execution of the target scenario test case.

[0097] S440: By predicting objects in a scene image using an object detection model that runs in the target scene use case, the predicted annotation of the scene image is obtained.

[0098] S450: Determine the evaluation result of the object detection model based on the baseline and predicted annotations corresponding to at least one scene image.

[0099] S460: Determine whether the evaluation result meets the preset conditions, which are the termination conditions for boundary detection; if the evaluation result does not meet the preset conditions, execute S470; if the evaluation result meets the preset conditions, end the evaluation.

[0100] S470: In response to the evaluation result not meeting the preset conditions, adjust the parameter value of at least one target scene parameter and return to execute S430 until the evaluation result meets the preset conditions.

[0101] Boundary detection refers to performance boundary detection, which involves detecting the performance boundaries of a model. If the evaluation results of the target detection model meet the preset boundary detection termination conditions, it indicates that the boundary detection of the target detection model has been completed, and the performance boundaries of the target detection model can then be determined based on the detection results.

[0102] By adjusting the parameter values ​​of the target scene, the scene effect of the target scene use case can be changed, thereby obtaining new scene images. The target detection model can then be re-evaluated based on these new images, updating the evaluation results. In this way, after one or more parameter adjustments, the evaluation results can meet preset conditions, thus achieving boundary detection of the target detection model. According to this embodiment, the performance boundaries of the target detection model can be found efficiently and accurately.

[0103] It should be understood that the implementation methods of steps S410 to S450 are the same as those of steps S110 to S150, and will not be repeated here.

[0104] In some embodiments, the preset conditions in step S460 can be set according to the relationship between the evaluation result and the model performance. For example, if the magnitude of the evaluation result of the object detection model is positively correlated with the model performance (i.e., the larger the evaluation result, the better the model performance), then the preset condition can be set to the evaluation result being less than or equal to a boundary threshold. Based on this, when determining whether the evaluation result meets the preset conditions, the evaluation result can be compared with the preset boundary threshold. If the evaluation result is greater than the boundary threshold, it is determined that the evaluation result does not meet the preset conditions; if the evaluation result is less than or equal to the boundary threshold, it is determined that the evaluation result meets the preset conditions. As another example, if the magnitude of the evaluation result of the object detection model is negatively correlated with the model performance (i.e., the smaller the evaluation result, the better the model performance), then the preset condition can be the evaluation result being greater than or equal to a boundary threshold. Based on this, when determining whether the evaluation result meets the preset conditions, the evaluation result can be compared with the preset boundary threshold. If the evaluation result is greater than or equal to the boundary threshold, it is determined that the evaluation result meets the preset conditions; if the evaluation result is less than the boundary threshold, it is determined that the evaluation result does not meet the preset conditions. The boundary threshold can be set according to the evaluation index used to calculate the evaluation result and in combination with the actual situation.

[0105] This method allows for accurate determination of whether the performance limits of the object detection model have been reached.

[0106] In some embodiments, step S470 may utilize a simulated annealing algorithm to adjust the parameter values ​​of at least one target scene parameter. For example... Figure 5 As shown, after calculating the evaluation results of the target detection model N times (N is an integer greater than or equal to 2), before performing the N+1th calculation on the evaluation results, the parameter values ​​of at least one target scene parameter are adjusted using the simulated annealing algorithm, which may include the following steps S510 to S570.

[0107] S510: Determine the first scene parameter that needs to be adjusted among the target scene parameters.

[0108] Optionally, assessors can select one or more scene parameters from the target scene parameters for parameter value adjustment based on actual needs. The scene parameter selected by the assessor from the target scene parameters will be used as the first scene parameter. For example, when the target scene parameters include weather parameters, object static attribute parameters, and object dynamic attribute parameters, the assessor can select only the weather parameter as the first scene parameter. Alternatively, all target scene parameters can be designated as the first scene parameter requiring parameter value adjustment.

[0109] S520: Obtain the first parameter value of the first scene parameter. The first parameter value is the parameter value of the first scene parameter when calculating the evaluation result of the object detection model for the Nth time.

[0110] S530: Determine whether the model performance indicated by the second evaluation result is lower than the model performance indicated by the first evaluation result; if yes, execute S540, otherwise execute S550.

[0111] The second evaluation result is the evaluation result of the target detection model obtained in the Nth calculation, and the first evaluation result is the evaluation result of the target detection model obtained in the (N-1)th calculation.

[0112] Depending on the evaluation metrics used to calculate the evaluation results, the method for determining whether the model performance indicated by the second evaluation result is lower than that indicated by the first evaluation result varies. For example, for evaluation metrics that are positively correlated with model performance, such as mAP and accuracy, if the second evaluation result is less than the first evaluation result, it is determined that the model performance indicated by the second evaluation result is lower than that indicated by the first evaluation result; if the second evaluation result is greater than or equal to the first evaluation result, it is determined that the model performance indicated by the second evaluation result is not lower than that indicated by the first evaluation result. For evaluation metrics that are negatively correlated with model performance, such as error rate, if the second evaluation result is greater than the first evaluation result, it is determined that the model performance indicated by the second evaluation result is lower than that indicated by the first evaluation result; if the second evaluation result is less than or equal to the first evaluation result, it is determined that the model performance indicated by the second evaluation result is not lower than that indicated by the first evaluation result.

[0113] S540: In response to the second evaluation result indicating that the model performance is lower than the first evaluation result indicating that the model performance is lower, determine to accept the first parameter value and execute S560.

[0114] When it is determined that the model performance indicated by the second evaluation result is lower than the model performance indicated by the first evaluation result, the probability of the first parameter value being accepted is determined to be 1, and the first parameter value is directly accepted.

[0115] S550: In response to the second evaluation result indicating that the model performance is higher than the first evaluation result indicating that the model performance is higher, calculate the probability that the first parameter value is accepted, and determine whether to accept the first parameter value based on the probability.

[0116] Optionally, calculating the probability that the first parameter value is accepted includes: using a preset probability calculation formula, calculating the probability that the first parameter value is accepted based on the first evaluation result, the second evaluation result, and the obtained current annealing temperature. For example, the probability calculation formula is shown below.

[0117]

[0118] Where P represents the probability that the first parameter value is accepted, e represents the natural index, and f k+1 This indicates the second evaluation result, f k This represents the first evaluation result, and T represents the annealing temperature. The annealing temperature is calculated according to the formula shown below.

[0119] T(k) = rateT(k+1)

[0120] Where T(k) represents the annealing temperature of the k-th iteration of the simulated annealing algorithm, T(k+1) represents the annealing temperature of the (k+1)-th iteration of the simulated annealing algorithm, and rate represents the annealing rate, rate∈(0,1). Optionally, rate can be set to a number between 0.8 and 0.99, which allows for sufficient transfer attempts at each temperature during the annealing process.

[0121] Before adjusting the parameter values ​​of the first scene using the simulated annealing algorithm, the relevant parameters of the simulated degradation algorithm can be initialized first. During initialization, the maximum number of iterations of the simulated annealing algorithm can be set, along with appropriate annealing temperature and annealing rate. A sufficiently high T(0) can be chosen as the initial value of the annealing temperature during initialization, ensuring that all state transitions can be accepted. The annealing rate (rate) is an adjustable parameter; subsequent iterations can update the degradation temperature based on the rate, for example, by gradually decreasing the value of the annealing temperature T.

[0122] If the model performance indicated by the second evaluation result is higher than that indicated by the first evaluation result, the first parameter value will not be discarded directly, but rather accepted with probability. Thus, using simulated annealing to adjust parameter values ​​ensures that the model avoids getting trapped in local minima when searching for the boundary of the object detection model. When accepting the first parameter value based on probability, it is possible to accept the first parameter value or not.

[0123] S560: In response to receiving the first parameter value, adjust the parameter value of the first scene parameter according to the first parameter value.

[0124] A corresponding parameter perturbation step size can be pre-set for each target scene parameter. When adjusting the parameter value of the first scene parameter based on its first parameter value, the parameter perturbation step size corresponding to the first scene parameter can be obtained. Then, the first parameter value is added to the parameter perturbation step size to obtain a first value, and the first parameter value is subtracted from the parameter perturbation step size to obtain a second value. The parameter interval formed by the first and second values ​​is used as the parameter value range. A value is randomly selected within this parameter value range, and any randomly selected value that is different from the first parameter value is used as the adjusted parameter value.

[0125] S570: In response to not accepting the first parameter value, obtain the second parameter value of the previously accepted first scene parameter, and adjust the parameter value of the first scene parameter according to the second parameter value.

[0126] Specifically, obtaining the second parameter value of the previously received first scene parameter can be achieved by obtaining the parameter value of the last first scene parameter received during the previous scene parameter adjustment as the second parameter value. For example, taking the first scenario parameter as rainfall and the evaluation index as mAP, in the (N-2)th calculation of the evaluation result, the parameter value corresponding to rainfall is 0, and the evaluation result is 0.7; in the (N-1)th calculation of the evaluation result, the parameter value corresponding to rainfall is 5, and the evaluation result is 0.65; before the Nth calculation of the evaluation result, the parameter value 5 is accepted because 0.65 is less than 0.7, and the parameter value corresponding to rainfall is adjusted to 7 based on parameter value 5. Then, in the Nth calculation of the evaluation result, the parameter value corresponding to rainfall is 7, and the evaluation result is 0.67; before the N+1th calculation of the evaluation result, the probability of parameter value 7 being accepted is calculated because 0.67 is greater than 0.65. If the parameter value 7 is accepted based on the calculated probability, the parameter value of rainfall is adjusted based on parameter value 7 before the N+1th calculation. If the parameter value 7 is not accepted based on the calculated probability, the parameter value of rainfall is adjusted based on parameter value 5 used in the (N-1)th calculation before the N+1th calculation.

[0127] Alternatively, a corresponding parameter perturbation step size can be pre-set for each target scene parameter. When adjusting the parameter value of the first scene parameter based on the second parameter value, the parameter perturbation step size corresponding to the first scene parameter can be obtained. Then, the second parameter value is added to the parameter perturbation step size to obtain the third value, and the second parameter value is subtracted from the parameter perturbation step size to obtain the fourth value. The parameter interval formed by the third and fourth values ​​is used as the parameter value range. A value is randomly selected within this parameter value range, and any randomly selected value that is different from the second parameter value is used as the adjusted parameter value.

[0128] The above method is mainly for adjusting the parameter values ​​of the first scene parameters after two or more (including two) evaluation results have been calculated, before the next evaluation result calculation. However, for cases where only one evaluation result calculation has been completed, i.e., only the first calculation of the evaluation result has been completed, when adjusting the parameter values ​​of the first scene parameters before the second calculation, the parameter values ​​from the first calculation (hereinafter referred to as the third parameter value) can be directly accepted by default. When adjusting the parameter values ​​of the first scene parameters, the parameter perturbation step size corresponding to the first scene parameter is obtained. Then, the third parameter value is added to the parameter perturbation step size to obtain the fifth value, and the third parameter value is subtracted from the parameter perturbation step size to obtain the sixth value. The parameter interval formed by the fifth and sixth values ​​is used as the parameter value range. A value is randomly selected within this parameter value range, and any randomly selected value different from the third parameter value is used as the adjusted parameter value, thereby achieving the adjustment of the parameter value.

[0129] Furthermore, optionally, after adjusting the parameter values, the newly adjusted target scene parameters can be stored. In this way, after the evaluation results meet the preset conditions, the latest parameter values ​​can be obtained, and then the boundary scenes that the target detection model can adapt to can be determined based on the parameter values.

[0130] The following are experimental results generated when evaluating an object detection model using the evaluation method provided in this embodiment of the present disclosure and applying it to an autonomous driving scenario use case. This object detection model is used for vehicle detection.

[0131] Figure 6 This image shows the initial scene image and the model's recognition result during the first round of evaluation of the object detection model. The left side shows the initial scene image, and the right side shows the model's recognition result after object detection. Each evaluation result is counted as one round. After the first round of evaluation, the mAP value of the object detection model is 0.406.

[0132] Figure 7This image shows the initial scene and the model's recognition result after the 50th round of evaluation of the object detection model. The left side shows the initial scene image, and the right side shows the model's recognition result after object detection. After the 50th round of evaluation, the mAP value of the object detection model is 0.262. Compared to the first round, the vehicles in this round are faster, smaller, and their positions have changed. Figure 7 The left side shows the changes in the initial state of the scene, with a significant change in the vehicle's position. Figure 7 The right side shows the change in the model's recognition performance. Due to the smaller size of the vehicle and its different position, the recognition performance deteriorates.

[0133] Figure 8 This image shows the initial scene and the model's recognition result after the 100th round of evaluation for the object detection model. The left side shows the initial scene, and the right side shows the model's recognition result after object detection. After the 100th round of evaluation, the mAP value of the object detection model is 0.190. Compared to the first round, this round saw rain, faster vehicle speeds, smaller vehicle sizes, and changes in vehicle positions. Figure 8 The left side shows the change in the initial state of the scene, and you can observe that the weather has changed to rainy. Figure 8 The right side shows the changes in the model's recognition performance. Due to the smaller size of the vehicles and the influence of rainy weather, the vehicle category was misidentified, resulting in a worse recognition effect.

[0134] The above experiments show that as the scene parameters in the autonomous driving scenario are adjusted, the first evaluation result of the target detection model also changes. This demonstrates that by adjusting the scene parameters, the evaluation result of the model can be adjusted, thereby enabling boundary testing of the model.

[0135] Figure 9 This is a block diagram illustrating an evaluation apparatus for a target detection model according to an exemplary embodiment. (Refer to...) Figure 9 The device 900 includes: a parameter acquisition module 910, a parameter value acquisition module 920, an image acquisition module 930, an object detection module 940, and a model evaluation module 950.

[0136] The parameter acquisition module 910 is used to acquire at least one target scenario parameter of the target scenario use case, wherein the target scenario parameter refers to the scenario parameter whose parameter value is adjustable in the target scenario use case.

[0137] The parameter value acquisition module 920 is used to acquire the parameter values ​​of the target scene parameters.

[0138] Image acquisition module 930 is used to run the target scene use case according to the parameter value of the target scene parameter, and acquire at least one scene image and the reference label of the scene image during the execution of the target scene use case, wherein the label is used to identify objects in the scene image.

[0139] The object detection module 940 is used to predict objects in the scene image by using an object detection model running in the target scene use case, and to obtain the predicted annotation of the scene image.

[0140] The model evaluation module 950 is used to calculate the evaluation result of the target detection model based on the baseline annotation and prediction annotation corresponding to at least one of the scene images. The evaluation result is used to indicate the prediction accuracy performance of the target detection model.

[0141] This embodiment provides an evaluation device for an object detection model. First, it obtains the parameter values ​​of the target scene parameters in the target scene use cases. Then, it runs the target scene use cases based on the parameter values, and obtains at least one scene image and its baseline annotation during the execution of the target scene use cases. Next, it uses the object detection model to predict objects in the scene images, obtaining predicted annotations for the scene images. Finally, it determines the evaluation result of the object detection model based on the baseline annotations and predicted annotations corresponding to at least one scene image. According to this embodiment, it is possible to simulate real-world scenarios based on scene parameters and scene use cases, thereby evaluating the object detection model using scene images from the simulated scenario. Compared to traditional evaluation methods using adversarial examples and formal verification, the evaluation method of this disclosure incorporates the constraints of real-world conditions, making it more reasonable, more practical, and resulting in more accurate model evaluation results.

[0142] Optionally, the device 900 further includes a parameter value adjustment module, configured to: terminate the evaluation of the target detection model in response to the evaluation result meeting a preset condition, wherein the preset condition is a boundary detection termination condition; and adjust the parameter value of at least one of the target scene parameters in response to the evaluation result not meeting the preset condition, and re-execute the step of running the target scene use case according to the parameter value of the target scene parameter until the evaluation result meets the preset condition.

[0143] Optionally, the parameter value adjustment module is further configured to: determine a first scene parameter among the target scene parameters that needs parameter adjustment; obtain a first parameter value of the first scene parameter, wherein the first parameter value is the parameter value of the first scene parameter when the evaluation result of the target detection model is calculated for the Nth time, and N is an integer greater than or equal to 2; determine to accept the first parameter value in response to the model performance indicated by the second evaluation result being lower than the model performance indicated by the first evaluation result; wherein the second evaluation result is the evaluation result of the target detection model calculated for the Nth time, and the first evaluation result is the evaluation result of the target detection model calculated for the (N-1)th time; calculate the probability that the first parameter value is accepted in response to the model performance indicated by the second evaluation result being higher than the model performance indicated by the first evaluation result, and determine whether to accept the first parameter value based on the probability; adjust the parameter value of the first scene parameter according to the first parameter value in response to accepting the first parameter value; and adjust the parameter value of the first scene parameter according to the second parameter value in response to not accepting the first parameter value.

[0144] Optionally, the parameter value adjustment module is further configured to: calculate the probability that the first parameter value is accepted based on the first evaluation result, the second evaluation result, and the obtained current annealing temperature using a preset probability calculation formula; the probability calculation formula is as follows:

[0145]

[0146] Wherein, P represents the probability that the first parameter value is accepted, e represents the natural index, and f k+1 This represents the second evaluation result, where f k This represents the first evaluation result, where T represents the annealing temperature; the annealing temperature is calculated according to the following formula:

[0147] T(k) = rateT(k+1)

[0148] Wherein, T(k) represents the annealing temperature of the k-th iteration, T(k+1) represents the annealing temperature of the (k+1)-th iteration, rate represents the annealing rate, and rate∈(0,1).

[0149] Optionally, the model evaluation module 950 is configured to: for each scene image, determine the evaluation index value corresponding to the scene image based on the baseline annotation and prediction annotation corresponding to the scene image; wherein, the evaluation index value refers to the evaluation value of the scene image under the corresponding evaluation index, and the evaluation index includes at least one of the following: average precision across all classes (mAP), accuracy, and error rate; and calculate the evaluation result of the target detection model based on the evaluation index value corresponding to at least one scene image.

[0150] Optionally, the target scenario use case is an autonomous driving scenario use case; the image acquisition module 930 is used to: generate a target scenario including a scenario map, objects, and a main vehicle based on the scenario configuration file of the target scenario use case and the parameter values ​​of the target scenario parameters; control the main vehicle to move in the scenario map according to a preset route, and control the main vehicle to collect at least one scenario image during the movement; and add the reference label to the corresponding scenario image for each scenario image.

[0151] Optionally, the target scene parameters include at least one of the following: weather parameters, object static attribute parameters, and object dynamic attribute parameters; wherein, the weather parameters include at least one of the following: rainfall, cloud cover, solar altitude angle, and solar azimuth angle; the object static attribute parameters include at least one of the following: size and color; and the object dynamic attribute parameters include at least one of the following: speed and orientation.

[0152] The target detection model evaluation device provided in this application embodiment can implement all the processes implemented in any of the above method embodiments, and will not be described again here to avoid repetition.

[0153] Figure 10 This is a block diagram illustrating an electronic device 1000 according to an exemplary embodiment. The electronic device 1000 may be a computer device, a laptop computer, a server, a vehicle controller, an in-vehicle terminal, an in-vehicle computer, or other types of electronic devices.

[0154] Reference Figure 10 The electronic device 1000 may include at least one processor 1010 and a memory 1020. The processor 1010 can execute instructions stored in the memory 1020. The processor 1010 is communicatively connected to the memory 1020 via a data bus. In addition to the memory 1020, the processor 1010 can also be communicatively connected to an input device 1030, an output device 1040, and a communication device 1050 via the data bus.

[0155] Processor 1010 can be any conventional processor. Processors may include central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), systems on chips (SoCs), application-specific integrated circuits (ASICs), or combinations thereof.

[0156] The memory 1020 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0157] In this embodiment of the present disclosure, the memory 1020 stores executable instructions, and the processor 1010 can read the executable instructions from the memory 1020 and execute the instructions to implement all or part of the steps of the target detection model evaluation method in the exemplary embodiment described above.

[0158] In addition to the methods and apparatus described above, exemplary embodiments of this disclosure also include a computer program product or a computer-readable storage medium storing the computer program product. The computer product includes computer program instructions that can be executed by a processor to perform all or part of the steps described in the exemplary embodiments above.

[0159] Computer program products can be written in any combination of one or more programming languages ​​to perform operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages, and scripting languages ​​(e.g., Python). The program code can be executed entirely on a user's computing device, partially on a user's device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0160] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media include: static random access memory (SRAM) having one or more electrically connected wires, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk, or any suitable combination thereof.

[0161] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of this disclosure. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. An evaluation method for a target detection model, characterized in that, The method includes: Obtain at least one target scenario parameter of the target scenario use case, wherein the target scenario parameter refers to a scenario parameter whose parameter value is adjustable in the target scenario use case; Obtain the parameter values ​​of the target scene parameters; Run the target scenario use case according to the parameter values ​​of the target scenario parameters, and obtain at least one scene image and the baseline annotation of the scene image during the execution of the target scenario use case. The annotation is used to identify objects in the scene image. The predicted labels of the scene images are obtained by predicting objects in the scene images using an object detection model that runs in the target scene use case. The evaluation result of the target detection model is calculated based on the baseline annotation and prediction annotation corresponding to at least one of the scene images. The evaluation result is used to indicate the prediction accuracy performance of the target detection model. The step of calculating the evaluation result of the object detection model based on the baseline annotation and prediction annotation corresponding to at least one of the scene images further includes: In response to the evaluation result meeting a preset condition, the evaluation of the target detection model ends, where the preset condition is a boundary detection termination condition; In response to the evaluation result not meeting the preset condition, the parameter value of at least one of the target scenario parameters is adjusted, and the process is repeated starting from the step of running the target scenario use case according to the parameter value of the target scenario parameter, until the evaluation result meets the preset condition; wherein, adjusting the parameter value of at least one of the target scenario parameters includes: Determine the first scene parameter among the target scene parameters that needs parameter adjustment; Obtain the first parameter value of the first scene parameter, where the first parameter value is the parameter value of the first scene parameter when calculating the evaluation result of the target detection model for the Nth time, and N is an integer greater than or equal to 2; In response to the second evaluation result indicating that the model performance is lower than the first evaluation result indicating that the model performance is lower, it is determined that the first parameter value is accepted; wherein, the second evaluation result is the evaluation result of the target detection model obtained in the Nth calculation, and the first evaluation result is the evaluation result of the target detection model obtained in the (N-1)th calculation; In response to the second evaluation result indicating that the model performance is higher than the first evaluation result indicating that the model performance is higher, the probability that the first parameter value is accepted is calculated, and the first parameter value is determined to be accepted based on the probability. In response to accepting the first parameter value, the parameter value of the first scene parameter is adjusted according to the first parameter value; In response to not accepting the first parameter value, the second parameter value of the previously accepted first scene parameter is obtained, and the parameter value of the first scene parameter is adjusted according to the second parameter value.

2. The method according to claim 1, characterized in that, The calculation of the probability that the first parameter value is accepted includes: Using a preset probability calculation formula, the probability that the first parameter value is accepted is calculated based on the first evaluation result, the second evaluation result, and the obtained current annealing temperature; the probability calculation formula is as follows: Wherein, P represents the probability that the first parameter value is accepted, e represents the natural index, and f k+1 This represents the second evaluation result, where f k This represents the first evaluation result, where T represents the annealing temperature; the annealing temperature is calculated according to the following formula: T(k) = rateT(k+1) Wherein, T(k) represents the annealing temperature of the k-th iteration, T(k+1) represents the annealing temperature of the (k+1)-th iteration, rate represents the annealing rate, and rate∈(0,1).

3. The method according to claim 1, characterized in that, The step of calculating the evaluation result of the object detection model based on the baseline annotation and predicted annotation corresponding to at least one of the scene images includes: For each scene image, an evaluation index value is determined based on the baseline annotation and prediction annotation corresponding to the scene image; wherein, the evaluation index value refers to the evaluation value of the scene image under the corresponding evaluation index, and the evaluation index includes at least one of the following: average precision across all classes mAP, accuracy, and error rate; The evaluation result of the target detection model is calculated based on the evaluation index value corresponding to at least one of the scene images.

4. The method according to claim 1, characterized in that, The target scenario use case is an autonomous driving scenario use case; The step of running the target scene test case according to the parameter values ​​of the target scene parameters, and obtaining at least one scene image and the baseline annotation of the scene image during the execution of the target scene test case, includes: Based on the scenario configuration file of the target scenario use case, and combined with the parameter values ​​of the target scenario parameters, a target scenario containing a scenario map, objects, and the main vehicle is generated. The system controls the main vehicle to move along a preset route on the scene map, and controls the main vehicle to capture at least one scene image during the movement. For each of the aforementioned scene images, add the aforementioned baseline annotation to the corresponding scene image.

5. The method according to claim 4, characterized in that, The target scene parameters include at least one of the following: weather parameters, static object attribute parameters, and dynamic object attribute parameters; The weather parameters include at least one of the following: rainfall, cloud cover, solar altitude angle, and solar azimuth angle; the static attribute parameters of the object include at least one of the following: size and color; and the dynamic attribute parameters of the object include at least one of the following: speed and orientation.

6. An evaluation device for a target detection model, characterized in that, The device includes: The parameter acquisition module is used to acquire at least one target scenario parameter of the target scenario use case, wherein the target scenario parameter refers to the scenario parameter whose parameter value is adjustable in the target scenario use case; The parameter value acquisition module is used to acquire the parameter values ​​of the target scene parameters; The image acquisition module is used to run the target scene test case according to the parameter value of the target scene parameter, and to acquire at least one scene image and the baseline annotation of the scene image during the execution of the target scene test case, wherein the annotation is used to identify objects in the scene image; The object detection module is used to predict objects in the scene image by using an object detection model running in the target scene use case, and to obtain the predicted annotation of the scene image. The model evaluation module is used to calculate the evaluation result of the target detection model based on the baseline annotation and prediction annotation corresponding to at least one of the scene images. The evaluation result is used to indicate the prediction accuracy performance of the target detection model. The parameter value adjustment module is used to: in response to the evaluation result meeting a preset condition, end the evaluation of the target detection model, wherein the preset condition is a boundary detection termination condition; in response to the evaluation result not meeting the preset condition, adjust the parameter value of at least one of the target scene parameters, and re-execute the step of running the target scene use case according to the parameter value of the target scene parameter until the evaluation result meets the preset condition; The parameter value adjustment module is further configured to: determine a first scene parameter among the target scene parameters that requires parameter adjustment; obtain a first parameter value of the first scene parameter, wherein the first parameter value is the parameter value of the first scene parameter when the evaluation result of the target detection model is calculated for the Nth time, and N is an integer greater than or equal to 2; determine to accept the first parameter value in response to the model performance indicated by the second evaluation result being lower than the model performance indicated by the first evaluation result; wherein the second evaluation result is the evaluation result of the target detection model calculated for the Nth time, and the first evaluation result is the evaluation result of the target detection model calculated for the (N-1)th time; calculate the probability that the first parameter value is accepted in response to the model performance indicated by the second evaluation result being higher than the model performance indicated by the first evaluation result, and determine whether to accept the first parameter value based on the probability; adjust the parameter value of the first scene parameter according to the first parameter value in response to accepting the first parameter value; and adjust the parameter value of the first scene parameter according to the second parameter value in response to not accepting the first parameter value.

7. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the evaluation method of the target detection model as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the evaluation method for the target detection model as described in any one of claims 1-5.

Images