Perturbation Analysis Method, Device, Electronic Device and Storage Medium of Prediction Model

Abstract

The present disclosure relates to a method, apparatus, electronic device, and storage medium for perturbation analysis of a prediction model, and relates to the technical field of image processing. The method includes: obtaining a test data set and a perturbation parameter set; performing perturbation transformation on test images in the test data set according to each group of perturbation parameters in the perturbation parameter set to obtain a perturbation data set corresponding to each group of perturbation parameters; determining a first accuracy of the prediction model under the perturbation parameters corresponding to the perturbation data set based on each perturbation data set; determining at least one group of target perturbation parameters in the perturbation parameter set for which the change in accuracy is less than or equal to a preset threshold; using the at least one group of target perturbation parameters as perturbation stable points of the prediction model to form a closed convex hull of the prediction model; wherein, when interference occurs under each perturbation stable point in the closed convex hull, the prediction accuracy of the prediction model is stable. The present disclosure accurately determines under what perturbation parameters the prediction model is stable through perturbation analysis of the prediction model.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and in particular, to a method, apparatus, electronic device, and storage medium for perturbance analysis of a prediction model. Background Art

[0002] With the continuous development of deep learning technologies, image prediction models based on deep learning technologies are increasingly widely used, such as being applied to fields such as intelligent robots, autonomous driving, and image processing.

[0003] Currently, in the actual application process of an image prediction model, the input image of the image prediction model is easily perturbed, such as being perturbed by noise, blurring, rotation, and scaling. Since the prediction model usually has poor robustness, the perturbation of the input image will affect the accuracy of the predicted object box output by the image prediction model, resulting in the prediction accuracy of the image prediction model possibly being unstable when perturbations occur. Therefore, analyzing the stability of an image prediction model when the input image is perturbed is an urgent problem to be solved. Summary of the Invention

[0004] To overcome the problems existing in the related art, the present disclosure provides a method, apparatus, electronic device, and storage medium for perturbance analysis of a prediction model.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided a method for perturbance analysis of a prediction model, including:

[0006] Obtaining a test data set and a perturbance parameter set, where the test data set includes multiple test images and the labeled object boxes of each test image, and the perturbance parameter set includes multiple groups of perturbance parameters;

[0007] Performing a perturbance transformation on the test images in the test data set according to each group of perturbance parameters in the perturbance parameter set to obtain a perturbed data set corresponding to each group of perturbance parameters, where the perturbed data set includes multiple perturbed images and the labeled object boxes of each perturbed image, and the perturbed image is an image obtained by performing a perturbance transformation on the test image;

[0008] Based on each perturbed data set, determining a first accuracy of the prediction model under the perturbance parameters corresponding to the perturbed data set, where the first accuracy is used to indicate the overall prediction accuracy of the prediction model for the perturbed images in the perturbed data set;

[0009] In the perturbance parameter set, determining at least one group of target perturbance parameters whose accuracy change amount is less than or equal to a preset threshold, where the accuracy change amount is used to indicate the change in the first accuracy of the prediction model under the perturbance parameters corresponding to it compared to a second accuracy, and the second accuracy is the accuracy of the prediction model for predicting the test data set;

[0010] Use the at least one set of target perturbation parameters as the perturbation stable points of the prediction model to form the closed convex hull of the prediction model; wherein, when interference occurs at each perturbation stable point in the closed convex hull, the prediction accuracy of the prediction model is stable.

[0011] According to a second aspect of the embodiments of the present disclosure, there is provided a perturbation analysis device for a prediction model, including:

[0012] A data acquisition module, configured to acquire a test data set and a perturbation parameter set, where the test data set includes multiple test images and the labeled object boxes of each test image, and the perturbation parameter set includes multiple sets of perturbation parameters;

[0013] A perturbation transformation module, configured to perform perturbation transformation on the test images in the test data set according to each set of perturbation parameters in the perturbation parameter set, to obtain a perturbation data set corresponding to each set of perturbation parameters, where the perturbation data set includes multiple perturbation images and the labeled object boxes of each perturbation image, and the perturbation image is an image obtained by performing perturbation transformation on the test image;

[0014] An accuracy determination module, configured to determine a first accuracy of the prediction model under the perturbation parameters corresponding to the perturbation data set based on each perturbation data set, where the first accuracy is used to indicate the overall prediction accuracy of the prediction model for the perturbation images in the perturbation data set;

[0015] A parameter determination module, configured to determine at least one set of target perturbation parameters in the perturbation parameter set, where the accuracy change amount is less than or equal to a preset threshold, and the accuracy change amount is used to indicate the change in the first accuracy of the prediction model under the perturbation parameters corresponding to it compared to the second accuracy, and the second accuracy is the accuracy of the prediction model for predicting the test data set;

[0016] A closed convex hull generation module, configured to use the at least one set of target perturbation parameters as the perturbation stable points of the prediction model to generate the closed convex hull of the prediction model; wherein, when interference occurs at each perturbation stable point in the closed convex hull, the prediction accuracy of the prediction model is stable.

[0017] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the steps of the perturbation analysis method for the prediction model provided in the first aspect of the present disclosure.

[0018] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the program instructions are executed by a processor, the steps of the perturbation analysis method of the prediction model provided in the first aspect of the present disclosure are implemented.

[0019] The technical solutions provided by the embodiments of the present disclosure at least include the following beneficial effects:

[0020] Through the perturbation analysis of the prediction model, it is possible to accurately determine under what perturbation parameters the prediction model is stable, that is, when the input image is perturbed based on the perturbation parameters, the prediction accuracy of the prediction model tends to remain unchanged (stable), thereby providing guidance for the practical application of the prediction model.

[0021] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure.

[0023] Figure 1 is a flowchart of a perturbation analysis method of a prediction model shown according to an exemplary embodiment.

[0024] Figure 2 is a block diagram of a perturbation analysis device of a prediction model shown according to an exemplary embodiment.

[0025] Figure 3 is a block diagram of an electronic device shown according to an exemplary embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] Hereinafter, the exemplary embodiments will be described in detail with reference to the accompanying drawings.

[0027] It should be noted that the related embodiments and the accompanying drawings are only for describing and explaining the exemplary embodiments provided by the present disclosure, rather than all embodiments of the present disclosure, nor should it be understood that the present disclosure is limited by the related exemplary embodiments.

[0028] It should be noted that the terms "first", "second", etc. used in the present disclosure are only used to distinguish different steps, devices or modules, etc. The related terms neither represent any specific technical meaning nor indicate the order or interdependence relationship between them.

[0029] It should be noted that the modification of the term "at least one" used in the present disclosure is illustrative rather than restrictive. Unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0030] It should be noted that the term "and / or" used in this disclosure is used to describe the association relationship between associated objects and generally represents at least three association relationships. For example, A and / or B can represent at least the following three association relationships: A exists alone, A and B exist simultaneously, and B exists alone.

[0031] It should be noted that the various steps recorded in the method embodiments of this disclosure can be executed in different orders and / or executed in parallel. Unless otherwise specified, the scope of this disclosure is not limited by the order of description of the steps in the relevant embodiments.

[0032] It should be noted that all actions of obtaining signals, information, or data in this disclosure are carried out on the premise of complying with the corresponding data protection regulations and policies of the country where the location is located and with the authorization given by the owner of the corresponding device.

[0033] Figure 1 is a flowchart of a perturbation analysis method for a prediction model shown according to an exemplary embodiment, as Figure 1 shown, the perturbation analysis method of the prediction model is used in an electronic device and includes the following steps S110 to step S150.

[0034] Step S110: Obtain a test data set and a perturbation parameter set. The test data set includes multiple test images and the labeled object boxes of each test image, and the perturbation parameter set includes multiple groups of perturbation parameters.

[0035] In the embodiments of this disclosure, the above prediction model can be any deep learning model that takes images as input data and is trained with a large number of training samples. It should be noted that the prediction model can be applied to fields such as intelligent robots and image processing. Optionally, the prediction model can be an autonomous driving model, an object detection model, an image classification model, etc.

[0036] The above test data set can include multiple test data samples. Each test data sample includes a test image and the labeled object box of the test image. The above test image can be an image collected in an actual test scenario; the above labeled object box can be a box manually labeled for the objects in the test image in the test image, or a box automatically labeled for the objects in the test image by a computer device. And, the above labeled object box can include position information and category information, etc. of the labeled object.

[0037] For example, when the above prediction model is an autonomous driving model, the above test data set may include multiple road images and the labeled object frames of each road image. Each road image is an image of the road in front of the vehicle collected during the actual road driving of the vehicle. The labeled object frames of each road image are frames manually labeled for the objects (such as people, animals, vehicles, traffic lights, and road signs, etc.) in the road image.

[0038] The above perturbation parameter set includes multiple pre-set groups of perturbation parameters. Each group of perturbation parameters may include parameter values under at least one dimension of perturbation indexes, and at least some of the parameter values of different groups of perturbation parameters are different. Each perturbation index in the above at least one dimension of perturbation indexes can be any index that can interfere with the image. Optionally, the perturbation index may include at least one of noise, blur, rotation, scaling, saturation, and hue. Optionally, the parameter values in the above multiple groups of perturbation parameters may be discontinuous; or, the parameter values in the above multiple groups of perturbation parameters may be continuous, so as to make the perturbation analysis of the prediction model more comprehensive.

[0039] It should be noted that the number of test images in the above test data set and the number of groups of perturbation parameters in the perturbation parameter set can be selected according to actual needs and are not limited here.

[0040] Step S120: Perform perturbation transformation on the test images in the test data set according to each group of perturbation parameters in the perturbation parameter set to obtain a perturbation data set corresponding to each group of perturbation parameters.

[0041] In the embodiments of the present disclosure, the perturbation data set includes multiple perturbation images and the labeled object frames of each perturbation image. The perturbation image is an image obtained by performing perturbation transformation on the test image. The above performing perturbation transformation on the test images in the test data set according to each group of perturbation parameters can be understood as using the parameter values of each dimension of perturbation indexes in each group of perturbation parameters to perform a perturbation process corresponding to the dimension of perturbation index on the test image to obtain the perturbation image corresponding to the test image. For example, when the above groups of perturbation parameters include parameter values of perturbation indexes such as blur and rotation, the electronic device uses the blur degree and rotation angle in the perturbation parameters to perform blur and rotation processing on each test image in the test image set to obtain the perturbation image corresponding to each test image.

[0042] It should be noted that when performing perturbation transformation on the above test images, the labeled object frames of each test image are also perturbed accordingly with the test image, so that the information labeled by the labeled object frames of the perturbation image and the test image is kept consistent. For example, when the above test image is rotated clockwise by 90°, each labeled object frame in the above test image is also rotated clockwise by 90° following the test image.

[0043] Step S130: Based on each perturbed dataset, determine a first accuracy of the prediction model under the perturbation parameter corresponding to the perturbed dataset.

[0044] The first accuracy is used to indicate the overall prediction accuracy of the prediction model for the perturbed images in the perturbed dataset. The above determining the first accuracy of the prediction model under the perturbation parameter corresponding to the perturbed dataset based on each perturbed dataset may include: inputting each perturbed image in the perturbed dataset into the prediction model respectively, and the prediction model outputs a predicted object box for each perturbed image; calculating the prediction accuracy of the prediction model for each perturbed image based on the labeled object box and the predicted object box of each perturbed image; determining the first accuracy of the prediction model under the perturbation parameter corresponding to the perturbed dataset based on the prediction accuracies of the prediction model for multiple perturbed images in the perturbed dataset.

[0045] For example, assume that the electronic device obtains M perturbed datasets through perturbation transformation, and each perturbed dataset includes N perturbed images, where M and N are integers greater than 1. Then, the electronic device can input the perturbed images in the e-th perturbed dataset among the M perturbed datasets into the prediction model to obtain the predicted object box of the f-th perturbed image in the e-th perturbed dataset, where e is an integer greater than or equal to 1 and less than or equal to M, and f is an integer greater than or equal to 1 and less than or equal to N; then, the electronic device can calculate the prediction accuracy of the prediction model for the f-th perturbed image in the e-th perturbed dataset based on the predicted object box and the labeled object box of the f-th perturbed image in the e-th perturbed dataset; finally, the electronic device determines the first accuracy of the prediction model under the perturbation parameter corresponding to the e-th perturbed dataset based on the prediction accuracies corresponding to the N perturbed images in the e-th perturbed dataset. In a similar manner, the electronic device can obtain the first accuracies of the prediction model under M groups of perturbation parameters (the perturbation parameters respectively corresponding to the M perturbed datasets).

[0046] Based on the labeled object boxes and predicted object boxes of each perturbed image, the prediction accuracy of the prediction model for each perturbed image can be calculated. For example, it can be calculating the mean of the intersection over union (IoU) between all the labeled object boxes and the predicted object boxes in the perturbed image, and directly using the calculated mean of the IoU as the prediction accuracy of the prediction model for the perturbed image. Alternatively, based on the labeled object boxes and predicted object boxes of each perturbed image, calculating the prediction accuracy of the prediction model for each perturbed image can include: calculating the IoU between all the labeled object boxes and the predicted object boxes in the perturbed image; generating a relationship matrix for the perturbed image based on the IoU between all the labeled object boxes and the predicted object boxes in the perturbed image; determining the mapping classification information from the labeled object boxes to the predicted object boxes of the perturbed image based on the relationship matrix of the perturbed image; and calculating the prediction accuracy of the prediction model for the perturbed image based on the mapping classification information from the labeled object boxes to the predicted object boxes of the perturbed image.

[0047] The relationship matrix of the above-mentioned perturbed image includes the IoU between each predicted object box and each labeled object box of the perturbed image. For example, assume that for image i (i is a positive integer greater than or equal to 1 and less than or equal to N) in a perturbed dataset including N (N is an integer greater than 1) perturbed images, it is labeled with Q i (Q i is a positive integer) labeled object boxes, and the set of labeled object boxes QB i of image i can be represented as qbox i 1 , qbox i 2 …qbox i Qi ; by inputting image i into the prediction model, P i predicted object boxes are labeled in image i by the prediction model, then the set of predicted object boxes PB i of image i can be represented as pbox i 1 , pbox i 2 …pbox i Qi ; then, the relationship matrix of the above-mentioned perturbed image can include the IoU between each labeled object box in QB i and each predicted object box in PB i .

[0048] The above-mentioned mapping classification information of the labeled object box to the predicted object box for the perturbation image can be determined based on the relationship matrix of the perturbation image. For each predicted object box in the perturbation image, the labeled object box with the largest intersection over union (IoU) in the perturbation image is determined, and the mapping relationship between the predicted object box and the determined labeled object box is established. This mapping relationship indicates that the predicted object box and the determined labeled object box are of the same object. Based on this, the mapping classification information of the labeled object box to the predicted object box for the above-mentioned perturbation image can include the mapping relationships between all the predicted object boxes and the labeled object boxes in the perturbation image.

[0049] For example, in the relationship matrix of the above-mentioned perturbation image, for the set of predicted object boxes PB i in the perturbation image, for the predicted object box pbox i j , the set of labeled object boxes QB i in it can be determined. The k i,j -th labeled object box in QB has a mapping relationship with pbox, where the k i,j -th labeled object box has the largest IoU with the predicted object box pbox i j , and it can be expressed by the following formula.

[0050] k i,j = max k IOU(qbox i k , pbox i j )

[0051] where k is any integer from 1 to Q i ; then, the mapping classification information of the labeled object box to the predicted object box for this perturbation image includes the mapping relationships between the predicted object boxes in PB i and the labeled object boxes in QB i .

[0052] In some embodiments, calculating the prediction accuracy of the prediction model for the perturbed image based on the mapping classification information from the labeled object bounding boxes to the predicted object bounding boxes of the perturbed image includes: calculating the accuracy of the prediction model for the perturbed image under at least one prediction accuracy metric based on the mapping classification information from the labeled object bounding boxes to the predicted object bounding boxes of the perturbed image; and calculating the prediction accuracy of the prediction model for the perturbed image based on the accuracy of the prediction model for the perturbed image under at least one prediction accuracy metric. Wherein, the prediction accuracy metrics include at least one of the following: a first metric for indicating the accuracy of marking the object bounding box; a second metric for indicating the accuracy of object classification; a third metric for indicating the ratio of unrecognized labeled object bounding boxes; and a fourth metric for indicating the ratio of misrecognized predicted object bounding boxes. In this embodiment, the accuracy under at least one of the above first, second, third, and fourth prediction accuracy metrics is calculated based on the mapping classification information from the labeled object bounding boxes to the predicted object bounding boxes of the perturbed image, and the prediction accuracy of the perturbed image is calculated based on the accuracy under at least one of the prediction accuracy metrics, thereby making the method for calculating the prediction accuracy of the perturbed image more flexible.

[0053] Exemplarily, the above electronic device may calculate the accuracy of the prediction model for the perturbed image under the first metric based on the mapping classification information from the labeled object bounding boxes to the predicted object bounding boxes of the perturbed image. The accuracy under the first metric is the accuracy of marking the object bounding box, that is, determining the accuracy of each predicted object bounding box predicted by the prediction model in marking the labeled object bounding box in the perturbed image, and taking the average value of the accuracies of all predicted object bounding boxes in marking the labeled object bounding box in the perturbed image as the accuracy under the first metric. Wherein, the accuracy of each predicted object bounding box in marking the labeled object bounding box in the perturbed image may be the maximum intersection over union between the predicted object bounding box and each labeled object bounding box.

[0054] Exemplarily, the above electronic device may calculate the accuracy of the prediction model for the perturbed image under the second metric based on the mapping classification information from the labeled object bounding boxes to the predicted object bounding boxes of the perturbed image. The accuracy under the second metric is the accuracy of object classification, that is, it may be compared whether the object category of the predicted object bounding box predicted by the prediction model is consistent with the object category of the labeled object bounding box with a mapping relationship. If they are consistent, it is 1; otherwise, it is 0, and the proportion of the predicted object bounding boxes with the comparison result of 1 is taken as the accuracy under the second metric.

[0055] Exemplarily, the above electronic device may calculate the accuracy of the prediction model for the perturbed image under the third metric based on the mapping classification information from the labeled object bounding boxes to the predicted object bounding boxes of the perturbed image. The accuracy under the third metric is the ratio of unrecognized labeled object bounding boxes, that is, the proportion of the predicted object bounding boxes that have no mapping relationship with the labeled object bounding boxes among all the predicted object bounding boxes predicted by the prediction model.

[0056] Exemplarily, the above electronic device may calculate the accuracy of the prediction model for the perturbed image under a fourth metric based on the mapping classification information of the labeled object box to the predicted object box of the perturbed image. The accuracy under the fourth metric is the ratio of the misidentified predicted object boxes, that is, among the predicted object boxes obtained by the prediction model, the proportion of the predicted object boxes in which there is no object in the image area framed in all the predicted object boxes.

[0057] Optionally, calculating the prediction accuracy of the prediction model for the perturbed image based on the accuracy of the prediction model for the perturbed image under at least one prediction accuracy metric may be calculated according to a preset calculation method.

[0058] For example, in the case where the above at least one prediction accuracy metric includes a first metric, a second metric, a third metric, and a fourth metric, the mean of the accuracies under the first metric and the second metric may be subtracted from the mean of the accuracies under the third metric and the fourth metric, and the resulting difference is the prediction accuracy of the perturbed image.

[0059] For another example, the above at least one prediction accuracy metric includes a first metric, a second metric, a third metric, and a fourth metric. Calculating the prediction accuracy of the prediction model for the perturbed image based on the accuracy of the prediction model for the perturbed image under at least one prediction accuracy metric may include: performing a weighted sum processing on the accuracy of the prediction model under the first metric and the accuracy under the second metric to obtain a weighted accuracy; determining the difference between the weighted accuracy and the sum of the accuracies of the prediction model under the third metric and the fourth metric as the prediction accuracy of the prediction model for the perturbed image.

[0060] Exemplarily, assume that for the image i in the above perturbed dataset including N perturbed images, the object classification set of the Q i labeled object boxes can be expressed as C QBi , and the object classification corresponding to the label object box qbox i j is QC i j ; the object classification set of the P i predicted object boxes obtained by the prediction model can be expressed as C PBi , and the object classification corresponding to the predicted object box pbox i j is PC i j . The accuracy S Box (QB i , pbox ij ), the accuracy under the first index can be represented by the formula S Box (QB i , pbox i j ) = IOU(qbox i ki,j , pbox i j ); the comparison result between the predicted object box and the labeled object box with a mapping relationship can be expressed as S C (C QBi , pbox i j )(the accuracy under the second index). If the comparison results are consistent, then S C (C QBi , pbox i j ) = 1; otherwise, S C (C QBi , pbox i j ) = 0; the ratio of the predicted model not recognizing the labeled object box in image i is represented as γ i (the accuracy under the third index); the ratio of the predicted object boxes misrecognized by the predicted model in image i is represented as ρ i (the accuracy under the fourth index). Then, the prediction accuracy P ri of the above - mentioned prediction model for image i can be represented by the following formula.

[0061]

[0062] Among them, α represents the weight of the accuracy of the prediction model under the first index; β represents the weight of the accuracy of the prediction model under the second index.

[0063] In this embodiment, to determine the first accuracy of the prediction model under the perturbation data set based on the prediction accuracy of the prediction model for multiple perturbed images in the perturbation data set, it can be to determine the average value of the prediction accuracies of the prediction model for multiple perturbed images in the perturbation data set as the first accuracy of the prediction model under the perturbation parameters corresponding to the perturbation data set.

[0064] For example, for the above - mentioned perturbation data set including N perturbed images, the first accuracy P r of the above - mentioned prediction model under the perturbation parameters corresponding to the perturbation data set can be represented by the following formula.

[0065]

[0066] Step S140: In the set of perturbation parameters, determine at least one set of target perturbation parameters whose accuracy change amount is less than or equal to a preset threshold.

[0067] Among them, the accuracy change amount is used to indicate the change in the first accuracy of the prediction model under the corresponding perturbation parameter compared to the second accuracy. The second accuracy is the accuracy of the prediction model for predicting the test data set, that is, by inputting each test image in the test data set into the prediction model and determining the second accuracy according to the accuracy of the prediction model for predicting each test image. Optionally, the accuracy change amount of each of the above perturbation parameters may be the absolute value of the difference between the first accuracy and the second accuracy of the prediction model under the perturbation parameter. In addition, the above preset threshold may be a change amount threshold preset in the above electronic device according to actual needs, which is not limited here. Through the above step S140, if the accuracy change amount corresponding to a certain set of perturbation parameters in the set of perturbation parameters is less than or equal to the preset threshold, then this set of perturbation parameters is used as a set of target perturbation parameters.

[0068] In some embodiments, the calculation process of the second accuracy includes: inputting each test image in the test data set into the prediction model respectively, and the prediction model outputs the predicted object box of each test image; based on the labeled object box and the predicted object box of each test image, calculate the prediction accuracy of the prediction model for each test image; based on the prediction accuracy of the prediction model for multiple test images, determine the second accuracy. The above calculation of the prediction accuracy of the prediction model for each test image based on the labeled object box and the predicted object box of each test image may be to calculate the mean of the intersection over union between the labeled object box and the predicted object box in the test image, and directly use the calculated mean of the intersection over union as the prediction accuracy of the prediction model for the test image.

[0069] Alternatively, the above calculation of the prediction accuracy of the prediction model for each test image based on the labeled object box and the predicted object box of each test image includes: calculating the intersection over union of the labeled object box and the predicted object box of each test image; generating a relationship matrix of the test image based on the intersection over union of the labeled object box and the predicted object box of each test image; determining the mapping classification information from the labeled object box to the predicted object box of each test image based on the relationship matrix of each test image; calculating the prediction accuracy of the prediction model for each test image based on the mapping classification information from the labeled object box to the predicted object box of each test image.

[0070] The relationship matrix of the above test images may include the intersection over union (IoU) between each predicted object bounding box and each labeled object bounding box of the test images. Based on the relationship matrix of each test image, determining the mapping classification information from the labeled object bounding box to the predicted object bounding box of each test image may be, for each predicted object bounding box in the test image, determining the labeled object bounding box with the largest IoU with it in the test image, and establishing a mapping relationship between the predicted object bounding box and the determined labeled object bounding box, where this mapping relationship represents that the predicted object bounding box and the determined labeled object bounding box are the same object. Among them, the mapping classification information from the labeled object bounding box to the predicted object bounding box of the above test images may include the mapping relationships between all the predicted object bounding boxes and the labeled object bounding boxes in the test image.

[0071] Optionally, calculating the prediction accuracy of the prediction model for each test image based on the mapping classification information from the labeled object bounding box to the predicted object bounding box of each test image includes: calculating the accuracy of the prediction model for each test image under at least one prediction accuracy metric based on the mapping classification information from the labeled object bounding box to the predicted object bounding box of each test image; calculating the prediction accuracy of the prediction model for each test image based on the accuracy of the prediction model for each test image under at least one prediction accuracy metric. Among them, the prediction accuracy metrics include at least one of the following: a first metric for indicating the accuracy of the marked object bounding box; a second metric for indicating the accuracy of object classification; a third metric for indicating the ratio of unrecognized labeled object bounding boxes; a fourth metric for indicating the ratio of misidentified predicted object bounding boxes.

[0072] Exemplarily, the above electronic device may calculate the accuracy of the prediction model for the test image under the first metric based on the mapping classification information from the labeled object bounding box to the predicted object bounding box of the test image. The accuracy under the first metric is the accuracy of the marked object bounding box, that is, determining the accuracy of each predicted object bounding box predicted by the prediction model in marking the labeled object bounding box in the test image, and taking the average value of the accuracies of all the predicted object bounding boxes in marking the labeled object bounding box in the test image as the accuracy under the first metric. Among them, the accuracy of each predicted object bounding box in marking the labeled object bounding box in the test image may be the largest IoU between the predicted object bounding box and each labeled object bounding box.

[0073] Exemplarily, the above electronic device may calculate the accuracy of the prediction model for the test image under the second metric based on the mapping classification information from the labeled object bounding box to the predicted object bounding box of the test image. The accuracy under the second metric is the accuracy of object classification, that is, it can be compared whether the object categories between the predicted object bounding box predicted by the prediction model and the labeled object bounding box with a mapping relationship are the same. If they are the same, it is 1, otherwise it is 0, and taking the proportion of the predicted object bounding boxes with the comparison result of 1 as the accuracy under the second metric.

[0074] Exemplarily, the above-mentioned electronic device may calculate the accuracy of the prediction model for the test image under a third metric based on the mapping classification information of the labeled object box to the predicted object box in the test image. The accuracy under the third metric is the ratio of the unrecognized labeled object boxes, that is, among the predicted object boxes predicted by the prediction model, the proportion of the predicted object boxes that have no mapping relationship with the labeled object boxes in all the predicted object boxes.

[0075] Exemplarily, the above-mentioned electronic device may calculate the accuracy of the prediction model for the test image under a fourth metric based on the mapping classification information of the labeled object box to the predicted object box in the test image. The accuracy under the fourth metric is the ratio of the misrecognized predicted object boxes, that is, among the predicted object boxes predicted by the prediction model, the proportion of the predicted object boxes in which there is no object in the image region framed by them in all the predicted object boxes.

[0076] Optionally, calculating the prediction accuracy of the prediction model for each test image based on the accuracy of the prediction model for each test image under at least one prediction accuracy metric may be to calculate the prediction accuracy of each test image according to a preset calculation method.

[0077] For example, at least one prediction accuracy metric includes a first metric, a second metric, a third metric, and a fourth metric; calculating the prediction accuracy of the prediction model for each test image based on the accuracy of the prediction model for each test image under at least one prediction accuracy metric may include: performing a weighted summation process on the accuracy of the prediction model under the first metric and the accuracy under the second metric to obtain a weighted accuracy; determining the difference between the weighted accuracy and the sum of the accuracy of the prediction model under the third metric and the accuracy under the fourth metric as the prediction accuracy of the prediction model for each test image.

[0078] It should be noted that since the calculation process of the above-mentioned second accuracy is similar to the calculation process of the above-mentioned first accuracy, for other introductions and descriptions of the calculation process of the second accuracy in step S140, please refer to the above step S130 and will not be elaborated here.

[0079] Step S150: Use at least one set of target perturbation parameters as the perturbation stable points of the prediction model to form the closed convex hull of the prediction model.

[0080] When interference occurs at each perturbation stable point in the closed convex hull, the prediction accuracy of the prediction model is stable. The above-mentioned perturbation stable points indicate that when the input image is perturbed with its corresponding perturbation parameters, the change in the prediction accuracy of the prediction model for the perturbed image is small or even unchanged, that is, the prediction accuracy of the prediction model is stable when interference occurs at the perturbation stable points. Of course, correspondingly, if the image input to the above prediction model is perturbed with the perturbation parameters that are not perturbation stable points in the above perturbation parameter set, the prediction accuracy of the above prediction model changes greatly.

[0081] In some embodiments, the above method further includes: determining the target ratio as the stability of the prediction model, where the target ratio is the ratio of the number of perturbation stable points in the closed convex hull to the number of perturbation parameters in the perturbation parameter set. In this embodiment, by determining the above stability, the possibility that the prediction model is in a stable state when perturbed can be intuitively determined.

[0082] In the embodiments of the present disclosure, by performing perturbation transformation on the test images in the test data set using each group of perturbation parameters in the perturbation parameter set, a perturbation parameter set corresponding to each group of perturbation parameters is obtained; then, based on each perturbation parameter set, the first accuracy of the prediction model under the perturbation parameters corresponding to the perturbation parameter set is determined; then, at least one group of target perturbation parameters with an accuracy change amount less than or equal to a preset threshold is determined from the perturbation parameter set; finally, each group of target perturbation parameters is used as the perturbation stable points of the prediction model to form the closed convex hull of the prediction model. In this way, through the perturbation analysis of the prediction model, it can be accurately determined under what perturbation parameters the prediction model is stable, that is, when the input image is perturbed based on the perturbation parameters, the prediction accuracy of the prediction model tends to be unchanged (stable), thereby providing guidance for the practical application of the prediction model.

[0083] Figure 2 is a block diagram of a perturbation analysis device for a prediction model shown according to an exemplary embodiment. Referring to Figure 2 , the device 200 includes: a data acquisition module 210, a perturbation transformation module 220, an accuracy determination module 230, a parameter determination module 240, and a closed convex hull generation module 250.

[0084] The data acquisition module 210 is configured to acquire a test data set and a perturbation parameter set, where the test data set includes multiple test images and the label object boxes of each test image, and the perturbation parameter set includes multiple groups of perturbation parameters.

[0085] The perturbation transformation module 220 is configured to perform perturbation transformation on the test images in the test data set according to each group of perturbation parameters in the perturbation parameter set, so as to obtain a perturbation data set corresponding to each group of perturbation parameters. The perturbation data set includes multiple perturbed images and the labeled object boxes of each perturbed image, and the perturbed image is an image obtained by performing perturbation transformation on the test image.

[0086] The accuracy determination module 230 is configured to determine a first accuracy of the prediction model under the perturbation parameters corresponding to the perturbation data set based on each perturbation data set. The first accuracy is used to indicate the overall prediction accuracy of the prediction model for the perturbed images in the perturbation data set.

[0087] The parameter determination module 240 is configured to determine at least one group of target perturbation parameters in the perturbation parameter set whose accuracy change amount is less than or equal to a preset threshold. The accuracy change amount is used to indicate the change in the first accuracy of the prediction model under the perturbation parameters corresponding thereto compared to the second accuracy, and the second accuracy is the accuracy of the prediction model for predicting the test data set.

[0088] The closed convex hull generation module 250 is configured to use the at least one group of target perturbation parameters as the perturbation stable points of the prediction model to generate a closed convex hull of the prediction model. Wherein, when interference occurs at each perturbation stable point in the closed convex hull, the prediction accuracy of the prediction model is stable.

[0089] In some embodiments, the accuracy determination module 230 is configured to: input each perturbed image in the perturbation data set into the prediction model respectively, and the prediction model outputs the predicted object boxes of each perturbed image; calculate the prediction accuracy of the prediction model for each perturbed image based on the labeled object boxes and the predicted object boxes of each perturbed image; determine the first accuracy of the prediction model under the perturbation parameters corresponding to the perturbation data set based on the prediction accuracy of the prediction model for multiple perturbed images in the perturbation data set.

[0090] In some embodiments, the accuracy determination model 230 is further configured to: calculate the intersection over union between all the labeled object boxes and the predicted object boxes in the perturbed image; generate a relationship matrix of the perturbed image based on the intersection over union between all the labeled object boxes and the predicted object boxes in the perturbed image; determine the mapping classification information from the labeled object box to the predicted object box of the perturbed image based on the relationship matrix of the perturbed image; calculate the prediction accuracy of the prediction model for the perturbed image based on the mapping classification information from the labeled object box to the predicted object box of the perturbed image.

[0091] In some embodiments, the accuracy determination model 230 is further configured to: for each predicted object box in the perturbed image, based on the relationship matrix of the perturbed image, determine a labeled object box in the perturbed image that has the largest intersection over union with the predicted object box; establish a mapping relationship between the predicted object box and the determined labeled object box, where the mapping relationship is used to indicate that the predicted object box and the determined labeled object box are the same object; wherein, the mapping classification information from the labeled object box to the predicted object box of the perturbed image includes: the mapping relationships between all the predicted object boxes and the labeled object boxes in the perturbed image.

[0092] In some embodiments, the accuracy determination model 230 is further configured to: based on the mapping classification information from the labeled object box to the predicted object box of the perturbed image, calculate the accuracy of the prediction model for the perturbed image under at least one prediction accuracy metric; based on the accuracy of the prediction model for the perturbed image under the at least one prediction accuracy metric, calculate the prediction accuracy of the prediction model for the perturbed image; wherein, the prediction accuracy metric includes at least one of the following: a first metric for indicating the accuracy of the marked object box; a second metric for indicating the accuracy of object classification; a third metric for indicating the ratio of unrecognized labeled object boxes; a fourth metric for indicating the ratio of misidentified predicted object boxes.

[0093] In some embodiments, the at least one prediction accuracy metric includes the first metric, the second metric, the third metric, and the fourth metric; the accuracy determination model 230 is further configured to: perform a weighted sum processing on the accuracy of the prediction model under the first metric and the accuracy under the second metric to obtain a weighted accuracy; determine the difference between the weighted accuracy and the sum of the accuracy of the prediction model under the third metric and the accuracy under the fourth metric as the prediction accuracy of the prediction model for the perturbed image.

[0094] In some embodiments, the apparatus is further configured to: determine a target ratio as the stability of the prediction model, where the target ratio is the ratio of the number of perturbed stable points in the closed convex hull to the number of perturbation parameters in the perturbation parameter set.

[0095] The perturbation analysis apparatus for the prediction model provided by the embodiments of the present disclosure can implement each process implemented by the above method embodiments and can achieve the same beneficial effects. To avoid repetition, it will not be elaborated here.

[0096] Figure 3It is a block diagram of an electronic device 300 shown according to an exemplary embodiment. The electronic device 300 may be a computer device, a laptop, a server, a vehicle controller, an in-vehicle terminal, an in-vehicle computer, or other types of electronic devices.

[0097] Referring to Figure 3 , the electronic device 300 may include at least one processor 310 and a memory 320. The processor 310 may execute instructions stored in the memory 320. The processor 310 is communicatively connected to the memory 320 via a data bus. In addition to the memory 320, the processor 310 may also be communicatively connected to an input device 330, an output device 340, and a communication device 350 via the data bus.

[0098] The processor 310 may be any conventional processor. The processor may include, such as, a central processing unit (CPU), a graphic process unit (GPU), a field programmable gate array (FPGA), a system on chip (SOC), an application specific integrated circuit (ASIC), or a combination thereof.

[0099] The memory 320 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as a static random access memory (SRAM), an electrically erasable programmable read-only memory (EEPROM), an erasable programmable read-only memory (EPROM), a programmable read-only memory (PROM), a read-only memory (ROM), a magnetic memory, a flash memory, a magnetic disk, or an optical disk.

[0100] In an embodiment of the present disclosure, executable instructions are stored in the memory 320, and the processor 310 may read the executable instructions from the memory 320 and execute the instructions to implement all or part of the steps of the perturbation analysis method of the prediction model in the above exemplary embodiment.

[0101] In addition to the above methods and devices, an exemplary embodiment of the present disclosure further includes a computer program product or a computer-readable storage medium storing the computer program product. The computer program product includes computer program instructions, and the computer program instructions may be executed by a processor to implement all or part of the steps described in the above exemplary embodiment.

[0102] A computer program product may be written in any combination of one or more programming languages for executing the operations of the embodiments of the present application. The programming languages include object-oriented programming languages such as Java, C++, etc., and also include conventional procedural programming languages such as the "C" language or similar programming languages, as well as scripting languages (such as Python). The program code may be executed entirely on the user computing device, partially on the user device, executed as a stand-alone software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0103] The computer-readable storage medium may employ any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination of the above. More specific examples of the readable storage medium include: static random access memory (SRAM) with one or more wire electrical connections, electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disks or optical disks, or any suitable combination of the above.

[0104] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the present disclosure. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common general knowledge or conventional technical means in the technical field not disclosed in the present disclosure. The specification and examples are only to be considered as exemplary, and the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for perturbation analysis of a prediction model, characterized in that, Including: Obtain a test data set and a perturbation parameter set, where the test data set includes multiple test images and the labeled object boxes of each test image, and the perturbation parameter set includes multiple groups of perturbation parameters; According to each group of perturbation parameters in the perturbation parameter set, perform perturbation transformation on the test images in the test data set to obtain a perturbation data set corresponding to each group of perturbation parameters. The perturbation data set includes multiple perturbed images and the labeled object boxes of each perturbed image, and the perturbed image is an image obtained by performing perturbation transformation on the test image; Based on each perturbation data set, determine a first accuracy of the prediction model under the perturbation parameters corresponding to the perturbation data set. The first accuracy is used to indicate the overall prediction accuracy of the prediction model for the perturbed images in the perturbation data set; In the perturbation parameter set, determine at least one group of target perturbation parameters whose accuracy change amount is less than or equal to a preset threshold. The accuracy change amount is used to indicate the change in the first accuracy of the prediction model under the perturbation parameters corresponding to it compared to the second accuracy, and the second accuracy is the accuracy of the prediction model for predicting the test data set; Use the at least one group of target perturbation parameters as the perturbation stable points of the prediction model to form a closed convex hull of the prediction model; wherein, when interference occurs under each perturbation stable point in the closed convex hull, the prediction accuracy of the prediction model is stable.

2. The method according to claim 1, characterized in that, The determining the first accuracy of the prediction model under the perturbation parameters corresponding to the perturbation data set based on each perturbation data set includes: Input each perturbed image in the perturbation data set into the prediction model, and the prediction model outputs the predicted object boxes of each perturbed image; Based on the labeled object boxes and predicted object boxes of each perturbed image, calculate the prediction accuracy of the prediction model for each perturbed image; Based on the prediction accuracy of the prediction model for multiple perturbed images in the perturbation data set, determine the first accuracy of the prediction model under the perturbation parameters corresponding to the perturbation data set.

3. The method according to claim 2, characterized in that, The calculating the prediction accuracy of the prediction model for each perturbed image based on the labeled object boxes and predicted object boxes of each perturbed image includes: Calculate the intersection over union between all labeled object boxes and predicted object boxes in the perturbed image; Based on the intersection over union between all labeled object boxes and predicted object boxes in the perturbed image, generate a relationship matrix of the perturbed image; Based on the relationship matrix of the perturbed image, determine the mapping classification information from the labeled object box to the predicted object box of the perturbed image; Based on the mapping classification information from the labeled object box to the predicted object box of the perturbed image, calculate the prediction accuracy of the prediction model for the perturbed image.

4. The method according to claim 3, characterized in that, The determining the mapping classification information from the labeled object box to the predicted object box of the perturbed image based on the relationship matrix of the perturbed image includes: For each predicted object box in the perturbed image, based on the relationship matrix of the perturbed image, determine the labeled object box with the largest intersection over union with the predicted object box in the perturbed image; Establish a mapping relationship between the predicted object bounding box and the determined labeled object bounding box, where the mapping relationship is used to indicate that the predicted object bounding box and the determined labeled object bounding box are the same object; Among them, the mapping classification information from the labeled object bounding box to the predicted object bounding box of the perturbed image includes: the mapping relationships between all predicted object bounding boxes and labeled object bounding boxes in the perturbed image.

5. The method according to claim 3, characterized in that, Calculating the prediction accuracy of the prediction model for the perturbed image based on the mapping classification information from the labeled object bounding box to the predicted object bounding box of the perturbed image includes: Based on the mapping classification information from the labeled object bounding box to the predicted object bounding box of the perturbed image, calculating the accuracy of the prediction model for the perturbed image under at least one prediction accuracy metric; Based on the accuracy of the prediction model for the perturbed image under the at least one prediction accuracy metric, calculating the prediction accuracy of the prediction model for the perturbed image; Among them, the prediction accuracy metric includes at least one of the following: The first metric for indicating the accuracy of the marked object bounding box; The second metric for indicating the accuracy of object classification; The third metric for indicating the ratio of unrecognized labeled object bounding boxes; The fourth metric for indicating the ratio of misidentified predicted object bounding boxes.

6. The method according to claim 5, wherein The at least one prediction accuracy metric includes the first metric, the second metric, the third metric, and the fourth metric; Calculating the prediction accuracy of the prediction model for the perturbed image based on the accuracy of the prediction model for the perturbed image under the at least one prediction accuracy metric includes: Performing a weighted summation process on the accuracy of the prediction model under the first metric and the accuracy under the second metric to obtain a weighted accuracy; Determining the difference between the weighted accuracy and the sum of the accuracy of the prediction model under the third metric and the accuracy under the fourth metric as the prediction accuracy of the prediction model for the perturbed image.

7. The method according to claim 1, characterized in that The method further includes: Determining the target ratio as the stability of the prediction model, where the target ratio is the ratio of the number of perturbation stable points in the closed convex hull to the number of perturbation parameters in the perturbation parameter set.

8. A perturbation analysis device for a prediction model, characterized in that, Including: A data acquisition module for acquiring a test data set and a perturbation parameter set, where the test data set includes multiple test images and the labeled object bounding boxes of each test image, and the perturbation parameter set includes multiple groups of perturbation parameters; A perturbation transformation module for performing perturbation transformation on the test images in the test data set according to each group of perturbation parameters in the perturbation parameter set to obtain a perturbation data set corresponding to each group of perturbation parameters, where the perturbation data set includes multiple perturbed images and the labeled object bounding boxes of each perturbed image, and the perturbed image is an image obtained by performing perturbation transformation on the test image; An accuracy determination module for determining a first accuracy of the prediction model under the perturbation parameters corresponding to the perturbation data set based on each perturbation data set, where the first accuracy is used to indicate the overall prediction accuracy of the prediction model for the perturbed images in the perturbation data set; A parameter determination module, configured to determine at least one set of target perturbation parameters in the perturbation parameter set, where the accuracy change amount is less than or equal to a preset threshold, and the accuracy change amount is used to indicate the change in the first accuracy of the prediction model under the corresponding perturbation parameter compared to the second accuracy, and the second accuracy is the accuracy of the prediction model for predicting the test data set; A closed convex hull generation module, configured to use the at least one set of target perturbation parameters as the perturbation stable points of the prediction model to generate a closed convex hull of the prediction model; wherein, when interference occurs at each perturbation stable point in the closed convex hull, the prediction accuracy of the prediction model is stable.

9. An electronic device, characterized in that, Comprising: A processor; A memory for storing executable instructions of the processor; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the perturbation analysis method of the prediction model according to any one of claims 1-7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, the perturbation analysis method of the prediction model according to any one of claims 1-7 is implemented.

Images