Methods, apparatus and electronic equipment for determining the performance of neural network models

Abstract

This invention provides a method, apparatus, and electronic device for determining the performance of a neural network model. The method involves acquiring a set of target image samples, including an initial image sample set and an image enhancement sample set. Each sample contains at least one target object, its size level, and ground truth box information. The neural network model is tested using this set to obtain a first prediction result for the participating samples. This result includes the first prediction box information of the first target object, which, combined with its size level and ground truth box information, determines the performance evaluation parameters of the neural network model. This approach uses an initial image sample set and an image enhancement sample set to test the neural network model. Based on the size level, ground truth box information, and first prediction box information of the first target object in the participating samples, the performance evaluation parameters of the neural network model are determined. This allows for the evaluation of the neural network model's detection performance on different datasets and target objects of different size levels, while simultaneously achieving a quantitative assessment of the neural network model's robustness.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to a method, apparatus, and electronic device for determining the performance of a neural network model. Background Technology

[0002] Pedestrian detection technology is widely used in fields such as intelligent robots, video surveillance, and autonomous driving. In machine learning, robustness is often used to reflect an algorithm's tolerance to data changes. Commonly used objective metrics such as precision, recall, F1-score, and AP can test the detection performance of different pedestrian detection algorithms on a dataset. When evaluating the robustness of a pedestrian detection algorithm, related techniques typically involve first adding noise or blurring to the pedestrian detection dataset, and then comparing the differences in prediction accuracy and recall before and after the perturbation on the same dataset to determine the strength of the model's robustness. However, this method can only provide a qualitative assessment of model robustness and cannot provide a quantitative judgment. Summary of the Invention

[0003] The purpose of this invention is to provide a method, apparatus, and electronic device for determining the performance of a neural network model, so as to quantitatively judge the robustness of the model.

[0004] This invention provides a method for determining the performance of a neural network model. The method includes: performing data augmentation processing on image samples in an initial image sample set to obtain a target image sample set containing the initial image sample set and an image augmentation sample set; wherein, the samples in the target image sample set contain at least one target object, the size level corresponding to the target object, and ground truth box information; applying the target image sample set to test the neural network model to be detected to obtain a first prediction result corresponding to the participating samples in the target image sample set; wherein, the first prediction result includes the first prediction box information corresponding to the first target object in the participating samples; determining performance evaluation parameters of the neural network model based on the size level and ground truth box information corresponding to the first target object in the participating samples, and the first prediction box information corresponding to the first target object; wherein, the performance evaluation parameters are used to characterize the stability of the neural network model for the initial image sample set and the image augmentation sample set, and the detection performance for target objects of different size levels.

[0005] This invention provides a performance determination device for a neural network model. The device includes: an enhancement processing module for performing data enhancement processing on image samples in an initial image sample set to obtain a target image sample set containing the initial image sample set and the image enhancement sample set; wherein, the samples in the target image sample set contain at least one target object, the size level corresponding to the target object, and ground truth box information; a testing module for testing the neural network model to be detected using the target image sample set to obtain a first prediction result corresponding to the participating samples in the target image sample set; wherein, the first prediction result includes the first prediction box information corresponding to the first target object in the participating samples; and a determination module for determining performance evaluation parameters of the neural network model based on the size level and ground truth box information corresponding to the first target object in the participating samples, and the first prediction box information corresponding to the first target object; wherein, the performance evaluation parameters characterize the stability of the neural network model for the initial image sample set and the image enhancement sample set, and the detection performance for target objects of different size levels.

[0006] The present invention provides an electronic device, including a processor and a memory, wherein the memory stores machine-executable instructions that can be executed by the processor, and the processor executes the machine-executable instructions to implement the performance determination method of the neural network model described above.

[0007] The present invention provides a machine-readable storage medium storing machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions cause the processor to implement the performance determination method of any of the above-mentioned neural network models.

[0008] The present invention provides a method, apparatus, and electronic device for determining the performance of a neural network model. The method involves performing data augmentation processing on image samples in an initial image sample set to obtain a target image sample set comprising the initial image sample set and an image augmentation sample set. Each sample in the target image sample set contains at least one target object, the size level corresponding to the target object, and ground truth box information. The target image sample set is then used to test the neural network model to be detected, yielding a first prediction result corresponding to the participating samples in the target image sample set. This first prediction result includes the first prediction box information corresponding to the first target object in the participating samples. Based on the size level and ground truth box information corresponding to the first target object in the participating samples, and the first prediction box information corresponding to the first target object, performance evaluation parameters for the neural network model are determined. These performance evaluation parameters characterize the stability of the neural network model for the initial image sample set and the image augmentation sample set, as well as its detection performance for target objects of different size levels. This method uses an initial image sample set and an image augmentation sample set to test the neural network model. Based on the size level, ground truth box information, and first prediction box information of the first target object in the participating samples, the performance evaluation parameters of the neural network model are determined. This method can evaluate the detection effect of the neural network model on target objects of different datasets and different size levels, while also quantitatively judging the robustness of the neural network model. Attached Figure Description

[0009] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0010] Figure 1 A flowchart illustrating a method for determining the performance of a neural network model, as provided in an embodiment of the present invention;

[0011] Figure 2 This is a schematic diagram of an image sample after data augmentation processing, provided in an embodiment of the present invention.

[0012] Figure 3 A schematic diagram of an image marker provided in an embodiment of the present invention;

[0013] Figure 4 This is another schematic diagram of image labeling provided in an embodiment of the present invention;

[0014] Figure 5 A flowchart illustrating a method for determining the performance of a neural network model, as provided in an embodiment of the present invention;

[0015] Figure 6 A flowchart illustrating another method for determining the performance of a neural network model provided in an embodiment of the present invention;

[0016] Figure 7 This is a schematic diagram of the structure of a neural network model performance determination device provided in an embodiment of the present invention;

[0017] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] In related technologies, when it is necessary to evaluate the robustness of pedestrian detection algorithms, the pedestrian detection dataset is usually first subjected to noise, blurring, or other perturbations. Then, the difference in the model's prediction accuracy, recall, and other metrics before and after the perturbation is compared to determine the strength of the model's robustness. However, this method can only qualitatively evaluate the robustness of the model and lacks an objective metric, making it impossible to quantitatively judge the robustness of the model.

[0020] To facilitate understanding of this embodiment, a method for determining the performance of a neural network model disclosed in this embodiment will first be introduced, such as... Figure 1 As shown, the method includes the following steps:

[0021] Step S102: Perform data augmentation processing on the image samples in the initial image sample set to obtain a target image sample set containing the initial image sample set and the image augmentation sample set; wherein, the samples in the target image sample set contain at least one target object, the size level corresponding to the target object, and the truth box information.

[0022] The initial image sample set typically includes multiple image samples. In practical implementation, data augmentation processing can be performed on the image samples in the initial image sample set. Data augmentation processing methods include at least one of the following: noise addition, mirroring, brightness adjustment, and blurring, such as... Figure 2The diagram illustrates an image sample after data augmentation. The augmented dataset is N times the size of the initial image sample set. The data augmentation method and the amplification factor can be set according to the application scenario. For example, if the initial image sample set is D1, to test the stability of the neural network model in response to different data transformations, the initial image sample set D1 is mirrored to obtain the mirrored image augmented sample set D2. Noise is added to the initial image sample set D1 to obtain the noisy image augmented sample set D3. Similarly, other data augmentation methods can be used to obtain multiple image augmented sample sets. Finally, the target image sample set D{D1,D2,……,Dn} can be obtained, which includes the initial image sample set D1 and the image augmented sample sets D2,……,Dn.

[0023] To evaluate the impact of single-dimensional data augmentation on the robustness of a neural network model, single-dimensional data augmentation can be performed on the initial image sample set D1. For example, taking brightness adjustment only on the initial image sample set D1 as an example, assuming the current brightness level of the initial image sample set D1 is used as a baseline, m higher brightness levels and n lower brightness levels are obtained. The brightness of the m higher brightness levels can increase sequentially, and the brightness of the n lower brightness levels can decrease sequentially. The number of m and n can be the same or different. Based on the m higher brightness levels... The initial image sample set D1 is sequentially brightened at each of the n high brightness levels to obtain the enhanced image sample sets Dg1...Dgm corresponding to each high brightness level. Then, the initial image sample set D1 is sequentially dimmed at each of the n low brightness levels to obtain the enhanced image sample sets Dd1...Ddn corresponding to each low brightness level. Finally, the target image sample set D{Dd1...Ddn,D1,Dg1...Dgm} is obtained. Based on this target image sample set, the impact of brightness on the robustness of the neural network model can be evaluated.

[0024] To improve the accuracy of neural network model performance evaluation, each sample in the target image sample set typically includes at least one target object, such as a pedestrian or a designated object. Each target object is pre-labeled with its corresponding size level and ground truth bounding box information. The ground truth bounding box information can include the target object's true bounding box and its coordinates. The size level can be determined based on the ratio between the target object's true bounding box and the total area of ​​the corresponding image. For example, different sizes of target objects in the target image sample set D can be labeled according to a certain threshold. This threshold can be set according to the application scenario. Taking pedestrians as an example, pedestrians with a true bounding box to total image area ratio less than 1:100 can be labeled as small targets, those between 1:100 and 1:20 as medium targets, and those greater than 1:20 as large targets. Figure 3 A schematic diagram of an image marker is shown. Figure 4 Another schematic diagram of image labeling is shown, in which, Figure 3 Both pedestrians in the middle were marked as targets. Figure 4 The test includes a pedestrian labeled as a large target and a pedestrian labeled as a small target. By labeling target objects of different sizes, we can test whether the neural network model has a stable detection effect on target objects of different sizes.

[0025] Step S104: Apply the target image sample set to test the neural network model to be detected, and obtain the first prediction result corresponding to the participating samples in the target image sample set; wherein, the first prediction result includes the first prediction box information corresponding to the first target object in the participating samples.

[0026] The aforementioned neural network model to be detected can be a pedestrian detection model for detecting pedestrians or a network model for detecting other objects, etc., and is usually a pre-trained neural network model; the aforementioned first target object can be any object to be detected in the participating samples; in specific implementation, image samples in the target image sample set can be input into the neural network model to be detected, and the neural network model to be detected outputs the first prediction result corresponding to each participating sample. The first prediction result usually includes the prediction box corresponding to the first target object in each participating sample, the coordinate value of the prediction box, and other first prediction box information.

[0027] Step S106: Based on the size level and ground truth box information corresponding to the first target object in the participating samples, and the first prediction box information corresponding to the first target object, determine the performance evaluation parameters of the neural network model; wherein, the performance evaluation parameters are used to characterize the stability of the neural network model for the initial image sample set and the image augmentation sample set and the detection performance for target objects of different size levels.

[0028] In practical implementation, the performance evaluation parameters of the neural network model can be calculated based on the size level and ground truth box information of the first target object in the participating samples, as well as the first prediction box information of the first target object predicted through the above steps. These performance evaluation parameters can be values ​​between 0 and 1. The robustness of the neural network model can be confirmed based on these performance evaluation parameters. For example, if the value is higher, it indicates that the neural network model is more robust, the prediction results for the initial image sample set and the image augmentation sample set are less different, the stability is higher, and the detection performance for target objects of different size levels is also better. Conversely, if the value is lower, it indicates that the neural network model is less robust, the prediction results for the initial image sample set and the image augmentation sample set are more different, the stability is lower, and the detection performance for target objects of different size levels is also worse.

[0029] The aforementioned method for determining the performance of a neural network model involves performing data augmentation on image samples in an initial image sample set to obtain a target image sample set containing both the initial and augmented image sample sets. Each sample in the target image sample set contains at least one target object, its corresponding size level, and ground truth box information. The neural network model to be detected is then tested using the target image sample set to obtain a first prediction result corresponding to the participating samples in the target image sample set. This first prediction result includes the first prediction box information corresponding to the first target object in the participating samples. Based on the size level and ground truth box information of the first target object in the participating samples, and the first prediction box information, performance evaluation parameters for the neural network model are determined. These performance evaluation parameters characterize the stability of the neural network model across the initial and augmented image sample sets and its detection performance for target objects of different size levels. This method uses an initial and augmented image sample set to test the neural network model, and determines the performance evaluation parameters based on the size level, ground truth box information, and first prediction box information of the first target object in the participating samples. This allows for the evaluation of the neural network model's detection performance on different datasets and target objects of different size levels, while simultaneously achieving a quantitative assessment of the neural network model's robustness.

[0030] In an optional implementation, the method is based on the method of the above embodiments, such as... Figure 5 As shown, the method includes the following steps:

[0031] Step S502: Perform data augmentation processing on the image samples in the initial image sample set to obtain a target image sample set containing the initial image sample set and the image augmentation sample set; wherein, the samples in the target image sample set contain at least one target object, the size level corresponding to the target object, and the truth box information.

[0032] Step S504: Apply the target image sample set to test the neural network model to be detected, and obtain the first prediction result corresponding to the participating samples in the target image sample set; wherein, the first prediction result includes the first prediction box information corresponding to the first target object in the participating samples.

[0033] Step S506: Based on the size level and ground truth box information corresponding to the first target object in the participating samples, and the first predicted box information corresponding to the first target object, calculate the first average set corresponding to the initial image sample set and the image enhancement sample set, respectively, and the second average set of predicted boxes of participating samples with the same size level; wherein, each average in the first average set is used to characterize the stability of the neural network model for the initial image sample set and the image enhancement sample set; the second average set is used to characterize the detection performance of the neural network model for target objects of different size levels.

[0034] Both the first and second sets of averages include: the average coordinate difference, the average size difference, and model performance parameters. The average coordinate difference is determined by the difference between corresponding coordinate values ​​in the ground truth box and the first predicted box information corresponding to the first target object in the participating samples. The average size difference is determined by the difference between corresponding sizes in the ground truth box and the first predicted box information corresponding to the first target object in the participating samples. The model performance parameters are parameter values ​​determined based on the performance of the predicted box relative to the ground truth box for the first target object in the participating samples. The model performance parameters include at least one of the following: recall, confidence, and precision.

[0035] The aforementioned average coordinate difference can include the average horizontal coordinate difference and the average vertical coordinate difference. The first set of average values ​​typically includes multiple average horizontal coordinate differences and multiple average vertical coordinate differences corresponding to the initial image sample set and the image enhancement sample set, respectively. The average horizontal coordinate difference for each sample set is the average obtained by summing and averaging the differences between the horizontal coordinate of the left boundary of each predicted box and the horizontal coordinate of the corresponding ground truth box in that sample set. The average vertical coordinate difference for each sample set is the average obtained by summing and averaging the differences between the vertical coordinate of the top boundary of each predicted box and the vertical coordinate of the corresponding ground truth box in that sample set.

[0036] The second set of average values ​​typically includes multiple average values ​​of the x-coordinate differences and multiple average values ​​of the y-coordinate differences corresponding to sets of target objects of different size levels. The average value of the x-coordinate difference for each size level is the average obtained by summing and averaging the differences between the left boundary x-coordinate of the predicted bounding box of each target object of that size level and the left boundary x-coordinate of the corresponding ground truth bounding box in the target image sample set. The average value of the y-coordinate difference for each size level is the average obtained by summing and averaging the differences between the top boundary y-coordinate of the predicted bounding box of each target object of that size level and the top boundary y-coordinate of the corresponding ground truth bounding box in the target image sample set.

[0037] Specifically, the difference between the left boundary x-coordinate of each predicted bounding box and the left boundary x-coordinate of the corresponding ground truth bounding box is calculated as follows: For each predicted bounding box, calculate the difference between the left boundary x-coordinate of the predicted bounding box and the left boundary x-coordinate of the corresponding ground truth bounding box; calculate the ratio between this difference and a specified width, and determine this ratio as the difference between the left boundary x-coordinate of the predicted bounding box and the left boundary x-coordinate of the corresponding ground truth bounding box; wherein, the specified width is the width of the ground truth bounding box corresponding to the predicted bounding box.

[0038] The difference between the top boundary ordinate of each predicted bounding box and the top boundary ordinate of the corresponding ground truth bounding box is calculated as follows: For each predicted bounding box, calculate the difference between the top boundary ordinate of the predicted bounding box and the top boundary ordinate of the corresponding ground truth bounding box; calculate the ratio between this difference and a specified height, and determine this ratio as the difference between the top boundary ordinate of the predicted bounding box and the top boundary ordinate of the corresponding ground truth bounding box; wherein, the specified height is the height of the ground truth bounding box corresponding to the predicted bounding box.

[0039] The aforementioned average size difference may include the average width difference and the average height difference. The first set of average values ​​typically includes multiple average width differences and multiple average height differences corresponding to the initial image sample set and the image enhancement sample set, respectively. The average width difference for each sample set is the average obtained by summing and averaging the differences between the width of each predicted box and the width of the corresponding ground truth box in that sample set. The average height difference for each sample set is the average obtained by summing and averaging the differences between the height of each predicted box and the height of the corresponding ground truth box in that sample set.

[0040] The second set of average values ​​typically includes sets of target objects at different size levels, each corresponding to multiple average width differences and multiple average height differences. The average width difference for each size level is the sum of the differences between the width of the predicted bounding box and the width of the corresponding ground truth bounding box for each target object at that size level in the target image sample set. Similarly, the average height difference for each size level is the sum of the differences between the height of the predicted bounding box and the height of the corresponding ground truth bounding box for each target object at that size level in the target image sample set.

[0041] Specifically, the difference between the width of each predicted bounding box and the width of the corresponding ground truth bounding box is calculated as follows: For each predicted bounding box, the difference between the width of the predicted bounding box and the width of the corresponding ground truth bounding box is calculated; the ratio between this difference and a specified width is calculated, and this ratio is determined as the difference between the width of the predicted bounding box and the width of the corresponding ground truth bounding box.

[0042] The difference between the height of each predicted bounding box and the height of its corresponding ground truth bounding box is calculated as follows: For each predicted bounding box, the difference between the height of the predicted bounding box and the height of its corresponding ground truth bounding box is calculated; the ratio between this difference and a specified height is calculated, and this ratio is determined as the difference between the height of the predicted bounding box and the height of its corresponding ground truth bounding box.

[0043] It should be noted that if no corresponding first specified prediction box is predicted for the first specified ground truth box, the left boundary x-coordinate, the top boundary y-coordinate, the width, and the height of the first specified prediction box are zero; if no corresponding second specified ground truth box exists for the second specified prediction box, the left boundary x-coordinate, the top boundary y-coordinate, the width, and the height of the second specified ground truth box are zero. In this case, the specified width needs to be adjusted to the width of the second specified prediction box, and the specified height needs to be adjusted to the height of the second specified prediction box.

[0044] The first set of averages typically includes multiple recall rates, multiple confidence rates, and multiple precision rates corresponding to the initial image sample set and the image enhancement sample set, respectively. The recall rate for each sample set is the average of the sums of the recall rates of each predicted box in that sample set. The confidence rate for each sample set is the average of the sums of the confidence rates of each predicted box in that sample set. The precision rate for each sample set is the average of the sums of the precision rates of each predicted box in that sample set.

[0045] The second set of averages typically includes multiple recall rates, multiple confidence rates, and multiple precision rates corresponding to sets of target objects at different size levels. The recall rate for each size level is the average of the sums of the recall rates of the predicted bounding boxes labeled with each target object at that size level in the target image sample set. The confidence rate for each size level is the average of the sums of the confidence rates of the predicted bounding boxes labeled with each target object at that size level in the target image sample set. The precision rate for each size level is the average of the sums of the precision rates of the predicted bounding boxes labeled with each target object at that size level in the target image sample set.

[0046] For ease of explanation, after the neural network model completes its prediction of the participating samples, the true coordinates and predicted coordinates can be represented by the x-coordinate of the left boundary of the rectangle, the y-coordinate of the top boundary, the width w, and the height h. The difference is the difference between the true coordinates and the predicted coordinates. If a false negative occurs, the difference is the negative true coordinate value; if a false positive occurs, the difference is the predicted coordinate value. Regarding confidence, if a false negative occurs, the model's confidence in that target object is 0. To balance the differences in predicted bounding boxes of different sizes, the differences in x, y, w, and h are divided by the true bounding box height (for y and h) or width (for x and w), and for false positives, they are divided by the predicted bounding box height or width.

[0047] To facilitate understanding, let's continue with the example of the target image sample set D{D1,D2,...,Dn}, where the target object is a pedestrian. We use the neural network model to predict the target image sample set D, obtaining the prediction result for each image, including the confidence score c of the bounding box. p The left boundary x-coordinate of the rectangle p The y-coordinate of the upper boundary of the rectangle p , width of rectangle w p and the height h of the rectangle p Assuming that pedestrians in each image of the target image sample set are labeled as large, medium, or small according to a certain threshold, then in the dataset D{D1,D2,...,Dn} and the labeled large, medium, and small target objects, calculate the average prediction confidence C{C1,C2,...,Cn} for each ground truth bounding box. n C small C medium C big For example, C1 refers to the average prediction confidence of each ground truth bounding box in each sample of the sample set D1, and so on, C... n This refers to the average prediction confidence of each ground truth bounding box in each sample of the sample set Dn; C small This refers to the average predicted confidence of the ground truth bounding boxes corresponding to all labeled small target objects in the sample set D1-Dn, and so on, C big This refers to the average predicted confidence of the ground truth bounding boxes corresponding to all large target objects labeled in the sample set D1-Dn. For ground truth bounding boxes that are missed, the confidence is 0.

[0048] In the dataset D{D1,D2,...,D... n} and the labeled large, medium, and small target objects, calculate the average coordinate difference x{x1, x2, ..., x} between each predicted bounding box and the ground truth bounding box. n ,x small ,x medium ,x big}、y{y1,y2,……,yn ,y small ,y medium ,y big}、w{w1,w2,……,w n ,w small ,w medium ,w big} and h{h1,h2,……,h n ,h small ,h medium ,h big}. Here, x1 refers to the average difference in the x-coordinate between each predicted bounding box and its corresponding ground truth bounding box in sample set D1, and so on, xn refers to the average difference in the x-coordinate between each predicted bounding box and its corresponding ground truth bounding box in sample set Dn; x small This refers to the average difference in the x-coordinate between the predicted bounding boxes and the ground truth bounding boxes for all small target objects marked in D1-Dn, and so on. big This refers to the average difference in x-coordinate between the predicted bounding boxes and the ground truth bounding boxes for all large target objects marked in D1-Dn; the difference in x and w between the predicted and ground truth bounding boxes is... w t x is the actual box width. t x is the x-coordinate of the left boundary of the true bounding box. p w represents the x-coordinate of the left boundary of the prediction box. p Here is the width of the predicted bounding box; the difference between the predicted bounding box and the ground truth bounding box is... Where h t y is the actual bounding box height. t The y-coordinate of the top boundary of the true bounding box. p h represents the ordinate of the upper boundary of the prediction box. p This represents the height of the predicted bounding box. If a missed detection occurs, then x... p w p y p and h p The value is 0; if a false positive occurs, then x is 0. t w t y t and h t It is 0, and w in the difference formula t h t Replace with w p h p .

[0049] With an appropriate IOU (Intersection over Union) threshold (e.g., 0.5), the model prediction results are calculated on the dataset D{D1, D2, ..., D...}. n The recall rates R{R1,R2,...,R} in the labeled large, medium, and small target data. n ,Rmall ,R medium ,R big Precision P{P D1 ,P D2 ,...,P Dn ,P small ,P medium ,P big The specific methods for calculating recall and precision can refer to relevant technologies. For example, the IOU threshold is 0.5. The overlap between each predicted box and the corresponding ground truth box is calculated. If the overlap is greater than 0.5, the prediction is considered accurate; if it is less than 0.5, the prediction is considered inaccurate.

[0050] Step S508: Determine the performance evaluation parameters of the neural network model based on the first set of average values ​​and the second set of average values; wherein, the performance evaluation parameters are used to characterize the stability of the neural network model for the initial image sample set and the image augmentation sample set and the detection performance for target objects of different size levels.

[0051] This step S508 is determined through the following steps one and two:

[0052] Step 1: Based on the first set of average values ​​and the second set of average values, determine the target average value set corresponding to the target image sample set.

[0053] Continuing with the example above, we can sum and average the n+3 precision values ​​in the set P to obtain the average value P. mean The average value R is obtained by summing and averaging the n+3 recall values ​​in the above set R. mean Similarly, calculate the average value P of the n+3 data points in the sets P, R, C, x, y, w, and h respectively. mean R mean C mean x mean y mean w mean and h mean This yields the set of target average values ​​mentioned above.

[0054] Step 2: Determine the performance evaluation parameters of the neural network model according to the preset weight coefficients and target average value set.

[0055] The aforementioned weighting coefficients can be preset according to actual needs, and the performance evaluation parameters of the neural network model can be calculated using the following formula: 1-(μ1×x) mean +μ2×y mean +μ3×w mean +μ4×h mean +μ5×P mean +μ6×R mean+μ7×C mean In this model, μ1 to μ5 can be 0.4, and μ6 to μ7 can be 1. The values ​​can be set according to actual needs. The final calculated performance evaluation parameters are usually between 0 and 1. The robustness of the model can be evaluated based on the calculated performance evaluation parameters. Generally, the higher the value of the performance evaluation parameter, the better the robustness of the model. Conversely, the lower the value of the performance evaluation parameter, the worse the robustness of the model. This method can obtain the performance evaluation parameters of the neural network model through weighted calculation. The calculation process is simple and can improve the efficiency of evaluating the robustness of the neural network model.

[0056] See Figure 6 The flowchart of another method for determining the performance of a neural network model is shown. Taking a pedestrian detection model as an example, with pedestrians as the target object, the original data (corresponding to the initial image sample set) is first augmented to expand it to N times its original size. For example, image mirroring, noise addition, and brightness adjustment can be performed on the original data. Large, medium, and small detection targets are marked. Specifically, based on the ratio of the pedestrian's true bounding box to the area of ​​the corresponding image, the original data and the augmented data are marked as large, medium, or small according to a preset threshold. Then, the original data and the augmented data are input into the pedestrian detection model. Various prediction indicators (corresponding to the first set of average values ​​mentioned above) under each data augmentation method are calculated. The prediction indicators of the model in large, medium, and small detection targets (corresponding to the second set of average values ​​mentioned above) are calculated, and the average value of each indicator (corresponding to the target average value set mentioned above) is calculated. Then, the robustness score of the pedestrian detection model (corresponding to the performance evaluation parameters mentioned above) is calculated based on the average value.

[0057] The aforementioned method for determining the performance of neural network models takes into account their detection performance across datasets with varying target sizes and recall, precision, confidence, and bounding box coordinates. It allows for rapid evaluation of detection performance while simultaneously calculating performance parameters. These quantified parameters can then be used to assess the robustness of the neural network model, addressing the current lack of objective metrics for model robustness. Furthermore, the data augmentation methods and object size thresholds can be adjusted to suit different business needs, resulting in a robustness testing scheme best suited to the specific requirements of the business.

[0058] This disclosure also provides a device for determining the performance of a neural network model, such as... Figure 7As shown, the apparatus includes: an enhancement processing module 70, used to perform data enhancement processing on image samples in an initial image sample set to obtain a target image sample set containing the initial image sample set and the image enhancement sample set; wherein, the samples in the target image sample set contain at least one target object, the size level corresponding to the target object, and ground truth box information; a testing module 71, used to test a neural network model to be detected using the target image sample set to obtain a first prediction result corresponding to the participating samples in the target image sample set; wherein, the first prediction result includes the first prediction box information corresponding to the first target object in the participating samples; and a determination module 72, used to determine the performance evaluation parameters of the neural network model based on the size level and ground truth box information corresponding to the first target object in the participating samples, and the first prediction box information corresponding to the first target object; wherein, the performance evaluation parameters are used to characterize the stability of the neural network model for the initial image sample set and the image enhancement sample set and the detection performance for target objects of different size levels.

[0059] The aforementioned neural network model performance determination device performs data augmentation processing on image samples in an initial image sample set to obtain a target image sample set containing both the initial image sample set and the image augmentation sample set. Each sample in the target image sample set contains at least one target object, the corresponding size level of the target object, and ground truth box information. The target image sample set is then used to test the neural network model to be detected, yielding a first prediction result corresponding to the participating samples in the target image sample set. This first prediction result includes the first prediction box information corresponding to the first target object in the participating samples. Based on the size level and ground truth box information of the first target object in the participating samples, and the first prediction box information, performance evaluation parameters for the neural network model are determined. These performance evaluation parameters characterize the stability of the neural network model for the initial image sample set and the image augmentation sample set, as well as its detection performance for target objects of different size levels. This device uses the initial image sample set and the image augmentation sample set to test the neural network model, and determines the performance evaluation parameters of the neural network model based on the size level, ground truth box information, and first prediction box information of the first target object in the participating samples. This allows for the evaluation of the neural network model's detection performance on different datasets and target objects of different size levels, while simultaneously achieving a quantitative judgment of the neural network model's robustness.

[0060] In an optional implementation, the determining module 72 is further configured to: calculate a first set of average values ​​corresponding to the initial image sample set and the image augmentation sample set, and a second set of average values ​​for the prediction boxes of participating samples with the same size level, based on the size level and ground truth box information corresponding to the first target object in the participating samples, and the first prediction box information corresponding to the first target object; wherein each average value in the first set of average values ​​is used to characterize the stability of the neural network model for the initial image sample set and the image augmentation sample set; the second set of average values ​​is used to characterize the detection performance of the neural network model for target objects of different size levels; and determine the performance evaluation parameters of the neural network model based on the first set of average values ​​and the second set of average values.

[0061] In an optional implementation, both the first set of average values ​​and the second set of average values ​​include: average coordinate difference, average size difference, and model performance parameters.

[0062] In an optional implementation, the model performance parameters include at least one of the following: recall, confidence, and precision.

[0063] In an optional implementation, the average coordinate difference is an average value determined based on the difference between corresponding coordinate values ​​in the ground truth box information and the first predicted box information corresponding to the first target object in the participating samples; the average size difference is an average value determined based on the difference between corresponding sizes in the ground truth box information and the first predicted box information corresponding to the first target object in the participating samples; and the model performance parameter is a parameter value determined based on the performance of the predicted box corresponding to the first target object in the participating samples relative to the ground truth box.

[0064] In an optional implementation, the determining module 72 is further configured to: determine a target average set corresponding to the target image sample set based on the first average set and the second average set; and determine the performance evaluation parameters of the neural network model according to preset weight coefficients and the target average set.

[0065] In one alternative implementation, the data augmentation process includes at least one of the following: noise addition, mirroring, brightness adjustment, and blurring.

[0066] The neural network model performance determination device provided in this embodiment of the invention has the same implementation principle and technical effect as the aforementioned neural network model performance determination method embodiment. For the sake of brevity, any parts not mentioned in the neural network model performance determination device embodiment can be referred to the corresponding content in the aforementioned neural network model performance determination method embodiment.

[0067] This disclosure also provides an electronic device, see [link to relevant documentation] Figure 8 As shown, the electronic device includes a processor 160 and a memory 161. The memory 161 stores machine-executable instructions that can be executed by the processor 160. The processor 160 executes the machine-executable instructions to implement the performance determination method of the neural network model described above.

[0068] Furthermore, Figure 8 The electronic device shown also includes a bus 162 and a communication interface 163, with the processor 160, the communication interface 163 and the memory 161 connected via the bus 162.

[0069] The memory 161 may include high-speed random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 163 (which can be wired or wireless), such as the Internet, wide area network, local area network, or metropolitan area network. The bus 162 may be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0070] Processor 160 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 160 or by instructions in software form. Processor 160 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 161. The processor 160 reads the information in memory 161 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiment, specifically executing the following steps:

[0071] Data augmentation processing is performed on the image samples in the initial image sample set to obtain a target image sample set containing the initial image sample set and the image augmentation sample set; wherein, the samples in the target image sample set contain at least one target object, the size level corresponding to the target object, and the ground truth box information;

[0072] The neural network model to be detected is tested using a set of target image samples to obtain the first prediction result corresponding to the participating samples in the set of target image samples; wherein, the first prediction result includes the first prediction box information corresponding to the first target object in the participating samples;

[0073] Based on the size level and ground truth box information of the first target object in the participating samples, as well as the first prediction box information of the first target object, the performance evaluation parameters of the neural network model are determined. The performance evaluation parameters are used to characterize the stability of the neural network model for the initial image sample set and the image augmentation sample set, and the detection performance of the target object for different size levels.

[0074] The specific implementation methods and working processes can be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0075] In an optional implementation, when the processor determines the performance evaluation parameters of the neural network model based on the size level and ground truth box information corresponding to the first target object in the participating samples, and the first prediction box information corresponding to the first target object, it specifically performs the following: calculating a first set of average values ​​corresponding to the initial image sample set and the image enhancement sample set, and a second set of average values ​​for the prediction boxes of participating samples with the same size level, based on the size level and ground truth box information corresponding to the first target object in the participating samples, and the first prediction box information corresponding to the first target object; wherein each average value in the first set of average values ​​is used to characterize the stability of the neural network model for the initial image sample set and the image enhancement sample set; the second set of average values ​​is used to characterize the detection performance of the neural network model for target objects of different size levels; and determining the performance evaluation parameters of the neural network model based on the first set of average values ​​and the second set of average values.

[0076] In an optional implementation, both the first set of average values ​​and the second set of average values ​​include: the average coordinate difference, the average size difference, and model performance parameters.

[0077] In an alternative implementation, the model performance parameters include at least one of the following: recall, confidence, and precision.

[0078] In an optional implementation, the average coordinate difference is the average value determined based on the difference between corresponding coordinate values ​​in the ground truth box information and the first predicted box information corresponding to the first target object in the participating samples; the average size difference is the average value determined based on the difference between corresponding sizes in the ground truth box information and the first predicted box information corresponding to the first target object in the participating samples; and the model performance parameter is a parameter value determined based on the performance of the predicted box corresponding to the first target object in the participating samples relative to the ground truth box.

[0079] In an optional implementation, when the processor executes the process of determining the performance evaluation parameters of the neural network model based on the first set of average values ​​and the second set of average values, it specifically performs the following steps: determining the target average value set corresponding to the target image sample set based on the first set of average values ​​and the second set of average values; and determining the performance evaluation parameters of the neural network model according to the preset weight coefficients and the target average value set.

[0080] In one alternative implementation, the data augmentation process includes at least one of the following: noise addition, mirroring, brightness adjustment, and blurring.

[0081] The above method involves data augmentation processing on image samples in the initial image sample set to obtain a target image sample set containing both the initial and augmented image sample sets. Each sample in the target image sample set contains at least one target object, its corresponding size level, and ground truth bounding box information. The target image sample set is then used to test the neural network model to be detected, yielding a first prediction result for each participating sample in the target image sample set. This first prediction result includes the first prediction bounding box information corresponding to the first target object in the participating samples. Based on the size level and ground truth bounding box information of the first target object in the participating samples, as well as the first prediction bounding box information, performance evaluation parameters for the neural network model are determined. These performance evaluation parameters characterize the stability of the neural network model across the initial and augmented image sample sets and its detection performance for target objects of different size levels. This method uses the initial and augmented image sample sets to test the neural network model, and determines its performance evaluation parameters based on the size level, ground truth bounding box information, and first prediction bounding box information of the first target object in the participating samples. This allows for the evaluation of the neural network model's detection performance on different datasets and target objects of different size levels, while simultaneously enabling a quantitative assessment of the model's robustness.

[0082] This disclosure also provides a machine-readable storage medium storing machine-executable instructions that, when invoked and executed by a processor, cause the processor to perform the following steps:

[0083] Data augmentation processing is performed on the image samples in the initial image sample set to obtain a target image sample set containing the initial image sample set and the image augmentation sample set; wherein, the samples in the target image sample set contain at least one target object, the size level corresponding to the target object, and the ground truth box information;

[0084] The neural network model to be detected is tested using a set of target image samples to obtain the first prediction result corresponding to the participating samples in the set of target image samples; wherein, the first prediction result includes the first prediction box information corresponding to the first target object in the participating samples;

[0085] Based on the size level and ground truth box information of the first target object in the participating samples, as well as the first prediction box information of the first target object, the performance evaluation parameters of the neural network model are determined. The performance evaluation parameters are used to characterize the stability of the neural network model for the initial image sample set and the image augmentation sample set, and the detection performance of the target object for different size levels.

[0086] The specific implementation methods and working processes can be referred to the corresponding processes in the above method embodiments, and will not be repeated here.

[0087] In an optional implementation, when the processor determines the performance evaluation parameters of the neural network model based on the size level and ground truth box information corresponding to the first target object in the participating samples, and the first prediction box information corresponding to the first target object, it specifically performs the following: calculating a first set of average values ​​corresponding to the initial image sample set and the image enhancement sample set, and a second set of average values ​​for the prediction boxes of participating samples with the same size level, based on the size level and ground truth box information corresponding to the first target object in the participating samples, and the first prediction box information corresponding to the first target object; wherein each average value in the first set of average values ​​is used to characterize the stability of the neural network model for the initial image sample set and the image enhancement sample set; the second set of average values ​​is used to characterize the detection performance of the neural network model for target objects of different size levels; and determining the performance evaluation parameters of the neural network model based on the first set of average values ​​and the second set of average values.

[0088] In an optional implementation, both the first set of average values ​​and the second set of average values ​​include: the average coordinate difference, the average size difference, and model performance parameters.

[0089] In an alternative implementation, the model performance parameters include at least one of the following: recall, confidence, and precision.

[0090] In an optional implementation, the average coordinate difference is the average value determined based on the difference between corresponding coordinate values ​​in the ground truth box information and the first predicted box information corresponding to the first target object in the participating samples; the average size difference is the average value determined based on the difference between corresponding sizes in the ground truth box information and the first predicted box information corresponding to the first target object in the participating samples; and the model performance parameter is a parameter value determined based on the performance of the predicted box corresponding to the first target object in the participating samples relative to the ground truth box.

[0091] In an optional implementation, when the processor executes the process of determining the performance evaluation parameters of the neural network model based on the first set of average values ​​and the second set of average values, it specifically performs the following steps: determining the target average value set corresponding to the target image sample set based on the first set of average values ​​and the second set of average values; and determining the performance evaluation parameters of the neural network model according to the preset weight coefficients and the target average value set.

[0092] In one alternative implementation, the data augmentation process includes at least one of the following: noise addition, mirroring, brightness adjustment, and blurring.

[0093] The above method involves data augmentation processing on image samples in the initial image sample set to obtain a target image sample set containing both the initial and augmented image sample sets. Each sample in the target image sample set contains at least one target object, its corresponding size level, and ground truth bounding box information. The target image sample set is then used to test the neural network model to be detected, yielding a first prediction result for each participating sample in the target image sample set. This first prediction result includes the first prediction bounding box information corresponding to the first target object in the participating samples. Based on the size level and ground truth bounding box information of the first target object in the participating samples, as well as the first prediction bounding box information, performance evaluation parameters for the neural network model are determined. These performance evaluation parameters characterize the stability of the neural network model across the initial and augmented image sample sets and its detection performance for target objects of different size levels. This method uses the initial and augmented image sample sets to test the neural network model, and determines its performance evaluation parameters based on the size level, ground truth bounding box information, and first prediction bounding box information of the first target object in the participating samples. This allows for the evaluation of the neural network model's detection performance on different datasets and target objects of different size levels, while simultaneously enabling a quantitative assessment of the model's robustness.

[0094] The computer program product of the neural network model performance determination method, apparatus and electronic device provided in the embodiments of the present invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0095] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for determining the performance of a neural network model, characterized in that, The method includes: Data augmentation processing is performed on image samples in the initial image sample set to obtain a target image sample set containing the initial image sample set and the image augmentation sample set; wherein, the samples in the target image sample set contain at least one target object, the size level corresponding to the target object, and the ground truth box information; The target image sample set is used to test the neural network model to be detected, and a first prediction result corresponding to the participating samples in the target image sample set is obtained; wherein, the first prediction result includes the first prediction box information corresponding to the first target object in the participating samples; Based on the size level and ground truth box information corresponding to the first target object in the participating samples, and the first prediction box information corresponding to the first target object, the performance evaluation parameters of the neural network model are determined; wherein, the performance evaluation parameters are used to characterize the stability of the neural network model for the initial image sample set and the image augmentation sample set and the detection performance for target objects of different size levels; The step of determining the performance evaluation parameters of the neural network model based on the size level and ground truth box information corresponding to the first target object in the participating samples, and the first prediction box information corresponding to the first target object, includes: Based on the size level and ground truth box information of the first target object in the participating samples, and the first predicted box information of the first target object, a first set of average values ​​corresponding to the initial image sample set and the image augmentation sample set, and a second set of average values ​​of the predicted boxes of participating samples with the same size level are calculated; wherein, each average value in the first set of average values ​​is used to characterize the stability of the neural network model for the initial image sample set and the image augmentation sample set; the second set of average values ​​is used to characterize the detection performance of the neural network model for target objects of different size levels; The performance evaluation parameters of the neural network model are determined based on the first set of average values ​​and the second set of average values.

2. The method according to claim 1, characterized in that, Both the first set of average values ​​and the second set of average values ​​include: average coordinate difference, average size difference, and model performance parameters.

3. The method according to claim 2, characterized in that, The model performance parameters include at least one of the following: recall, confidence, and precision.

4. The method according to claim 2, characterized in that, The average coordinate difference is determined based on the difference between the corresponding coordinate values ​​in the ground truth box information and the first prediction box information corresponding to the first target object in the participating samples. The average size difference is determined based on the difference between the corresponding sizes in the ground truth box information and the first prediction box information corresponding to the first target object in the participating samples. The model performance parameters are parameter values ​​determined based on the performance of the predicted bounding box corresponding to the first target object in the participating samples relative to the ground truth box.

5. The method according to claim 1, characterized in that, The steps for determining the performance evaluation parameters of the neural network model based on the first set of average values ​​and the second set of average values ​​include: Based on the first set of average values ​​and the second set of average values, determine the target set of average values ​​corresponding to the target image sample set; The performance evaluation parameters of the neural network model are determined according to the preset weight coefficients and the target average value set.

6. The method according to claim 1, characterized in that, Data augmentation methods include at least one of the following: noise addition, mirroring, brightness adjustment, and blurring.

7. A device for determining the performance of a neural network model, characterized in that, The device includes: An enhancement processing module is used to perform data enhancement processing on image samples in an initial image sample set to obtain a target image sample set containing the initial image sample set and the image enhancement sample set; wherein, the samples in the target image sample set contain at least one target object, the size level corresponding to the target object, and truth box information; The testing module is used to test the neural network model to be detected using the target image sample set, and to obtain the first prediction result corresponding to the participating samples in the target image sample set; wherein, the first prediction result includes the first prediction box information corresponding to the first target object in the participating samples; The determination module is used to determine the performance evaluation parameters of the neural network model based on the size level and ground truth box information corresponding to the first target object in the participating samples, and the first prediction box information corresponding to the first target object; wherein, the performance evaluation parameters are used to characterize the stability of the neural network model for the initial image sample set and the image augmentation sample set and the detection performance for target objects of different size levels; The determining module is further configured to: calculate, based on the size level and ground truth box information corresponding to the first target object in the participating samples, and the first prediction box information corresponding to the first target object, calculate a first set of average values ​​corresponding to the initial image sample set and the image augmentation sample set, and a second set of average values ​​for the prediction boxes of participating samples with the same size level; wherein, each average value in the first set of average values ​​is used to characterize the stability of the neural network model for the initial image sample set and the image augmentation sample set; the second set of average values ​​is used to characterize the detection performance of the neural network model for target objects of different size levels; and determine the performance evaluation parameters of the neural network model based on the first set of average values ​​and the second set of average values.

8. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing machine-executable instructions that can be executed by the processor, the processor executing the machine-executable instructions to implement the performance determination method of the neural network model according to any one of claims 1 to 6.

9. A machine-readable storage medium, characterized in that, The machine-readable storage medium stores machine-executable instructions, which, when invoked and executed by a processor, cause the processor to implement the performance determination method for the neural network model according to any one of claims 1 to 6.

Images