Abnormity assessment method and device for image model

By selecting appropriate accuracy comparison methods based on image model categories and introducing output consistency verification, the problem of singularity and instability of image model evaluation in the prior art is solved, and a comprehensive evaluation of image model is achieved, satisfying multiple services. need.

CN119027695APending Publication Date: 2024-11-26SHANGHAI BIREN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411002125.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing image model exception evaluation method is only evaluated from the perspective of accuracy, lacks comprehensiveness, cannot meet the needs of multiple business scenarios, and the evaluation method is single, so it cannot be evaluated from the perspective of stability of the output results.

Method used

By receiving the processing results of multiple inputs by the image model, the appropriate accuracy comparison method is determined according to the model category, the accuracy evaluation is carried out, and the output consistency verification is introduced. Combined with the comparison of relative error and absolute error, the output consistency is verified using MD5 values ​​to verify the output consistency , verify model feedback for exception input.

Benefits of technology

It realizes a more comprehensive evaluation of the image model, meets multiple needs, and can be evaluated from the perspectives of accuracy and stability, improving the accuracy and comprehensiveness of the evaluation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027695A_ABST
    Figure CN119027695A_ABST
Patent Text Reader

Abstract

The invention discloses an anomaly evaluation method and device for an image model. The anomaly evaluation method comprises the steps of receiving a plurality of corresponding output results obtained after a current image model processes multiple inputs; wherein the multiple inputs comprise at least two same inputs; based on the category of the current image model, determining a precision comparison mode corresponding to the category; according to the determined precision comparison mode, performing precision evaluation on the current image model for each output result to obtain a precision evaluation result corresponding to each output result; and verifying the output consistency based on a plurality of output results corresponding to the same input. According to the invention, the anomaly evaluation of the image model can be carried out more comprehensively, and various different requirements are met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to an abnormality assessment method and device for an image model. Background Art

[0002] In the field of image processing in artificial intelligence, there are many unit processing operations related to images, such as image flipping operations, image accumulation operations, etc. These unit processing operations of images process the input image data to obtain output data. These unit processing operations are also called image processing operators or image models. The image model here refers specifically to this type of model related to small unit operation processing, and does not include large, complete image neural network models.

[0003] Regarding the above-mentioned image models or image operators, they need to be evaluated for abnormalities in practical applications to determine whether the implementation of the corresponding image models or image operators meets actual needs. At present, considering that a lot of the output data of the image model will be used in the training and reasoning of the AI ​​big model, and the accuracy requirements for image data in the corresponding applications are relatively strict, which will affect the training and reasoning effects of the entire AI big model. Based on this, the existing abnormality evaluation of the image model is usually an accuracy evaluation of the image model.

[0004] The existing accuracy evaluation of image models mainly uses the peak signal-to-noise ratio (PSNR) comparison method, that is, the PSNR of the image model output is compared with the expected output, and whether the processing accuracy of the image model meets the requirements is determined based on the comparison result.

[0005] The above-mentioned anomaly assessment method of the image model only performs anomaly assessment from the perspective of the accuracy of the image model. The assessment angle is single, and only the PSNR method is used for accuracy assessment. As a result, the accuracy assessment result can only reflect the overall accuracy of the image, and lacks a personalized comparison method. This makes the anomaly assessment of the entire image model not comprehensive enough and unable to meet a variety of business scenarios. Summary of the invention

[0006] The present application provides an image model anomaly assessment method and device, which can perform anomaly assessment of image models more comprehensively and meet a variety of different needs.

[0007] To achieve the above objectives, this application adopts the following technical solutions:

[0008] A method for evaluating abnormality of an image model, comprising:

[0009] Receiving a plurality of corresponding output results obtained after the current image model processes a plurality of inputs; wherein the plurality of inputs include at least two identical inputs;

[0010] Based on the category of the current image model, determining an accuracy comparison method corresponding to the category;

[0011] According to the determined accuracy comparison method, the accuracy of the current image model is evaluated for each output result to obtain an accuracy evaluation result corresponding to each output result;

[0012] Verify the consistency of outputs based on multiple outputs corresponding to the same input.

[0013] Preferably, the method further comprises: for the abnormal input received by the current image model, verifying whether the current image model feeds back information of the input abnormality.

[0014] Preferably, for the image model of the data handling category, the corresponding accuracy comparison method is bit consistency comparison; and / or,

[0015] For image models that are not data handling categories, the corresponding accuracy comparison method is PSNR comparison and / or AI accuracy standard comparison; and / or,

[0016] For image models in the standard library, evaluate in advance whether the image model needs to meet the bit consistency requirements based on the current hardware conditions and the category of the image model; if necessary, determine the corresponding accuracy comparison method as bit consistency comparison; if not, determine the corresponding accuracy comparison method as PSNR comparison and / or AI accuracy standard comparison.

[0017] Preferably, the comparison of the AI ​​accuracy standard is a comparison combining relative error and absolute error.

[0018] Preferably, the comparison of the relative error and the absolute error comprises:

[0019] When performing accuracy evaluation on the current image model for each output result, calculating the absolute value of the difference between each pixel of the output result and the corresponding pixel of the expected output;

[0020] Compare the absolute value with the standard error threshold atol+rtol*|expected|;

[0021] If the absolute value of each pixel is less than or equal to the standard error threshold, it is determined that the accuracy evaluation result corresponding to the corresponding output result meets the accuracy requirement;

[0022] If the absolute value of any pixel is greater than the standard error threshold, it is determined that the accuracy evaluation result corresponding to the corresponding output result does not meet the accuracy requirement;

[0023] The atol and rtol are respectively a preset absolute error threshold and a relative error threshold, and the expected is a pixel value of a corresponding pixel of the expected output.

[0024] Preferably, the corresponding relative error threshold and absolute error threshold are set in advance for different combinations of categories of various image models and input data types.

[0025] Preferably, the comparison combining relative error and absolute error further comprises: setting the bit consistency comparison to be adopted for a specified combination of the category of the image model and the input data type in advance;

[0026] If the combination of the current image model and the input data type belongs to the specified combination, when the accuracy of the current image model is evaluated for each output result, a bit consistency comparison method is used to perform the accuracy evaluation.

[0027] Preferably, the verification of output consistency based on multiple output results corresponding to the same input includes:

[0028] For the same input, determine whether the corresponding multiple output results are consistent. If they are consistent, the output consistency verification is passed; otherwise, the output consistency verification is failed.

[0029] Preferably, the method of determining whether the corresponding multiple output results are consistent includes: determining whether the MD5 values ​​of the multiple output results corresponding to the same input are the same, if they are the same, determining that the corresponding multiple output results are consistent, if they are not the same, determining that the corresponding multiple output results are inconsistent.

[0030] An abnormality assessment device for an image model comprises: a receiving unit, an accuracy assessment unit, an output consistency verification unit and an output unit;

[0031] The receiving unit is used to receive corresponding multiple output results obtained after the current image model processes multiple inputs; wherein the multiple inputs include at least two identical inputs;

[0032] The accuracy evaluation unit is used to determine the accuracy comparison method corresponding to the category of the current image model based on the category of the current image model; and is also used to perform accuracy evaluation on the current image model for each output result according to the determined accuracy comparison method, obtain the accuracy evaluation result corresponding to each output result, and output it to the output unit;

[0033] The output consistency verification unit is used to verify the output consistency based on multiple output results corresponding to the same input, and output the verification result to the output unit;

[0034] The output unit is used to determine and output an abnormal evaluation result based on the accuracy evaluation result and the verification result of the output consistency.

[0035] Preferably, the device further comprises an abnormal input verification unit, which is used to verify whether the current image model feeds back information of input abnormality for the abnormal input received by the current image model, and output the verification result to the output unit;

[0036] The output unit is used to further determine the abnormality assessment result based on the verification result of the abnormal input.

[0037] A computer-readable storage medium stores computer instructions, which, when executed by a processor, can implement any of the above-mentioned image model anomaly assessment methods.

[0038] A computer program product comprises computer executable instructions, which, when executed by a processor, implement any one of the above-mentioned methods for assessing abnormality of an image model.

[0039] It can be seen from the above technical solution that in the present application, the corresponding multiple output results obtained after the current image model processes multiple inputs are received; based on the category of the current image model, the accuracy comparison method corresponding to the category is determined; according to the determined accuracy comparison method, the current image model is accuracy evaluated for each output result, and the accuracy evaluation result corresponding to each output result is obtained; through this processing, the corresponding accuracy comparison method is adopted for different categories of image models, so that the accuracy comparison can meet a variety of different practical needs. Next, based on the multiple output results corresponding to the same input, the output consistency is verified. Through this processing, on the basis of the accuracy evaluation, the output consistency verification is introduced, so that the abnormality evaluation of the image model can be performed from the perspective of the output stability of the image model, and the abnormality evaluation of the image model can be performed more comprehensively. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the basic process of the abnormality assessment method of the image model in this application;

[0041] Figure 2 It is a basic flow chart of the abnormality assessment method of the image model in the specific embodiment of the present application;

[0042] Figure 3 Schematic diagram of the basic structure of the abnormality assessment device of the image model in this application. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical means and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings.

[0044] Figure 1 FIG. 1 is a basic flow chart of the abnormality assessment method of the image model in this application. Figure 1 As shown, the method includes:

[0045] Step 101, receiving multiple corresponding output results obtained after the current image model processes multiple inputs.

[0046] The current image model in this application refers to the image model that currently needs to be evaluated for abnormality, that is, the image model to be tested.

[0047] The current image model receives input image data and processes the image data accordingly to obtain corresponding output results. This step is used to receive the output results obtained after the current image model processes the image data.

[0048] In addition to accuracy assessment, this application also requires verification of output consistency for the same input. Therefore, the current image model needs to process multiple inputs to obtain corresponding output results, and the multiple inputs include at least two identical inputs.

[0049] Step 102: Based on the category of the current image model, determine the accuracy comparison method corresponding to the category.

[0050] In this application, personalized accuracy comparison methods are set in advance for different categories of image models.

[0051] Exemplarily, for image models of the data handling type, since the pixel values ​​of the image pixels are not changed during the model processing process, only the image pixel positions are rearranged, the corresponding accuracy comparison method is set to bit consistency comparison, requiring that the output result of the image model must be completely consistent with the expected output result; for image models of the non-data handling type, since the pixel values ​​may be changed during the model processing process, the corresponding accuracy comparison method is set to PSNR comparison and / or AI accuracy standard comparison, and a certain error is allowed in the comparison result.

[0052] Through this personalized accuracy comparison mode setting, more optional accuracy comparison modes are added, so that a more appropriate accuracy comparison mode can be adopted according to actual needs and the category of the image model.

[0053] Step 103 , according to the determined accuracy comparison method, the accuracy of the current image model is evaluated for each output result to obtain an accuracy evaluation result corresponding to each output result.

[0054] For each output result obtained after the input is processed by the image model, a corresponding accuracy evaluation is required.

[0055] The accuracy comparison method corresponding to the current image model has been determined in the aforementioned step 102. In this step, the accuracy of the current image model is evaluated based on each output result according to the determined accuracy comparison method. For any output result, the specific processing method for evaluating the accuracy of the current image model based on the output result is the same, and the accuracy comparison method determined in step 102 is used.

[0056] Step 104 , verifying the consistency of the outputs based on the multiple output results corresponding to the same input.

[0057] In addition to evaluating the accuracy of the image model, the anomaly assessment method of the present application also includes verification of output consistency. The reason for this verification is that in practical applications, there may be stability requirements for the processing of image models. In other words, for the image model, it may be required that the different outputs obtained after the image model processing for the same input are relatively stable.

[0058] However, accuracy assessment can only evaluate the accuracy of the data processed by the image model in an absolute sense to determine whether it meets the pre-set accuracy requirements; although multiple output results can meet the accuracy requirements, the differences between multiple outputs obtained from the same input cannot be measured, so it is impossible to perform abnormality assessment from the perspective of stability. Based on this, this application introduces output consistency verification, which is used to evaluate whether the multiple output results obtained after the image model is processed for the same input at different times are stable enough.

[0059] When verifying the output consistency, multiple output results corresponding to multiple identical inputs can be compared to see whether the comparison results meet the requirements, thereby completing the output consistency verification and realizing the evaluation of the stability of the image model output.

[0060] The verification of specific output consistency can be performed simultaneously with the accuracy assessment, or can be performed serially, or can be limited to being performed only after the accuracy assessment is determined to meet the requirements based on the output results.

[0061] So far, Figure 1 The process of the abnormality assessment method of the image model shown ends.

[0062] As above, through this application Figure 1The processing of the process shown, on the one hand, can adapt to a variety of different needs by setting personalized accuracy evaluation methods for different image model categories; on the other hand, by introducing output consistency verification, it can realize the evaluation of the stability of the image model output, so as to conduct a more comprehensive abnormality assessment of the image model.

[0063] The specific implementation of this application is described below through specific embodiments. Figure 2 Schematic diagram of the specific process of the image model abnormality assessment method in the specific embodiment of the present application. Figure 2 As shown, the method includes:

[0064] Step 201: The current image model processes the input image data to obtain a corresponding output result.

[0065] This step is used to process the image model, and the specific processing may be: image transfer, image logic operation, image superposition, etc. The output result obtained after processing is used to perform abnormal evaluation of the image model.

[0066] Step 202: Determine the corresponding accuracy comparison method according to the category of the current image model.

[0067] In this embodiment, for the image model of the transport type, the corresponding accuracy comparison method is determined to be bit consistency comparison, which requires that the output result is compared with the corresponding pixel of the expected output (for example, it can be a Golden output), and the value of each bit is completely consistent; for the image model of the non-transport type, the corresponding accuracy comparison method is determined to be PSNR comparison and / or AI accuracy standard comparison.

[0068] Among them, PSNR comparison is the comparison method used in the existing accuracy evaluation. The whole image of the output result is compared with the whole image of the expected output, and the PSNR value is calculated. If the PSNR value is greater than or equal to the set PSNR threshold, it is determined that the accuracy evaluation result meets the requirements, otherwise, it is determined that the accuracy evaluation result does not meet the requirements. Among them, the PSNR threshold can be set according to actual needs.

[0069] The comparison of AI accuracy standards can be, for example, a comparison of relative error and absolute error. The comparison of relative error and absolute error can specifically include:

[0070] 1) Calculate the difference between the actual value of each pixel of the output result and the expected value of the corresponding pixel of the expected output, and take the absolute value of the difference to obtain |actual-expected|;

[0071] 2) Compare the absolute value calculated in 1) with the standard error threshold atol+rtol*|expected|;

[0072] atol and rtol are the preset absolute error threshold and relative error threshold respectively. The corresponding thresholds can be set according to actual needs. At the same time, the corresponding relative error threshold and absolute error threshold can be set in advance for different combinations of various image model categories and input data types; when actually performing accuracy assessment, the corresponding atol and rtol are determined according to the combination of the current image model category and the input data type, and the calculation and comparison of this step are performed;

[0073] In more detail, when setting atol and rtol in advance, you can refer to the atol and rtol used in the Pytorch test. If the standard error threshold X determined by the atol and rtol used in the Pytorch test cannot be met, you can appropriately relax the corresponding atol and rtol according to the current hardware conditions and the category evaluation of the image model, that is, relax the standard error threshold X. In some special cases, you can even customize dedicated atol and rtol;

[0074] 3) If the absolute value calculated corresponding to each pixel in the output result is less than or equal to the standard error threshold, it is determined that the accuracy evaluation result corresponding to the corresponding output result meets the accuracy requirement; if the absolute value calculated corresponding to a certain pixel in the output result is greater than the standard error threshold, it is determined that the accuracy evaluation result corresponding to the corresponding output result does not meet the accuracy requirement;

[0075] It can be seen from the above conditions for judging whether the accuracy requirements are met that the comparison method combining relative error and absolute error requires that the absolute value of each pixel meets the requirements to pass the accuracy assessment. Therefore, this comparison method can perform accuracy assessment from the detailed perspective of each pixel, rather than just from the overall perspective of the output result.

[0076] In addition, in the comparison of relative error and absolute error, considering that the corresponding output results may need to meet the bit consistency requirements under the combination of certain image model categories and data types, the accuracy evaluation can be performed using the bit consistency requirements under the corresponding combinations of corresponding marks when setting the relative error threshold and the absolute error threshold. An exemplary relative error threshold and absolute error threshold setting table is given below as shown in Table 1, in which the relative error threshold and the absolute error threshold are set for different combinations of image model categories and data types, and some of the combinations are set using the bit consistency requirements.

[0077]

[0078] Table 1

[0079] The second column in Table 1 gives multiple data types, {} represents a set of relative error thresholds and absolute error thresholds, the values ​​of the relative error thresholds and the absolute error thresholds correspond to each data type in sequence, and "completely consistent" means that the corresponding combination adopts the bit consistency requirement.

[0080] The comparison of AI accuracy standards mainly adopts the above-mentioned relative error and absolute error combination comparison method.

[0081] For non-data handling image models, PSNR comparison can be used alone, or AI precision standard comparison can be used alone, or both comparison methods can be combined for precision assessment. When the precision assessment results under both comparison methods meet the requirements, it is determined that the precision assessment has passed.

[0082] In addition to setting the corresponding accuracy comparison method according to the data handling category and the non-data handling category as mentioned above, a separate process for determining an accuracy comparison method can also be given for the image model in the specified standard library. Specifically, for the image model in the standard library, it is pre-evaluated whether the image model needs to meet the bit consistency requirements based on the current hardware conditions and the category of the image model; if necessary, the corresponding accuracy comparison method is determined to be bit consistency comparison; if not, the corresponding accuracy comparison method is determined to be PSNR comparison and / or AI accuracy standard comparison.

[0083] The determination of the corresponding accuracy comparison method here takes into account that under the current good hardware conditions, some simple image models including data handling models should be able to meet the bit consistency requirements under normal implementation conditions. Therefore, for such simple image models, a bit consistency comparison method is required; and for some slightly more complex simple image models, under the current hardware conditions, it is reasonable that they do not meet the bit consistency, and a certain error should be allowed to exist. In this case, they can be set to use the PSNR comparison and / or AI accuracy standard comparison method.

[0084] For image models in non-standard libraries, the corresponding accuracy comparison method can be determined as PSNR comparison and / or AI accuracy standard comparison.

[0085] Step 203 , according to the determined accuracy comparison method, based on the output results obtained in step 201 , the accuracy of the current image model is evaluated to obtain an accuracy evaluation result corresponding to each output result.

[0086] In this step, the output result of step 201 is compared with the expected output according to the accuracy comparison method determined in step 202, so as to determine the accuracy evaluation result.

[0087] Through the processing of steps 201 to 203, the accuracy evaluation of the current image model for one output result is completed. Next, the accuracy evaluation for multiple output results can be completed through subsequent loop processing and repeated execution of steps 201 to 203 for multiple times.

[0088] Step 204, when the accuracy evaluation result obtained in step 203 meets the accuracy requirement, output consistency verification is performed.

[0089] In this embodiment, after each accuracy evaluation result meets the accuracy requirement, the output consistency verification is performed. This processing method can terminate the abnormal evaluation process and the system resources in time when the accuracy evaluation does not meet the requirements. Of course, if you still want to perform output consistency verification even when the accuracy evaluation does not meet the requirements, you can also perform the output consistency verification of step 204 in parallel with the processing of the aforementioned steps 202 to 203.

[0090] The specific method of performing output consistency verification in this embodiment is: for the same input, determine whether the corresponding multiple output results are consistent. If they are consistent, it is determined that the output consistency verification has passed; otherwise, it is determined that the output consistency verification has not passed.

[0091] When performing output consistency verification, it is necessary to compare multiple output results, so it is necessary to record the output results corresponding to the same input before. Among them, when judging whether multiple output results are consistent, it is possible to judge whether the MD5 values ​​of the corresponding multiple output results are the same. If they are the same, it is determined that the corresponding multiple output results are consistent. If they are not the same, it is determined that the corresponding multiple output results are inconsistent. The MD5 value is also the hash value of the output result. This method of comparing MD5 values ​​can effectively determine whether multiple output results are consistent on the one hand, and on the other hand, it does not need to store and compare all bits of multiple output results, which greatly saves storage space and improves processing speed.

[0092] Step 205: Verify whether the current image model feeds back information indicating the input abnormality for the abnormal input received by the current image model.

[0093] Normally, when the input image data of the image model is an abnormal value, the image model should give an error code, i.e., information used to prompt the input abnormality. In this embodiment, the verification of abnormal input is further introduced, that is, the prompt of the error code is added to the abnormal evaluation process, so that more abnormal behaviors of the image model can be covered by comparison in this way, thereby improving the stability of the image model.

[0094] Specifically, abnormal data can be input into the current image model to determine whether the current image model feeds back information about the abnormal input, i.e., an error code. If it does, it is determined that the verification of the abnormal input has been passed. If it does not, it is determined that the verification of the abnormal input has not been passed. Feedback of the abnormal input information may include displaying the abnormal input information or playing the abnormal input information.

[0095] The specific input anomaly verification process can be performed after the output consistency verification is passed, or it can also be performed in parallel with the output consistency verification and accuracy assessment process.

[0096] At this point, the abnormality assessment process for one output result ends. Through the above process, abnormality assessment can be performed from three aspects to obtain corresponding abnormality assessment results. Next, it is also possible to return to step 201 to perform abnormality assessment on the next output result, and perform output consistency verification in step 203 based on multiple output results corresponding to the same input for multiple times.

[0097] Through the above Figure 2 The processing process of the specific embodiment shown in the figure sets the corresponding accuracy comparison method in a personalized manner for different categories of image models, thereby effectively meeting various different accuracy comparison requirements; at the same time, through output consistency verification and abnormal input verification, the quality and stability of the image model can be evaluated, so as to perform a more comprehensive and multi-faceted abnormality evaluation on the image model.

[0098] The above is a specific implementation of the abnormality assessment method of the image model in the present application. The present application also provides an abnormality assessment device for an image model, which can be used to implement the above abnormality assessment method. Figure 3 FIG. 1 is a schematic diagram of the basic structure of the abnormality assessment device. Figure 3 As shown, the device includes: a receiving unit, an accuracy evaluation unit, an output consistency verification unit and an output unit.

[0099] A receiving unit, configured to receive a plurality of corresponding output results obtained after the current image model processes a plurality of inputs; wherein the plurality of inputs include at least two identical inputs;

[0100] The accuracy evaluation unit is used to determine the accuracy comparison method corresponding to the category based on the category of the current image model; and is also used to perform accuracy evaluation on the current image model for each output result according to the determined accuracy comparison method, obtain the accuracy evaluation result corresponding to each output result, and output it to the output unit;

[0101] An output consistency verification unit, used to verify the output consistency based on multiple output results corresponding to the same input, and output the verification result to the output unit;

[0102] The output unit is used to determine the abnormal evaluation result for output based on the accuracy evaluation result and the verification result of the output consistency.

[0103] Optionally, to implement abnormal input verification, the device may further include an abnormal input verification unit, which is used to verify whether the current image model feeds back information about the input abnormality for the abnormal input received by the current image model, and output the verification result to the output unit;

[0104] The output unit can be used to further determine the abnormal evaluation result based on the verification result of the abnormal input.

[0105] Optionally, in the accuracy evaluation unit, for the image model of the data handling category, the corresponding accuracy comparison method is bit consistency comparison; and / or,

[0106] For image models that are not data handling categories, the corresponding accuracy comparison method is PSNR comparison and / or AI accuracy standard comparison; and / or,

[0107] For image models in the standard library, evaluate in advance whether the image model needs to meet the bit consistency requirements based on the current hardware conditions and the category of the image model; if necessary, determine the corresponding accuracy comparison method as bit consistency comparison; if not, determine the corresponding accuracy comparison method as PSNR comparison and / or AI accuracy standard comparison.

[0108] Optionally, in the accuracy evaluation unit, the comparison of the AI ​​accuracy standard may specifically be a comparison combining relative error and absolute error.

[0109] Optionally, in the accuracy evaluation unit, the comparison of the relative error and the absolute error may specifically include:

[0110] When performing accuracy evaluation on the current image model for each output result, calculating the absolute value of the difference between each pixel of the output result and the corresponding pixel of the expected output;

[0111] Compare the absolute value with the standard error threshold atol+rtol*|expected|;

[0112] If the absolute value of each pixel is less than or equal to the standard error threshold, it is determined that the accuracy evaluation result corresponding to the corresponding output result meets the accuracy requirement;

[0113] If the absolute value of any pixel is greater than the standard error threshold, it is determined that the accuracy evaluation result corresponding to the corresponding output result does not meet the accuracy requirement;

[0114] Wherein, atol and rtol are the preset absolute error threshold and relative error threshold respectively, and expected is the pixel value of the corresponding pixel expected to be output.

[0115] Optionally, in the accuracy evaluation unit, the corresponding relative error threshold and absolute error threshold may be set in advance for different combinations of categories of various image models and input data types.

[0116] Optionally, in the accuracy evaluation unit, the process of comparing the relative error and the absolute error may further include:

[0117] Pre-set the bit consistency comparison for the specified combination of image model category and input data type;

[0118] If the current image model and the input data type combination belong to the specified combination, the bit consistency comparison method is used to perform accuracy evaluation on the current image model for each output result.

[0119] Optionally, in the output consistency verification unit, the process of verifying the output consistency based on multiple output results corresponding to the same input may specifically include:

[0120] For the same input, determine whether the corresponding multiple output results are consistent. If they are consistent, the output consistency verification is passed; otherwise, the output consistency verification is failed.

[0121] Optionally, in the output consistency verification unit, the method for determining whether the corresponding multiple output results are consistent may specifically include: determining whether the MD5 values ​​of the multiple output results corresponding to the same input are the same; if they are the same, determining that the corresponding multiple output results are consistent; if they are not the same, determining that the corresponding multiple output results are inconsistent.

[0122] The present application also provides a computer-readable storage medium, which stores instructions, and when the instructions are executed by a processor, the steps in the upsampling method or data access method as described above can be executed. In practical applications, the computer-readable medium can be included in each device / apparatus / system in the above-mentioned embodiments, or it can exist independently without being assembled into the device / apparatus / system. Among them, the instructions are stored in the computer-readable storage medium, and when the instructions stored therein are executed by a processor, the steps in the abnormality assessment method of the image model as described above can be executed.

[0123] According to the embodiments disclosed in the present application, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof, but it is not used to limit the scope of protection of the present application. In the embodiments disclosed in the present application, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0124] The present application also provides a computer program product, comprising computer executable instructions, which, when executed by a processor, implement the above-mentioned abnormality assessment method for the image model.

[0125] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An abnormality assessment method for an image model, It is characterized in that include: Receiving a plurality of corresponding output results obtained after the current image model processes a plurality of inputs; wherein the plurality of inputs include at least two identical inputs; Based on the category of the current image model, determining an accuracy comparison method corresponding to the category; According to the determined accuracy comparison method, the accuracy of the current image model is evaluated for each output result to obtain an accuracy evaluation result corresponding to each output result; Verify the consistency of outputs based on multiple outputs corresponding to the same input.

2. The method according to claim 1, It is characterized in that The method further includes: verifying whether the current image model feeds back information indicating the input abnormality for the abnormal input received by the current image model.

3. The method according to claim 1 or 2, It is characterized in that For image models of the data handling category, the corresponding accuracy comparison method is bit consistency comparison; and / or, For image models that are not data handling categories, the corresponding accuracy comparison method is PSNR comparison and / or AI accuracy standard comparison; and / or, For image models in the standard library, evaluate in advance whether the image model needs to meet the bit consistency requirements based on the current hardware conditions and the category of the image model; if necessary, determine the corresponding accuracy comparison method as bit consistency comparison; if not, determine the corresponding accuracy comparison method as PSNR comparison and / or AI accuracy standard comparison.

4. The method according to claim 3, It is characterized in that The comparison of the AI ​​accuracy standards is a comparison combining relative error and absolute error.

5. The method according to claim 4, It is characterized in that The comparison of the relative error and the absolute error combined includes: When performing accuracy evaluation on the current image model for each output result, calculating the absolute value of the difference between each pixel of the output result and the corresponding pixel of the expected output; Compare the absolute value with the standard error threshold atol+rtol*|expected|; If the absolute value of each pixel is less than or equal to the standard error threshold, it is determined that the accuracy evaluation result corresponding to the corresponding output result meets the accuracy requirement; If the absolute value of any pixel is greater than the standard error threshold, it is determined that the accuracy evaluation result corresponding to the corresponding output result does not meet the accuracy requirement; The atol and rtol are respectively a preset absolute error threshold and a relative error threshold, and the expected is a pixel value of a corresponding pixel of the expected output.

6. The method according to claim 5, It is characterized in that The corresponding relative error threshold and absolute error threshold are set in advance for different combinations of various image model categories and input data types.

7. The method according to claim 6, It is characterized in that The comparison combining relative error and absolute error further includes: setting bit consistency comparison for a specified combination of image model category and input data type in advance; If the combination of the current image model and the input data type belongs to the specified combination, when the accuracy of the current image model is evaluated for each output result, a bit consistency comparison method is used to perform the accuracy evaluation.

8. The method according to claim 1, It is characterized in that The verification of output consistency based on multiple output results corresponding to the same input includes: For the same input, determine whether the corresponding multiple output results are consistent. If they are consistent, the output consistency verification is passed; otherwise, the output consistency verification is failed.

9. The method according to claim 8, It is characterized in that The method of determining whether the corresponding multiple output results are consistent includes: determining whether the MD5 values ​​of the multiple output results corresponding to the same input are the same, if they are the same, determining that the corresponding multiple output results are consistent, if they are not the same, determining that the corresponding multiple output results are inconsistent.

10. An abnormality assessment device for an image model, It is characterized in that include: Receiving unit, accuracy assessment unit, output consistency verification unit and output unit; The receiving unit is used to receive corresponding multiple output results obtained after the current image model processes multiple inputs; wherein the multiple inputs include at least two identical inputs; The accuracy evaluation unit is used to determine the accuracy comparison method corresponding to the category of the current image model based on the category of the current image model; and is also used to perform accuracy evaluation on the current image model for each output result according to the determined accuracy comparison method, obtain the accuracy evaluation result corresponding to each output result, and output it to the output unit; The output consistency verification unit is used to verify the output consistency based on multiple output results corresponding to the same input, and output the verification result to the output unit; The output unit is used to determine and output an abnormal evaluation result based on the accuracy evaluation result and the verification result of the output consistency.

11. The abnormality assessment device according to claim 10, It is characterized in that The device further comprises an abnormal input verification unit, which is used to verify whether the current image model feeds back information of input abnormality for the abnormal input received by the current image model, and output the verification result to the output unit; The output unit is used to further determine the abnormality assessment result based on the verification result of the abnormal input.

12. A computer-readable storage medium having computer instructions stored thereon, It is characterized in that When the instructions are executed by a processor, the abnormality assessment method of the image model described in any one of claims 1 to 9 can be implemented.

13. A computer program product, It is characterized in that The method comprises computer executable instructions, which, when executed by a processor, implement the abnormality assessment method of the image model as claimed in any one of claims 1 to 9.