Model robustness evaluation method and device, storage medium and program product

Abstract

The invention provides a model robustness evaluation method and device, a storage medium and a program product, relates to the technical field of computer vision, can evaluate the robustness of a deep neural network in an image classification task under natural noise, and provides support for constructing a more robust deep neural network model. The method comprises the steps that natural noise is added to each original image contained in a first image set, a second image set containing a plurality of noise images is obtained, and each noise image corresponds to one original image; and then, inputting the second image set into a deep neural network model to obtain a prediction category of each noise image. And finally, based on the first image set, the second image set, the category of each original image and the prediction category of each noise image, determining the robustness of the deep neural network in the image classification task under the natural noise.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and particularly to a method, device, storage medium, and program product for evaluating model robustness. Background Art

[0002] In the field of computer vision, perturbation noise is one of the important factors affecting the performance of deep neural network models. According to the source and characteristics of the noise, perturbation noise is mainly divided into systematic noise, natural noise, and adversarial noise.

[0003] Among them, natural noise mostly stems from various objective condition changes (such as electronic interference, changes in lighting conditions, etc.) encountered by image capture devices in complex environments, resulting in problems such as color deviation, defocus, distortion, and aberration in the images. These minor image changes may not pose interference to the human visual system, but deep neural network models are vulnerable to their influence and are prone to performance degradation. In this context, evaluating the robustness of deep neural network models in the face of natural noise is of great significance for establishing a safe and reliable deep neural network model.

[0004] However, there is currently a lack of a unified evaluation strategy and framework for the research on evaluating the robustness of deep neural network models under natural noise. Therefore, how to scientifically and systematically evaluate the robustness of deep neural network models under the influence of natural noise has become a technical problem to be solved urgently. Summary of the Invention

[0005] This application provides a method, device, storage medium, and program product for evaluating model robustness, which can evaluate the robustness of a deep neural network model in an image classification task under natural noise and provide support for constructing a more robust deep neural network model.

[0006] To achieve the above object, this application adopts the following technical solutions:

[0007] In a first aspect, this application provides a method for evaluating model robustness, the method includes: adding natural noise to each original image included in a first image set to obtain a second image set including a plurality of noisy images, and each noisy image corresponds to an original image. Then, input the second image set into a deep neural network model to obtain the predicted category of each noisy image. Finally, based on the first image set, the second image set, the category of each original image, and the predicted category of each noisy image, determine the robustness of the deep neural network model in an image classification task under natural noise.

[0008] Based on the above technical solution, the present application adds natural noise to each original image in the first image set, so as to simulate the natural noise interference in the real scene and provide data support for evaluating the robustness of the deep neural network model under natural noise. Then, according to the first image set, the second image set, the category of each original image, and the predicted category of each noisy image, the robustness of the deep neural network model in the image classification task under natural noise can be effectively evaluated, and further provide an optimization direction and technical basis for constructing a more robust deep neural network model.

[0009] Optionally, the natural noise includes at least one of visual attribute noise, visual noise, visual deformation noise, visual blur noise, and weather interference noise.

[0010] Among them, visual attribute noise refers to the noise caused by abnormal changes in illumination, color, or contrast. Visual noise refers to the noise that appears during image acquisition, transmission, or processing. Visual deformation noise refers to the noise caused by the deformation of the shape or structure of an object in the image. Visual blur noise refers to the noise caused by the loss of details in the image or the unclear edges of the image. Weather interference refers to the noise caused by the influence of natural weather during image acquisition.

[0011] Optionally, the natural noise included in each noisy image is the same. On this basis, based on the first image set, the second image set, the category of each original image, and the predicted category of each noisy image, to determine the robustness of the deep neural network model in the image classification task under natural noise, it may specifically include: for each noisy image, based on the noisy image and the original image corresponding to the noisy image, determine the distortion rate of the noisy image. Then, based on the predicted category of each noisy image and the category of each original image, determine the classification accuracy of the deep neural network model. Finally, based on the classification accuracy and the distortion rate of each noisy image, determine the robustness of the deep neural network model in the image classification task under natural noise.

[0012] Among them, the distortion rate of the noisy image is used to characterize the degree of difference between the noisy image and the original image corresponding to the noisy image under natural noise.

[0013] Optionally, based on the classification accuracy and the distortion rate of each noisy image, to determine the robustness of the deep neural network model in the image classification task under natural noise, it may specifically include: based on the distortion rate of each noisy image, determine the distortion rate of the second image set. Then, based on the classification accuracy and the distortion rate of the second image set, determine the classification accuracy of the deep neural network model at the distortion rate of the second image set. Finally, based on the classification accuracy of the deep neural network model at the distortion rate of the second image set, determine the robustness of the deep neural network model in the image classification task under natural noise.

[0014] Optionally, based on the distortion rate and classification accuracy of the second image set, determine the classification accuracy of the deep neural network model at the distortion rate of the second image set. Specifically, it may include: taking the product of the classification accuracy and the distortion rate of the second image set as the classification accuracy of the deep neural network model at the distortion rate of the second image set.

[0015] Optionally, there are multiple second image sets. Each second image set contains the same natural noise, and the perturbation levels of the natural noise contained in each second image set are different. The perturbation level is used to characterize the degree of interference of the natural noise on the quality of the original image.

[0016] On this basis, based on the first image set, the second image set, the category of each original image, and the predicted category of each noisy image, determine the robustness of the deep neural network model in the image classification task under natural noise. Specifically, it may further include: for each second image set, determine the distortion rate of the second image set based on the distortion rate of each noisy image in the second image set. Then, for each second image set, determine the classification accuracy of the deep neural network model for the second image set based on the predicted category of each noisy image and the category of each original image. Finally, based on the number of second image sets, the distortion rate of each second image set, and the classification accuracy of the deep neural network model for each second image set, determine the robustness of the deep neural network model in the image classification task under natural noise.

[0017] Among them, the distortion rate of each noisy image is used to characterize the degree of difference between the noisy image and the original image corresponding to the noisy image under natural noise.

[0018] Optionally, there are multiple second image sets, and the natural noise contained in each second image set is different. On this basis, based on the first image set, the second image set, the category of each original image, and the predicted category of each noisy image, determine the robustness of the deep neural network model in the image classification task under natural noise. Specifically, it may further include: for each second image set, determine the distortion rate of the second image set based on the distortion rate of each noisy image in the second image set. Then, for each second image set, determine the classification accuracy of the deep neural network model for the second image set based on the predicted category of each noisy image and the category of each original image. Finally, based on the distortion rate of each second image set and the classification accuracy of the deep neural network model for each second image set, determine the robustness of the deep neural network model in the image classification task under different natural noises.

[0019] In a second aspect, the present application provides a model robustness evaluation device, including:

[0020] A first processing unit, configured to add natural noise to each original image included in a first image set to obtain a second image set including a plurality of noisy images.

[0021] A second processing unit, configured to input the second image set into a deep neural network model to obtain a predicted class of each noisy image.

[0022] A determination unit, configured to determine the robustness of the deep neural network model in an image classification task under natural noise based on the first image set, the second image set, the class of each original image, and the predicted class of each noisy image.

[0023] Optionally, the determination unit is further configured to, for each noisy image, determine a distortion rate of the noisy image based on the noisy image and the original image corresponding to the noisy image.

[0024] Optionally, the determination unit is further configured to determine a classification accuracy of the deep neural network model based on the predicted class of each noisy image and the class of each original image.

[0025] Optionally, the determination unit is further configured to determine the robustness of the deep neural network model in an image classification task under natural noise based on the classification accuracy and the distortion rate of each noisy image.

[0026] Optionally, the determination unit is further configured to determine a distortion rate of the second image set based on the distortion rate of each noisy image.

[0027] Optionally, the determination unit is further configured to determine a classification accuracy of the deep neural network model at the distortion rate of the second image set based on the classification accuracy and the distortion rate of the second image set.

[0028] Optionally, the determination unit is further configured to determine the robustness of the deep neural network model in an image classification task under natural noise based on the classification accuracy of the deep neural network model at the distortion rate of the second image set.

[0029] Optionally, the determination unit is further configured to, for each second image set, determine a distortion rate of the second image set based on the distortion rate of each noisy image in the second image set.

[0030] Optionally, the determination unit is further configured to determine the robustness of the deep neural network model in an image classification task under natural noise based on the number of second image sets, the distortion rate of each second image set, and the classification accuracy of the deep neural network model for each second image set.

[0031] Optionally, the determination unit is further configured to determine the robustness of the deep neural network model in an image classification task under different natural noises based on the distortion rate of each second image set and the classification accuracy of the deep neural network model for each second image set.

[0032] In a third aspect, the present application provides a model robustness evaluation device, which includes: a processor and a communication interface. The communication interface is coupled to the processor, and the processor is configured to run a computer program or instruction to implement the model robustness evaluation method described in any one of the first aspect and any possible implementation manner of the first aspect.

[0033] In a fourth aspect, the present application provides a computer-readable storage medium, in which instructions are stored. When the instructions are run on a terminal, the terminal is caused to execute the model robustness evaluation method described in any one of the first aspect and any possible implementation manner of the first aspect.

[0034] In a fifth aspect, the present application provides a computer program product, which includes computer instructions. When the computer instructions are run on a computer, the computer is caused to execute the model robustness evaluation method described in any one of the first aspect and any possible implementation manner of the first aspect.

[0035] It can be understood that the beneficial effects that can be achieved by the above-mentioned second aspect to the fifth aspect can refer to the beneficial effects in any possible design manner of the model robustness evaluation method described in any one of the first aspect and any possible implementation manner of the first aspect, and will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic architecture diagram of a model robustness evaluation system provided by an embodiment of the present application;

[0037] Figure 2 It is a schematic flowchart of a model robustness evaluation method provided by an embodiment of the present application;

[0038] Figure 3 It is a schematic diagram of different natural noises provided by an embodiment of the present application;

[0039] Figure 4 It is a schematic flowchart of a process for obtaining a second image set containing visual attribute noise provided by an embodiment of the present application;

[0040] Figure 5 It is a schematic flowchart of a process for obtaining a second image set containing visual noise provided by an embodiment of the present application;

[0041] Figure 6 It is a schematic flowchart of a process for obtaining a second image set containing visual deformation noise provided by an embodiment of the present application;

[0042] Figure 7 It is a schematic flowchart of a process for obtaining a second image set containing visual blur noise provided by an embodiment of the present application;

[0043] Figure 8 A schematic flowchart of a process for obtaining a second image set containing weather interference noise provided by an embodiment of the present application;

[0044] Figure 9 A schematic structural diagram of a model robustness evaluation device provided by an embodiment of the present application;

[0045] Figure 10 A schematic structural diagram of another model robustness evaluation device provided by an embodiment of the present application. Detailed implementation manners

[0046] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0047] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone.

[0048] The terms "first" and "second" in the specification and drawings of the present application are used to distinguish different objects or different treatments of the same object, rather than to describe the specific order of the object.

[0049] In addition, the terms "including" and "having" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally further include other steps or units not listed, or may optionally further include other steps or units inherent to these processes, methods, products, or devices.

[0050] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, using words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0051] In the description of the present application, unless otherwise specified, the meaning of "a plurality" refers to two or more.

[0052] In recent years, deep learning technology has achieved remarkable results in multiple fields such as image recognition, speech recognition, and natural language processing, driving the rapid development of artificial intelligence technology. Especially in the field of computer vision, the application of deep neural networks has greatly improved the performance of tasks such as image classification, face detection, object detection, and semantic segmentation.

[0053] However, with the wide application of deep learning technology, deep neural networks are also facing more and more external attack challenges. Attackers can generate adversarial samples by constructing tiny perturbations, which can mislead the deep neural network model into making wrong judgments. This phenomenon reveals the vulnerability of deep neural networks when facing attacks and also exposes its limitations in terms of robustness.

[0054] In the field of computer vision, the perturbation noises in images are mainly divided into systematic noise (which can also be called image processing noise), natural noise, and adversarial noise. Among them, adversarial noise is generated depending on the internal information of the model and can construct adversarial samples through the gradient information calculated by the backpropagation algorithm. Systematic noise has nothing to do with the deep neural network model or the input image, but is caused by the software and hardware inconsistencies in the training and inference systems of the deep neural network model. This kind of inconsistency may stem from various reasons, such as differences in preprocessing operations between different deep learning platforms, support limitations of algorithm libraries on edge devices, and the use of different algorithm libraries in the training and inference stages.

[0055] Natural noise mostly results from various objective condition changes encountered by image capture devices in complex environments, such as electronic interference, changes in lighting conditions, etc., leading to problems such as color deviation, defocus, distortion, and aberration in the image data. Compared with deep neural networks, the human visual system has stronger robustness and can well handle the tiny changes in images. However, for deep neural networks, these tiny changes may lead to a significant decline in their performance. Against this background, evaluating the robustness of deep neural network models when facing natural noise is of great significance for establishing secure and reliable deep neural network models.

[0056] Currently, related technologies have proposed research on the robustness of deep neural network models in image classification tasks for common perturbation noises. Based on the ImageNet image dataset, two benchmark test datasets have been established, including various types of visual perturbations such as noise, blur, weather, translation, rotation, and scaling, providing an important tool for evaluating the robustness of deep neural network models when facing natural noise. In addition, related technologies have been extended on this basis, and extensive evaluations have been carried out on various different deep neural network structures, further promoting the in-depth development of this field.

[0057] However, there is currently a lack of unified evaluation strategies and frameworks for the natural noise robustness evaluation of deep neural network models. At the same time, when facing real scenarios, there is a lack of effective scalable evaluation indicators. Especially in computer vision tasks such as image classification, face detection, object detection, and semantic segmentation, there are significant differences in evaluation indicators, making it difficult to form a unified evaluation standard.

[0058] In summary, although deep learning technology has made significant progress in the field of computer vision, how to scientifically and systematically evaluate the robustness of deep neural network models under the influence of natural noise has become a key technical problem that needs to be solved urgently.

[0059] In order to solve the above technical problems, an embodiment of the present application provides a model robustness evaluation method, which can add natural noise to each original image in the first image set to obtain a second image set, so as to simulate the natural noise interference in the real scene and provide data support for evaluating the robustness of the deep neural network model under natural noise. Then, the second image set can be input into the deep neural network model to obtain the predicted category of each noise image in the second image set. Finally, based on the first image set, the second image set, the predicted category of each noise image and the category of each original image, the robustness of the deep neural network model in the image classification task under natural noise can be effectively evaluated, providing optimization direction and technical basis for building a more robust deep neural network model.

[0060] The implementation of the embodiments of the present application will be described in detail below in conjunction with the accompanying drawings.

[0061] Figure 1 The following is a schematic diagram of the architecture of a model robustness evaluation system provided in an embodiment of the present application. Figure 1 As shown, the model robustness evaluation system includes: a terminal 101 and a server 102.

[0062] The terminal 101 may be a device that provides voice and / or data connectivity to a user, a device with wireless connection function, or other device connected to a wireless modem. The terminal device may be at least one of a desktop computer, a laptop, a wireless terminal, and a laptop portable computer. In one embodiment, the electronic device has a communication function and can access a wired network or a wireless network.

[0063] The embodiment of the present application does not limit the number of terminals 101 in the model robustness evaluation system, and may include: Figure 1 More or fewer terminals 101.

[0064] The server 102 can be a high-performance server that provides various services on the Internet. It can be an independent physical server, a server cluster composed of multiple physical servers, or at least one of the cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks, and big data or artificial intelligence platforms. The embodiments of the present application do not limit this. Of course, the server can also include other functions to provide more comprehensive and diverse services.

[0065] The embodiments of the present application do not limit the number of servers 102 in the model robustness evaluation system, and may include more or fewer servers 102 than Figure 1 those in

[0066] In some embodiments, multiple deep neural network models are deployed in the server 102. On this basis, the user holding the terminal 101 can select the deep neural network model to be evaluated through the model robustness evaluation system deployed in the terminal 101, upload the first image set (i.e., the original image set), and trigger the evaluation operation. The terminal 101 can send an evaluation message to the server 102 in response to the user's evaluation operation. The evaluation message includes the first image set and the deep neural network model to be evaluated.

[0067] After receiving the evaluation message, the server 102 can add natural noise to each original image included in the first image set to obtain a second image set including multiple noisy images. Then, the server 102 can input the second image set into the deep neural network model to be evaluated to obtain the predicted category of each noisy image. Finally, the server 102 can determine the robustness of the deep neural network model in the image classification task under natural noise based on the first image set, the second image set, the category of each original image, and the predicted category of each noisy image.

[0068] Figure 2 It is a schematic flowchart of a model robustness evaluation method provided by the embodiments of the present application. As Figure 2 shown, this method is executed by the Figure 1 server shown, and this method includes:

[0069] S201. Add natural noise to each original image included in the first image set to obtain a second image set including multiple noisy images.

[0070] Wherein, each noisy image corresponds to an original image.

[0071] In the embodiments of the present application, natural noise may include at least one of visual attribute noise, visual noise, visual deformation noise, visual blur noise, and weather interference noise. Among them, visual attribute noise refers to the noise caused by abnormal changes in illumination, color, or contrast. Visual noise refers to the noise that appears during image acquisition, transmission, or processing. Visual deformation noise refers to the noise caused by the deformation of the shape or structure of an object in the image. Visual blur noise refers to the noise caused by the loss of details in the image or the unclear edges of the image. Weather interference refers to the noise caused by the influence of natural weather during image acquisition.

[0072] In some embodiments, image processing software is installed in the server, and the image processing software is used to perform various image processing operations (such as adding noise) on the image. The user of the holding terminal can upload the first image set (i.e., the original image set) to the model robustness evaluation system deployed in the terminal, and then trigger the submission operation. After that, the terminal can send a noise addition message to the server in response to the user's submission operation, and the noise addition message may include the first image set.

[0073] After receiving the noise addition message, the server can use the image processing software to read each original image included in the first image set. Then, for each original image in the first image set, the server can use the image processing software to add natural noise to the original image to obtain the noise image corresponding to the original image.

[0074] S202. Input the second image set into the deep neural network model to obtain the predicted category of each noise image.

[0075] In the embodiments of the present application, the predicted category of each noise image refers to the predicted category obtained by inputting the noise image into the deep neural network model.

[0076] The embodiments of the present application do not limit the type of the deep neural network model. For example, the deep neural network model may include a residual convolutional neural network (ResNet) model, a densely connected convolutional network (DenseNet), etc.

[0077] In some embodiments, a deep neural network model is deployed in the server. The server can input each noise image in the second image set into the deep neural network model. For each noise image, the deep neural network model can identify and classify the object included in the noise image to obtain the predicted category of the noise image.

[0078] Exemplarily, the second image set includes a noisy image A(x1) and a noisy image A(x2). The noisy image A(x1) contains a cat, and the noisy image A(x2) contains a dog. Based on this, after the server inputs the noisy image A(x1) and the noisy image A(x2) into the deep neural network model, the category of the noisy image A(x1) can be obtained as cat, and the category of the noisy image A(x2) as dog.

[0079] S203. Determine the robustness of the deep neural network model in the image classification task under natural noise based on the first image set, the second image set, the category of each original image, and the predicted category of each noisy image.

[0080] In the embodiments of the present application, the robustness of the deep neural network model in the image classification task under natural noise refers to the ability of the deep neural network model to still maintain a high classification accuracy and stability when facing natural noise in the image. Specifically, the robustness can reflect the tolerance of the deep neural network model to natural noise, that is, when the original image is interfered by natural noise, whether the deep neural network model can correctly identify the image content and output an accurate prediction result.

[0081] In the embodiments of the present application, the category of the original image refers to the category to which the object contained in the original image manually labeled belongs. For example, if the original image x i contains the object cat, the category of the original image can be labeled as cat.

[0082] In some embodiments, the server can determine the classification accuracy rate of the deep neural network model based on the predicted category of each noisy image, the category of each original image, and the number of noisy images included in the second image set. Then, the server can determine the robustness of the deep neural network model in the image classification task under natural noise based on the classification accuracy rate of the deep neural network model and the degree of difference between each noisy image and the original image corresponding to each noisy image.

[0083] Based on the above technical solutions, the present application can simulate the natural noise interference in the real scene by adding natural noise to each original image in the first image set, providing reliable data support for evaluating the robustness of the deep neural network model under natural noise. In addition, by comprehensively considering the classification accuracy rate of the deep neural network model and the degree of difference between the noisy image and the original image, the robustness of the deep neural network model under natural noise types can be more comprehensively evaluated, thereby providing support for the optimization and improvement of the deep neural network model.

[0084] In the embodiments of the present application, the natural noise included in each noise image in the second image set may be the same or different. The following takes the following two cases as examples to introduce the above S203: Case 1: The natural noise included in each noise image is the same; and Case 2: The natural noise included in each noise image is different.

[0085] 1. The natural noise included in each noise image is the same.

[0086] In the embodiments of the present application, the second image set may be one or multiple. The following takes the following two cases as examples to introduce the above S203: Case A: The second image set is one; and Case B: The second image set is multiple.

[0087] A. The second image set is one.

[0088] In some embodiments, when the natural noise included in each noise image in the second image set is the same, the above S203 may specifically include the following S203A - S203C.

[0089] S203A: For each noise image, based on the noise image and the original image corresponding to the noise image, determine the distortion rate of the noise image.

[0090] Among them, the distortion rate of the noise image is used to characterize the degree of difference between the noise image and the original image corresponding to the noise image under natural noise. The distortion rate of the noise image is positively correlated with the noise content of the noise image. The noise content refers to the amount of natural noise included in the noise image. For example, the higher the distortion rate of the noise image, the more natural noise the noise image contains, indicating that the noise image and the original image corresponding to the noise image are less visually similar. The smaller the distortion rate of the noise image, the less natural noise the noise image contains, indicating that the noise image and the original image corresponding to the noise image are more visually similar.

[0091] In some embodiments, for each noise image, the server may use a preset image distortion rate algorithm to calculate the degree of difference between the noise image and the original image corresponding to the noise image, and obtain the distortion rate of the noise image.

[0092] The embodiments of the present application do not limit the preset distortion rate algorithm. For example, the preset distortion rate algorithm may include an image quality evaluation algorithm based on a convolutional neural network, a distortion evaluation algorithm based on a visual attention model, an L2 norm method, etc.

[0093] Exemplarily, taking the server using the L2 norm to obtain the distortion rate of the noise image as an example. Assume that each noise image in the second image set contains visual noise. The server may obtain the distortion rate of the noise image with reference to the following formula 1.

[0094]

[0095] Among them, x i represents the original image, represents the image with visual noise (i.e., the noise image), ||·||2 represents the L2 norm, s represents the perturbation level of the visual noise, and j represents the j-th type of sub-noise included in the visual noise, represents the noise image of the distortion rate.

[0096] S203B. Based on the predicted category of each noise image and the category of each original image, determine the classification accuracy of the deep neural network model.

[0097] In some embodiments, for each noise image, the server can determine whether the predicted category of the noise image is consistent with the category of the original image corresponding to the noise image, and obtain the judgment result corresponding to the noise image. Then, the server can determine the classification accuracy of the deep neural network model based on the judgment result corresponding to each noise image and the number of noise images included in the second image set.

[0098] Exemplarily, the server can determine the classification accuracy of the deep neural network model with reference to the following formula 2.

[0099]

[0100] Among them, represents the classification accuracy of the deep neural network model. y i represents the category of the original image, represents the predicted category of the noise image, The specific calculation formula of can refer to the following formula 3. N represents the number of noise images included in the second image set. represents the indicator function. When and y i are equal, the value of is 1. When and y i are not equal, the value of is 0.

[0101]

[0102] Among them, f θ (·) is the deep neural network model, is the noise image.

[0103] S203C. Based on the classification accuracy and the distortion rate of each noise image, determine the robustness of the deep neural network model in the image classification task under natural noise.

[0104] In an alternative embodiment, the server may determine the distortion rate of the second image set based on the distortion rate of each noisy image. Subsequently, the server may determine the classification accuracy of the deep neural network model at the distortion rate of the second image set based on the classification accuracy and the distortion rate of the second image set. Finally, the server may determine the robustness of the deep neural network model in the image classification task under natural noise based on the classification accuracy of the deep neural network model at the distortion rate of the second image set.

[0105] In some embodiments, the server may determine the average value of the distortion rates of all the noisy images included in the second image set as the distortion rate of the second image set. Subsequently, the server may use the product of the distortion rate of the second image set and the classification accuracy as the classification accuracy of the deep neural network model at the distortion rate of the second image set (hereinafter referred to as the visual perception adversarial classification accuracy corresponding to the second image set). Finally, the server may determine the robustness of the deep neural network model in the image classification task under natural noise based on the visual perception adversarial classification accuracy corresponding to the second image set.

[0106] Exemplarily, the calculation formula for the distortion rate of the second image set may refer to Formula 4 below, and the calculation formula for the visual perception adversarial classification accuracy corresponding to the second image set may refer to Formula 5 below.

[0107]

[0108] where N represents the number of noisy images included in the second image set, represents the distortion rate of the second image set.

[0109]

[0110] where represents the distortion rate of the second image set, represents the classification accuracy of the deep neural network model, represents the visual perception adversarial classification accuracy corresponding to the second image set.

[0111] Based on the above technical solutions, by calculating the distortion rate of the noisy image relative to the original image, the impact of noise on the image quality can be quantified, and thus the information loss of the image under noise interference can be understood. By comprehensively considering the distortion rate of the second image set and the classification accuracy of the deep neural network model, the robustness of the deep neural network model under natural noise types can be evaluated more accurately and comprehensively, so that users can take targeted measures to optimize the deep neural network model, thereby improving the performance of the deep neural network model in actual scenarios.

[0112] B. There are multiple second image sets.

[0113] In some embodiments, when there are multiple second image sets, and the natural noise included in each second image set is the same while the perturbation levels of the natural noise included in each second image set are different, the above S203 may specifically include the following steps:

[0114] Among them, the perturbation level of natural noise can be used to characterize the degree of interference of natural noise on the quality of the original image. The perturbation level of natural noise is positively correlated with the degree of interference of natural noise on the quality of the original image. For example, the higher the perturbation level, the greater the degree of interference of natural noise on the quality of the original image, resulting in problems such as blurring, distortion, loss of details, and decrease in resolution of the original image. The lower the perturbation level, the smaller the degree of interference of natural noise on the quality of the original image, enabling the original image to maintain a high level of clarity and image details, that is, the difference between the noisy image and the original image is small.

[0115] The embodiments of the present application do not limit the representation method of the perturbation level of natural noise. For example, the perturbation level can be represented by numbers, such as from level 1 to level 6, or for another example, the perturbation level can be represented by letters, such as from level A to level Z.

[0116] In an alternative embodiment, for each second image set, the server can determine the distortion rate of the second image set based on the distortion rate of each noisy image in the second image set. After that, the server can determine the classification accuracy of the deep neural network model for the second image set based on the predicted category of each noisy image and the category of each original image. On this basis, the server can determine the robustness of the deep neural network model in the image classification task under natural noise based on the number of second image sets, the distortion rate of each second image set, and the classification accuracy of the deep neural network model for each second image set.

[0117] In some embodiments, the server can determine the distortion rate of each second image set with reference to the method in the above S203C. In addition, the server can also determine the classification accuracy of the deep neural network model for each second image set with reference to the method in the above S203B. After determining the distortion rate of each second image set and the classification accuracy of the deep neural network model for each second image set, the server can also determine the visual perception adversarial classification accuracy corresponding to each second image set with reference to the method in the above S203C, which will not be elaborated here.

[0118] On the above basis, the server can determine the ratio between the sum of the visual perception adversarial classification accuracies corresponding to each second image set and the number of second image sets as the average visual perception adversarial classification accuracy corresponding to the multiple second image sets. Finally, the server can determine the robustness of the deep neural network model in the image classification task under natural noise based on the average visual perception adversarial classification accuracy.

[0119] Exemplarily, assume that the number of the second image sets is S. Based on this, the server can calculate the average visual perception adversarial classification accuracy corresponding to multiple second image sets with reference to the above formula 5 and the following formula 6.

[0120]

[0121] Among them, represents the visual perception adversarial classification accuracy corresponding to the second image set, S represents the number of the second image sets, and mACAVP j represents the average visual perception adversarial classification accuracy corresponding to multiple second image sets.

[0122] Based on the above technical solution, by combining the distortion rate of each second image set and the classification accuracy of each second image set by the deep neural network model, more data support can be provided for evaluating the robustness of the deep neural network model, thereby improving the accuracy of the robustness evaluation of the deep neural network model under natural noise.

[0123] II. The natural noise contained in each noise image is different.

[0124] In some embodiments, when there are multiple second image sets and the natural noise contained in each second image set is different, the above S203 may specifically include:

[0125] In an optional implementation manner, for each second image set, the server can determine the distortion rate of the second image set based on the distortion rate of each noise image in the second image set. After that, the server can determine the classification accuracy of the deep neural network model for the second image set based on the predicted category of each noise image and the category of each original image. Finally, the server can determine the robustness of the deep neural network model in the image classification task under different natural noises based on the distortion rate of each second image set and the classification accuracy of the deep neural network model for each second image set.

[0126] In some embodiments, the server can determine the distortion rate of each second image set with reference to the method in the above S203C. In addition, the server can also determine the classification accuracy of the deep neural network model for each second image set with reference to the method in the above S203B. After determining the distortion rate of each second image set and the classification accuracy of the deep neural network model for each second image set, the server can also determine the visual perception adversarial classification accuracy corresponding to each second image set with reference to the method in the above S203C, which will not be elaborated here.

[0127] Based on the above, the server can determine the visual perception adversarial classification accuracy corresponding to multiple second image sets based on the average visual perception adversarial classification accuracy corresponding to each second image set and the number of second image sets, and determine the robustness of the deep neural network model in the image classification task under different natural noises based on the visual perception adversarial classification accuracy corresponding to multiple second image sets.

[0128] Exemplarily, assume that there are 5 second image sets, and each second image set contains different visual noises. Based on this, the server can refer to the above formula 5 to obtain the visual perception adversarial classification accuracy corresponding to each second image set. After that, the server can refer to the following formula 7 to obtain the visual perception adversarial classification accuracy corresponding to 5 second image sets.

[0129]

[0130] Among them, MACAV(A) can represent the visual perception adversarial classification accuracy corresponding to multiple second image sets with visual noises. MACAV(A) can be used to evaluate the robustness of the deep neural network model under visual noises.

[0131] Exemplarily, assume that there are 5 second image sets, and each second image set contains different natural noises. Among them, the visual perception adversarial classification accuracy corresponding to the second image set with visual noise is MACAV(A), the visual perception adversarial classification accuracy corresponding to the second image set with visual attribute noise is MACAV(R), the visual perception adversarial classification accuracy corresponding to the second image set with visual blur noise is MACAV(B), the visual perception adversarial classification accuracy corresponding to the second image set with visual deformation noise is MACAV(I), and the visual perception adversarial classification accuracy corresponding to the second image set with weather interference noise is MACAV(H). Based on this, the server can refer to the following formula 8 to obtain the visual perception adversarial classification accuracy MACAV_average corresponding to 5 second image sets. MACAV_average can be used to evaluate the robustness of the deep neural network model in the image classification task under different natural noises.

[0132]

[0133] Based on the above technical solutions, by evaluating the robustness of the deep neural network model under various natural noises, the robustness of the deep neural network model in dealing with various natural noises can be comprehensively reflected, providing a basis for constructing a more stable and reliable deep neural network model.

[0134] 1. Visual attribute noise

[0135] The visual attributes of an image refer to various features that can be perceived and recognized by the visual system in the image. Such asFigure 3 As shown, visual attributes may include brightness, contrast, saturation, hue, and color temperature. Visual attributes can either be inherent to the image itself or added through image processing techniques.

[0136] Specifically, as Figure 4 shown, the server can obtain a second image set with visual attribute noise by adjusting the brightness, contrast, saturation, hue, or color temperature of each original image in the first image set.

[0137] The following will introduce the specific methods for adjusting the brightness, contrast, saturation, hue, or color temperature of each original image.

[0138] 1.1. Brightness

[0139] Brightness, that is, the light and dark degree of the image, is usually related to the intensity of light. In image processing, adjusting the image brightness can make the image look brighter or darker without changing the image contrast or color.

[0140] Specifically, for each original image in the first image set, the server can adjust the brightness value of each pixel point in the original image to obtain a noise image after brightness adjustment.

[0141] Exemplarily, the server can adjust the brightness of the original image with reference to the following formula 9.

[0142] I brightness (x,y) = I clean (x,y) + b (Formula 9)

[0143] Among them, (x,y) represents the coordinate position of the pixel point in the original image. I clean (x,y) represents the pixel value of the pixel point (x,y) in the original image, and the pixel point (x,y) is the pixel point with coordinates (x,y) in the image. I brightness (x,y) represents the pixel value of the pixel point (x,y) in the noise image after brightness adjustment. b represents the brightness value. When b > 0, the image brightness increases; when b < 0, the image brightness decreases; when b = 0, the image brightness remains unchanged.

[0144] 1.2. Contrast

[0145] Contrast refers to the degree of difference between the brightest and darkest parts of the image. Images with high contrast usually have clear and distinct edges and details, while images with low contrast appear relatively blurred and the details are not prominent enough.

[0146] Specifically, for each original image in the first image set, the server can adjust the contrast factor of each pixel point in the original image to obtain a noise image with adjusted contrast.

[0147] Exemplarily, the server can adjust the contrast of the original image with reference to Formula 10 below.

[0148] I contrast (x,y) = aI clean (x,y) (Formula 10)

[0149] Among them, I Contrast (x,y) represents the pixel value of the pixel point (x,y) in the noise image after contrast adjustment. a represents the contrast factor. When a > 1, the image contrast increases. When 0 < a < 1, the image contrast decreases. When a = 0, the image contrast remains unchanged. The meaning of I clean (x,y) can be referred to the introduction in 1.1 above.

[0150] 1.3. Saturation

[0151] Saturation refers to the purity or intensity of a color. An image with high saturation has bright colors, while an image with low saturation appears duller, even approaching a grayscale image.

[0152] Specifically, the server can convert each original image in the first image set from the RGB color space to the HSV color space to obtain each original image in the HSV color space. Then, for each original image in the HSV color space, the server can adjust the saturation factor of each pixel point in the original image to obtain a noise image with adjusted saturation.

[0153] Exemplarily, the server can adjust the saturation of the original image with reference to Formula 11 below.

[0154] I saturation (x,y) = cHSV(I clean (x,y)) (Formula 11)

[0155] Among them, HSV(I clean (x,y)) represents the pixel value of the pixel point (x,y) in the original image in the HSV color space. I saturation (x,y) represents the pixel value of the pixel point (x,y) in the noise image after saturation adjustment. c represents the saturation factor. When c > 1, the image saturation increases, that is, the image colors are more vivid. When 0 < c < 1, the image saturation decreases. When c = 0, the image saturation remains unchanged. The meaning of I clean (x,y) can be referred to the introduction in 1.1 above.

[0156] 1.4. Hue

[0157] Hue is the type or category of color, which is a visual attribute for differentiating different colors. For example, red, orange, yellow, green, blue, purple, etc. are different hues.

[0158] In the HSV color space, hue is usually represented by an angle, and the value range is generally from 0° to 360°. Different angles correspond to different colors. 0° or 360° represents red, 60° represents yellow, 120° represents green, 180° represents cyan, 240° represents blue, 300° represents magenta, etc., thus forming a color wheel.

[0159] Specifically, in combination with the above content, the server can first convert each original image in the first image set from the RGB color space to the HSV color space to obtain each original image in the HSV color space. Then, for each original image in the HSV color space, the server can adjust the hue value of each pixel point in the original image and perform modulo operation on the adjusted image to obtain a noise image with adjusted hue.

[0160] Exemplarily, the server can adjust the hue of the original image with reference to the following formula 12.

[0161] I hue (x,y) = (HSV(I clean (x,y)) + d) mod 360 (Formula 12)

[0162] Among them, I hue (x,y) represents the pixel value of the pixel point (x,y) in the noise image with adjusted hue. d represents the hue value, and mod represents the modulo operation. Through the modulo operation, it can be ensured that the hue value of the adjusted noise image is within the range of 0° to 360°. The meaning of I clean (x,y) can be referred to the introduction in 1.1 above, and the meaning of (HSV(I clean (x,y)) can be referred to the introduction in 1.3 above.

[0163] 1.5. Color Temperature

[0164] Color temperature is used to describe the color characteristics of a light source. For example, a light source with a low color temperature (such as a candle) emits warm-colored light that is biased towards red and yellow, while a light source with a high color temperature emits cold-colored light that is biased towards blue.

[0165] Specifically, for each original image, the server can adjust the intensity values of the red, green, and blue channels of each pixel point in the original image to obtain a noise image with adjusted color temperature.

[0166] Exemplarily, the server can adjust the color temperature of the original image with reference to the following formula 13.

[0167] I temperature (x, y) = I clean (x, y) + e (Formula 13)

[0168] where e represents the intensity value, and I temperature (x, y) represents the pixel value of the pixel point (x, y) in the noise image after adjusting the color temperature. L clean (x, y) can be referred to the introduction in 1.1 above.

[0169] 2. Visual Noise

[0170] Visual noise refers to the noise that appears during image acquisition, transmission, or processing. For example Figure 3 As shown, visual noise can include Gaussian noise, speckle noise, Poisson noise, salt-and-pepper noise, and impulse noise.

[0171] Specifically, as Figure 5 shown, the server can add Gaussian noise, speckle noise, Poisson noise, salt-and-pepper noise, or impulse noise to each original image included in the first image set to obtain a second image set with visual noise.

[0172] The following will introduce the specific methods of adding Gaussian noise, speckle noise, Poisson noise, salt-and-pepper noise, or impulse noise to the original image.

[0173] 2.1 Gaussian Noise

[0174] Gaussian noise refers to the noise whose probability density function follows a normal distribution (also known as Gaussian distribution), usually manifested as randomly distributed fine particles on the image, similar to dust or stains.

[0175] Specifically, Gaussian noise usually appears in the form of additive noise, that is, adding a random variable that follows a normal distribution to each pixel point in the image to simulate the actual noise. On this basis, for each original image, the server can add a random variable that follows a normal distribution to each pixel point in the original image to obtain a noise image with Gaussian noise.

[0176] Exemplarily, the server can add Gaussian noise to the original image with reference to the following Formula 14.

[0177] I gaussian_noise (x, y) = I clean (x, y) + N(0, σ 2 ) (Formula 14)

[0178] Among them, I gaussian_noise (x, y) represents the pixel value of the pixel point (x, y) in the image with Gaussian noise. N(0, σ 2 ) represents a normal distribution random variable with a mean of 0 and a variance of σ 2 . The meaning of I clean (x, y) can be referred to the introduction in 1.1 above.

[0179] 2.2, Speckle Noise

[0180] Speckle noise is a common multiplicative noise, usually generated by physical processes such as coherent scattering or multipath propagation in coherent imaging systems. For example, speckle noise is likely to appear in laser imaging, synthetic aperture radar imaging, and ultrasonic imaging.

[0181] Specifically, speckle noise usually appears in the form of multiplicative noise, that is, multiplying the noise by the original image to simulate the actual speckle noise. On this basis, for each original image, the server can multiply each pixel point in the original image by a random variable subject to a normal distribution to obtain a noisy image with speckle noise.

[0182] Exemplarily, the server can add speckle noise to the original image with reference to the following formula 15.

[0183] I speckle_noise (x, y) = I clean (x, y)(1 + N(0, σ 2 ) (Formula 15)

[0184] Among them, I speckle_noise (x, y) represents the pixel value of the pixel point (x, y) in the image with speckle noise. The meaning of I clean (x, y) can be referred to the introduction in 1.1 above, and the meaning of N(0, σ 2 ) can be referred to the introduction in 2.1 above.

[0185] 2.3, Poisson Noise

[0186] Poisson noise, also known as shot noise, is the noise caused by the randomness of photons reaching the sensor, and is usually obvious in images taken under low light conditions or high ISO settings.

[0187] Specifically, for each original image, the server can process each pixel point in the original image based on a Poisson distribution random variable to obtain a noisy image with Poisson noise.

[0188] Exemplarily, the server can add Poisson noise to the original image through the following formula 16.

[0189] I poisson_noise (x, y) = Poisson(I clean (x, y)) (Equation 16)

[0190] where Poisson(I clean (x, y)) represents a Poisson distribution random variable with a mean of I clean (x, y). I poisson_noise (x, y) is the pixel value of the pixel point (x, y) in the image with Poisson noise. I clean (x, y) can be referred to the introduction in 1.1 above.

[0191] 2.4, Salt-and-Pepper Noise

[0192] Salt-and-pepper noise is a randomly occurring black-and-white noise, which appears as randomly occurring white or black pixel points on the image. This kind of noise is usually caused by sudden interference or errors in the processes such as image sensors, transmission channels, and decoding processing.

[0193] Specifically, for each original image, the server can generate a random number with a uniform distribution for each pixel point in the original image. Then, the server can randomly set a noise probability value. On this basis, the server can add salt-and-pepper noise to each pixel point in the original image based on the noise probability value and each random number to obtain a noisy image with salt-and-pepper noise.

[0194] Exemplarily, the server can add salt-and-pepper noise to the original image with reference to Equation 17 below.

[0195]

[0196] where γ represents a random number sampled from the uniform distribution U(0, 1), 0 ≤ γ ≤ 1, and the uniform distribution U(0, 1) is a continuous probability distribution, 0 is the lower bound of the uniform distribution, and 1 is the upper bound of the uniform distribution. p represents the probability value of noise generation. I saltpepper_noise (x, y) represents the pixel value of the pixel point (x, y) in the image with salt-and-pepper noise. I clean (x, y) can be referred to the introduction in 1.1 above.

[0197] When , I saltpepper_noise (x, y) is 0, that is, the pixel of the pixel point (x, y) in the original image is set to black. When , I saltpepper_noise (x, y) is 255, that is, the pixel of the pixel point (x, y) in the original image is set to white. In other cases, I saltpepper_noise (x, y) and I clean(x, y) is equal, that is, the pixel value of the pixel point (x, y) in the original image is not changed.

[0198] 2.5, Impulse Noise

[0199] Impulse noise refers to isolated noise points or noise pulses in an image with significantly different pixel values from the surrounding pixels.

[0200] Specifically, for each original image in the first image set, the server can generate a uniformly distributed random number for each pixel point in the original image. Then, the server can randomly set a noise probability value and a set of impulse noises. On this basis, the server can process each pixel point in the original image based on the noise probability value, the set of impulse noises, and each random number to obtain a noisy image with impulse noise.

[0201] Exemplarily, the server can add impulse noise to the original image with reference to the following formula 18.

[0202]

[0203] Where, I impulse_noise (x, y) represents the pixel value of the pixel point (x, y) in the image with impulse noise. v i represents any noise value in the set of impulse noises V = {v1, v2,..., v n}. When γ < p, I impulse_noise (x, y) is v i . In other cases, I impulse_noise (x, y) and I clean (x, y) are equal, that is, the pixel value of the pixel point (x, y) in the original image is not changed. I clean (x, y) can refer to the introduction in 1.1 above.

[0204] 3. Visual Deformation Noise

[0205] Visual deformation refers to geometric or structural changes in the visual perception of an image caused by various factors. These deformations are mainly caused by imaging conditions (such as viewing angle changes, lens distortion), physical effects (such as thermal expansion and contraction, gravity), etc. As Figure 3 shown, visual deformation can include elastic transformation, rotation transformation, scaling transformation, random masking, and random crop.

[0206] Specifically, as Figure 6As shown, the server can perform elastic transformation, rotation transformation, scaling transformation, random masking or random cropping on each original image in the first image set to obtain a second image set with visual deformation noise.

[0207] The following will introduce the specific methods of how to perform elastic transformation, rotation transformation, scaling transformation, random masking or random cropping on the original images.

[0208] 3.1. Elastic Transformation

[0209] Elastic transformation is a kind of transformation that simulates the continuous and smooth deformation of a physical elastic object under the action of external forces. Its core feature is to achieve non-linear deformation by applying a random displacement field to the image pixels.

[0210] Specifically, for any pixel point in the original image, the displacement field specifies the displacement of this pixel point relative to its original position after transformation. On this basis, the server can generate a random displacement field. Then, for each original image, the server can perform elastic transformation on each pixel point in the original image based on the random displacement field to obtain a noise image after elastic transformation.

[0211] Exemplarily, the server can perform elastic transformation on the original image with reference to the following formula 19 and formula 20.

[0212] x' = x + u x (x, y) · σ (Formula 19)

[0213] y' = x + u y (x, y) · σ (Formula 20)

[0214] Among them, (x, y) represents the coordinates of the pixel point in the original image. u x (x, y) represents the displacement amount of the pixel point (x, y) along the x-axis direction, and u y (x, y) represents the displacement amount of the pixel point (x, y) along the y-axis direction. σ represents the elastic transformation control factor. (x', y') represents the new coordinates of the pixel point (x, y) in the noise image after elastic transformation.

[0215] 3.2. Rotation Transformation

[0216] Rotation transformation is a common operation in image processing, which rotates the image around a certain center point (i.e., the rotation center) by a specific angle. Rotation transformation can be used to correct the image orientation or analyze different orientations of objects.

[0217] Specifically, for each original image, the server can rotate each pixel point in the original image based on the rotation center and the rotation angle to obtain a noise image after rotation transformation.

[0218] Exemplarily, the server can perform a rotation transformation on the original image with reference to the following formulas 21 and 22.

[0219] x' = (x - c x ) cosθ + (y - c y ) sinθ + c x (Formula 21)

[0220] y′ = (x - c x ) sinθ + (y - c y ) cosθ + c y (Formula 22)

[0221] Among them, the meaning of (x, y) can be referred to the introduction in the above 3.1. (c x , c y ) represents the coordinates of the rotation center in the original image, θ represents the rotation angle, and (x', y') represents the new coordinates of the pixel point (x, y) in the noise image after the rotation transformation.

[0222] 3.3. Scaling Transformation

[0223] The scaling transformation refers to the operation of changing the image size according to the scaling ratio, which is often used to adjust the image resolution or change the image scale.

[0224] Specifically, for each original image, the server can adjust the position of each pixel point in the original image according to the scaling ratio to obtain the noise image after the scaling transformation.

[0225] Exemplarily, the server can perform a scaling transformation on the original image with reference to the following formulas 23 and 24.

[0226] x' = c x + (x - c x ) · s x (Formula 23)

[0227] y' = c y + (y - c y ) · s y (Formula 24)

[0228] Among them, s x represents the scaling factor in the x-axis direction, and s y represents the scaling factor in the y-axis direction. (x', y') represents the new coordinates of the pixel point (x, y) in the noise image after the scaling transformation. The meaning of (x, y) can be referred to the introduction in the above 3.1, and the meaning of (c x , c y ) can be referred to the introduction in the above 3.2.

[0229] 3.4. Random Mask

[0230] A random mask refers to the operation of randomly selecting one or more rectangular block areas in an image as mask areas and setting all pixel values in these areas to 0 or other specified values.

[0231] Specifically, for each original image in the first image set, the server can randomly select a partial area in the original image as the mask area, and then set the pixel values of all pixel points in the mask area to the mask pixel values (such as black, gray) to obtain the second image set after masking.

[0232] Exemplarily, the server can perform random masking on the original image with reference to the following formula 25.

[0233]

[0234] Among them, Z mask represents the mask area in the original image, C represents the mask pixel value, and I random_masking (x, y) represents the pixel value of the pixel point (x, y) in the noisy image after masking. The meaning of I clean (x, y) can be introduced in 1.1 above.

[0235] When (x, y) ∈ Z mask , that is, when the pixel point (x, y) in the original image is located in the mask area, set the pixel value I random_masking (x, y) to C. In other cases, the pixel value of the pixel point (x, y) remains unchanged.

[0236] 3.5. Random cropping

[0237] Random cropping refers to randomly selecting a sub-region from the original image and cropping it out as a new image. The position and size of this sub-region are usually determined according to random rules.

[0238] Specifically, for each original image, the server can randomly determine a cropping area, and then crop the original image according to the position and size of the cropping area to obtain the cropped noisy image.

[0239] Exemplarily, assume that the upper left coordinate of the cropping area is (x0, y0), the width of the cropping area is w, and the height is h. The coordinate of the pixel point a in the original image A is (x, y). If x0 ≤ x ≤ x0 + w and x0 ≤ x ≤ x0 + h, then the new coordinate of the pixel point a in the cropped noisy image A_noise is (x′, y′). The calculation formula for the new coordinate (x', y') can refer to the following formula 26 and formula 27.

[0240] x' = x - x0 (formula 26)

[0241] y' = y - y0 (Equation 27)

[0242] 4. Visual Blur Noise

[0243] Visual blur refers to the loss of details in an image, unclear edges of the image, or a decrease in the overall contrast of the image. Visual blur can be caused by various factors such as optical reasons, defects in the imaging system, object or camera movement, etc. As Figure 3 shown, visual blur can include defocus blur, frosted blur, motion blur, zoom blur, and Gaussian blur.

[0244] Specifically, as Figure 7 shown, the server can perform defocus blur, frosted blur, motion blur, zoom blur, or Gaussian blur processing on each original image in the first image set to obtain a second image set containing visual blur noise.

[0245] The following will introduce the methods for performing defocus blur, frosted blur, motion blur, zoom blur, or Gaussian blur processing on the original image.

[0246] 4.1 Defocus Blur

[0247] Defocus blur refers to the phenomenon that an object becomes blurred in an image due to inaccurate focusing of the optical system (such as a camera lens).

[0248] Specifically, for each original image, the server can perform a convolution operation on the pixel value of each pixel point in the original image with a defocus blur kernel to obtain a noise image containing defocus blur.

[0249] Exemplarily, the server can perform defocus blur processing on the original image with reference to the following Equation 28.

[0250] I defocus_blur (x, y) = (I clean * K defocus )(x, y) (Equation 28)

[0251] where the meaning of I clean (x, y) can be referred to the introduction in 1.1 above. K defocus represents the defocus blur kernel (which can also be called the Gaussian kernel), * represents the convolution operation, and I defocus_blur (x, y) represents the pixel value of the pixel point (x, y) in the image containing defocus blur.

[0252] 4.2 Frosted Blur

[0253] Frosted blur is caused by an actual frosted glass blocking the camera lens. Frosted blur has a partial, mosaic-like blur effect.

[0254] Specifically, for each original image, the server can perform a convolution operation on the pixel value of each pixel point in the original image with a uniform kernel to obtain a noisy image with frosted blur.

[0255] Exemplarily, the server can perform frosted blur processing on the original image with reference to the following formula 29.

[0256] I forsted_blur (x,y) = (I clean *K uniform )(x,y) (Formula 29)

[0257] Among them, the meaning of I clean (x,y) can refer to the introduction in 1.1 above. I frosted_blur (x,y) represents the pixel value of the pixel point (x,y) in the image with frosted blur. K uniform represents the uniform kernel, and K uniform can be an n×n matrix, and each element value in the matrix is

[0258] 4.3. Motion Blur

[0259] Motion blur is caused by relative motion between the camera and the object being photographed, resulting in a blurred trailing shadow along the direction of motion in the image during exposure. For example, at a slow shutter speed, if the camera or the object being photographed moves, motion blur will occur.

[0260] Specifically, for each original image, the server can perform a convolution operation on the pixel value of each pixel point in the original image with a linear kernel to obtain a noisy image with motion blur.

[0261] Exemplarily, the server can perform motion blur processing on the original image with reference to the following formula 30.

[0262] I motion_blur (x,y) = (I clean *K motion (x,y) (Formula 30)

[0263] Among them, the meaning of I clean (x,y) can refer to the above introduction. I frosted_blur (x,y) represents the pixel value of the pixel point (x,y) in the image with motion blur. K motion represents a directional linear kernel.

[0264] 4.4. Zoom Blur

[0265] Zoom blur refers to the phenomenon that during the shooting process, the focal length of the lens changes, resulting in a blur radiating from the center in the area outside the center of the image.

[0266] Specifically, for each original image, the server can perform a convolution operation on the pixel value of each pixel point in the original image with a zoom blur kernel to obtain a noisy image with zoom blur.

[0267] Exemplarily, the server can perform zoom blur processing on the original image with reference to the following formula 31.

[0268] I zoom_blur (x, y) = (I clean * K zoom )(x, y) (Formula 31)

[0269] Among them, the meaning of I clean (x, y) can be referred to the introduction in 1.1 above. I zoom_blur (x, y) represents the pixel value of the pixel point (x, y) in the image with zoom blur. K zoom represents the zoom blur kernel, and K zoom can be a radial kernel, the value at the center position is the largest, and the value of K zoom gradually decreases from the center to the outside.

[0270] 4.5. Gaussian Blur

[0271] As a very commonly used image processing technology, the core principle of Gaussian blur is to perform a weighted average operation with the help of the Gaussian function to achieve the smoothing and blurring effects of the image.

[0272] Specifically, for each original image, the server can perform a convolution operation on the pixel value of each pixel point in the original image with a Gaussian kernel to obtain a second image set with Gaussian blur.

[0273] Exemplarily, the server can perform Gaussian blur processing on the original image with reference to the following formula 32.

[0274] I gaussian_blur (x, y) = (I clean * K gaussian )(x, y) (Formula 32)

[0275] Among them, the meaning of I clean (x, y) can be referred to the introduction in 1.1 above. I gaussian_blur (x, y) represents the pixel value of the pixel point (x, y) in the image with Gaussian blur. K gaussian represents the Gaussian kernel, and the calculation formula of K gaussian can be referred to the following formula 33.

[0276]

[0277] Among them, σ represents the standard deviation of the Gaussian function, and K gaussian (x, y) represents the Gaussian kernel corresponding to the pixel point (x, y) in the original image.

[0278] 5. Weather interference noise

[0279] Weather interference noise refers to the noise introduced during the image acquisition process due to the influence of natural weather. Such as Figure 3 shown, weather interference can include snow, rain, frost, fog, and spatter.

[0280] Specifically, as Figure 8 shown, the server can add snowflake effects, raindrop effects, frost effects, fog effects, or spatter effects to each original image included in the first image set to obtain a second image set with weather interference noise.

[0281] The following will introduce the methods for simulating snowflake effects, raindrop effects, frost effects, fog effects, or spatter effects.

[0282] 5.1. Snow

[0283] Snow is a form of precipitation that can interfere with the automatic exposure and white balance functions of the camera during the shooting process.

[0284] Specifically, for each original image, the server can determine the snowflake area in the original image, and then add white pixel points within the snowflake area to simulate the snowflake effect, and finally obtain a noise image with the snowflake effect.

[0285] Exemplarily, the server can simulate the snowflake effect in the original image with reference to the following formula 34.

[0286]

[0287] Among them, I clean (x, y) and the meaning of (x, y) can refer to the introduction in 1.1 above. Z snow represents the snowflake area in the original image, 255 represents the white pixel value, α represents the transparency parameter of the white pixel point, and I snow (x, y) represents the pixel value of the pixel point (x, y) in the noise image with the snowflake effect.

[0288] When (x, y) ∈ Z snow , that is, when the pixel point (x, y) is within the snowflake area, I snow(x, y) is equal to α·255 + (1 - α)·I clean (x, y). In other cases, I snow (x, y) is equal to I clean (x, y), that is, the pixel value of the pixel point (x, y) in the original image remains unchanged.

[0289] 5.2. Rain

[0290] When shooting in a rainy environment, rainwater is extremely likely to condense into water droplets and adhere to the camera lens, which will cause the captured image to have problems such as blurred vision and light spots.

[0291] Specifically, for each original image, the server can determine the rain area in the original image, then add multiple lines to the pixel points in the rain area to simulate the raindrop effect, and finally obtain a noisy image with the raindrop effect.

[0292] Exemplarily, the server can simulate the raindrop effect in the original image with reference to the following formula 35.

[0293]

[0294] Among them, I clean The meanings of (x, y) and (x, y) can refer to the introduction in 1.1 above. Z rain represents the rain area in the original image, α represents the transparency parameter of the line, and I rain (x, y) represents the pixel value of the pixel point (x, y) in the noisy image with the raindrop effect.

[0295] When (x, y) ∈ Z rain That is, when the pixel point (x, y) is in the rain area, I snow (x, y) is α·255 + (1 - α)·I clean (X, y). In other cases, I rain (x, y) is equal to I clean (X, y), that is, the pixel value of the pixel point (X, y) in the original image remains unchanged.

[0296] 5.3. Frost

[0297] Frost is an ice crystal deposit formed by the direct sublimation of water vapor when it rapidly cools on a cold surface at low temperature. In an image, the irregular shape and uneven transparency of frost will cause the shape and edges of objects in the figure to be distorted.

[0298] Specifically, for each original image, the server can add Gaussian blur and random noise to each pixel point in the original image to simulate the frost effect and obtain a noisy image with the frost effect.

[0299] Exemplarily, the server can simulate the effect of frost in the original image with reference to the following formula 36.

[0300] I frost (x,y) = αI gaussian_blur (X,y) + (1 - α)I random_noise (x,y) (Formula 36)

[0301] Wherein, I random_noise (x,y) represents the pixel value of the pixel point (x,y) in the image with random noise, I gaussian_blur (x,y) represents the pixel value of the pixel point (x,y) in the image with Gaussian blur, α represents the transparency parameter, I frost (x,y) represents the pixel value of the pixel point (x,y) in the noisy image with the effect of frost.

[0302] 5.4. Fog

[0303] Fog is formed by a large number of tiny water droplets suspended in the air, which can significantly reduce the visibility of the environment. In an image, fog will weaken the contrast of the image and blur the details, resulting in the edges of objects in the image becoming unclear.

[0304] Specifically, for each original image, the server can add Gaussian blur to each pixel point in the original image and adjust the contrast of each pixel point to simulate the effect of fog and obtain a noisy image with the effect of fog.

[0305] Exemplarily, the server can simulate the effect of fog in the original image with reference to the following formula 37.

[0306] I fog (x,y) = αI gaussian_blur (x,y) + (1 - α)I contrast (x,y) (Formula 37)

[0307] Wherein, I contrast (x,y) represents the pixel value of the pixel point (x,y) in the image after adjusting the contrast. I gussian_blur (x,y) represents the pixel value of the pixel point (x,y) in the image with Gaussian blur. α represents the transparency parameter. I fog (x,y) represents the pixel value of the pixel point (x,y) in the noisy image with the effect of fog.

[0308] 5.5. Splashing

[0309] Splashing is usually composed of mud spots or other small particulate matter. When these substances adhere to the surface of the lens, they will form stains that block the line of sight, resulting in local areas of the image becoming opaque or blurred.

[0310] Specifically, for each original image, the server can determine the splashing area in the original image, and then add randomly shaped black and / or brown spots to the pixel points in the splashing area to simulate the splashing effect, thereby obtaining a noisy image with a splashing effect.

[0311] Exemplarily, the server can simulate the splashing effect in the original image with reference to the following formula 38.

[0312]

[0313] Among them, the meaning of I clean (x, y) can refer to the introduction in 1.1 above. Z splash represents the splashing area, α represents the transparency of the spots, and I spatter (x, y) represents the pixel value of the pixel point (x, y) in the noisy image with a splashing effect.

[0314] When (x, y) ∈ Z splash , that is, when the pixel point (x, y) is located in the splashing area, I spatter (x, y) is α·255 + (1 - α)·I clean (x, y). In other cases, I spatter (x, y) is equal to I clean (x, y), that is, the pixel value of the pixel point (x, y) in the original image is not changed.

[0315] Based on the above technical solution, different natural noises can be added to the first image set by different methods to simulate the interference of different natural noises in the real scene. This method of simulating natural noises can provide high-quality and diverse data support for evaluating the robustness of the deep neural network model in the natural noise environment, and can also improve the actual application effect of the deep neural network model in the complex natural environment.

[0316] Figure 9 The following is a schematic structural diagram of a model robustness evaluation device provided by an embodiment of the present application. As Figure 9 shown, the device includes:

[0317] A first processing unit 901, configured to add natural noises to each original image included in the first image set to obtain a second image set including multiple noisy images.

[0318] A second processing unit 902, configured to input the second image set into the deep neural network model to obtain the predicted category of each noisy image.

[0319] A determination unit 903, configured to determine the robustness of a deep neural network model in an image classification task under natural noise based on a first image set, a second image set, the category of each original image, and the predicted category of each noisy image.

[0320] Optionally, the determination unit 903 is further configured to, for each noisy image, determine the distortion rate of the noisy image based on the noisy image and the original image corresponding to the noisy image.

[0321] Optionally, the determination unit 903 is further configured to determine the classification accuracy of the deep neural network model based on the predicted category of each noisy image and the category of each original image.

[0322] Optionally, the determination unit 903 is further configured to determine the robustness of the deep neural network model in the image classification task under natural noise based on the classification accuracy and the distortion rate corresponding to each noisy image.

[0323] Optionally, the determination unit 903 is further configured to determine the distortion rate of the second image set based on the distortion rate of each noisy image.

[0324] Optionally, the determination unit 903 is further configured to determine the classification accuracy of the deep neural network model at the distortion rate of the second image set based on the classification accuracy and the distortion rate of the second image set.

[0325] Optionally, the determination unit 903 is further configured to determine the robustness of the deep neural network model in the image classification task under natural noise based on the classification accuracy of the deep neural network model at the distortion rate of the second image set.

[0326] Optionally, the determination unit 903 is further configured to, for each second image set, determine the distortion rate of the second image set based on the distortion rate corresponding to each noisy image in the second image set.

[0327] Optionally, the determination unit 903 is further configured to determine the robustness of the deep neural network model in the image classification task under natural noise based on the number of second image sets, the distortion rate of each second image set, and the classification accuracy of the deep neural network model for each second image set.

[0328] Optionally, the determination unit 903 is further configured to determine the robustness of the deep neural network model in the image classification task under different natural noises based on the distortion rate of each second image set and the classification accuracy of the deep neural network model for each second image set.

[0329] Figure 10Fig. shows another possible schematic structural diagram of the model robustness evaluation device involved in the above embodiments. The model robustness evaluation device includes: a processor 1001 and a communication interface 1002. The processor 1001 is used to control and manage the operations of the model robustness evaluation device, and the communication interface 1002 is used to support the communication between the model robustness evaluation device and other network entities. The model robustness evaluation device may further include a memory 1003 and a bus 1004. The memory 1003 is used to store the program codes and data of the model robustness evaluation device.

[0330] Among them, the memory 1003 may be a memory in the model robustness evaluation device, etc. The memory may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid state drive; the memory may further include a combination of the above types of memory.

[0331] The above-mentioned processor 1001 may be to implement or execute various exemplary logic blocks, modules and circuits described in connection with the disclosure of the present application. The processor may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in connection with the disclosure of the present application. The processor may also be a combination that implements computing functions, such as a combination including one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0332] The bus 1004 may be an extended industry standard architecture (EISA) bus, etc. The bus 1004 may be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, Figure 10 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0333] Through the description of the above embodiments, those skilled in the art can clearly understand that for the convenience and simplicity of description, only the above division of each functional module is used as an example. In actual applications, the above functions may be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working processes of the systems, devices and units described above may refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0334] An embodiment of the present application provides a computer program product containing instructions. When the computer program product runs on a computer, the computer is caused to execute the model robustness evaluation method in the above method embodiment.

[0335] An embodiment of the present application further provides a computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions run on a computer, the computer is caused to execute the model robustness evaluation method in the method flow shown in the above method embodiment.

[0336] Among them, a computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a register, a hard disk, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above, or any other form of computer-readable storage medium well-known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an application specific integrated circuit (ASIC). In an embodiment of the present application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0337] An embodiment of the present invention provides a computer program product containing instructions. When the instructions run on a computer, the computer is caused to execute the model robustness evaluation method described in the embodiment of the present application.

[0338] Since the model robustness evaluation device, computer-readable storage medium, and computer program product in the embodiment of the present invention can be applied to the above method, the technical effects that can be obtained can also refer to the above method embodiment, and are not described in detail in the embodiment of the present invention here.

[0339] In several embodiments provided by the present application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of devices or units can be in electrical, mechanical, or other forms.

[0340] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0341] In addition, in each embodiment of the present application, the functional units can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0342] The above is only the specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for evaluating model robustness, characterized in that, The method includes: Adding natural noise to each original image included in the first image set to obtain a second image set including a plurality of noisy images; each of the noisy images corresponds to an original image; Inputting the second image set into a deep neural network model to obtain the predicted category of each of the noisy images; Determining the robustness of the deep neural network model in an image classification task under the natural noise based on the first image set, the second image set, the category of each original image, and the predicted category of each noisy image.

2. The method according to claim 1, wherein The natural noise includes at least one of the following: Visual attribute noise; the visual attribute noise refers to the noise caused by abnormal changes in illumination, color, or contrast; Visual noise; the visual noise refers to the noise that appears during image acquisition, transmission, or processing; Visual deformation noise; the visual deformation refers to the noise caused by the deformation of the shape or structure of an object in the image; Visual blur; the visual blur refers to the noise caused by the loss of details in the image or the unclear edges of the image; Weather interference noise; the weather interference noise refers to the noise caused during image acquisition due to the influence of natural weather.

3. The method according to claim 1 or 2, characterized in that, The natural noise included in each of the noisy images is the same; The determining the robustness of the deep neural network model in the image classification task under the natural noise based on the first image set, the second image set, the category of each original image, and the predicted category of each noisy image includes: For each of the noisy images, determining the distortion rate of the noisy image based on the noisy image and the original image corresponding to the noisy image; the distortion rate of the noisy image is used to characterize the degree of difference between the noisy image and the original image corresponding to the noisy image under the natural noise; Determining the classification accuracy of the deep neural network model based on the predicted category of each noisy image and the category of each original image; Determining the robustness of the deep neural network model in the image classification task under the natural noise based on the classification accuracy and the distortion rate of each noisy image.

4. The method according to claim 3, characterized in that, The determining the robustness of the deep neural network model in the image classification task under the natural noise based on the classification accuracy and the distortion rate of each noisy image includes: Determining the distortion rate of the second image set based on the distortion rate of each noisy image; Determining the classification accuracy of the deep neural network model at the distortion rate of the second image set based on the classification accuracy and the distortion rate of the second image set; Determining the robustness of the deep neural network model in the image classification task under the natural noise based on the classification accuracy of the deep neural network model at the distortion rate of the second image set.

5. The method according to claim 4, wherein The determining the classification accuracy of the deep neural network model at the distortion rate of the second image set based on the distortion rate and the classification accuracy of the second image set includes: Take the product of the classification accuracy rate and the distortion rate of the second image set as the classification accuracy rate of the deep neural network model at the distortion rate of the second image set.

6. The method according to claim 1 or 2, characterized in that, There are multiple second image sets, and the natural noise contained in each second image set is the same. The perturbation levels of the natural noise contained in each second image set are different, and the perturbation level is used to characterize the interference degree of the natural noise on the quality of the original image. Determining the robustness of the deep neural network model in the image classification task under the natural noise based on the first image set, the second image set, the category of each original image, and the predicted category of each noise image includes: For each second image set, determine the distortion rate of the second image set based on the distortion rate of each noise image in the second image set; the distortion rate of each noise image is used to characterize the difference degree between the noise image and the original image corresponding to the noise image under the natural noise. For each second image set, determine the classification accuracy rate of the deep neural network model for the second image set based on the predicted category of each noise image and the category of each original image. Determine the robustness of the deep neural network model in the image classification task under the natural noise based on the number of the second image sets, the distortion rate of each second image set, and the classification accuracy rate of the deep neural network model for each second image set.

7. The method according to claim 1 or 2, characterized in that, There are multiple second image sets, and the natural noise contained in each second image set is different. Determining the robustness of the deep neural network model in the image classification task under the natural noise based on the first image set, the second image set, the category of each original image, and the predicted category of each noise image includes: For each second image set, determine the distortion rate of the second image set based on the distortion rate of each noise image in the second image set; the distortion rate of each noise image is used to characterize the difference degree between the noise image and the original image corresponding to the noise image under the natural noise. For each second image set, determine the classification accuracy rate of the deep neural network model for the second image set based on the predicted category of each noise image and the category of each original image. Determine the robustness of the deep neural network model in the image classification task under the natural noise based on the distortion rate of each second image set and the classification accuracy rate of the deep neural network model for each second image set.

8. A model robustness evaluation device, characterized in that The device includes: A first processing unit, configured to add natural noise to each original image included in the first image set to obtain a second image set including a plurality of noise images; each noise image corresponds to one original image. A second processing unit, configured to input the second image set into a deep neural network model to obtain the predicted category of each noise image. A determination unit, configured to determine the robustness of the deep neural network model in an image classification task under the natural noise based on the first image set, the second image set, the category of each original image, and the predicted category of each noise image.

9. A model robustness evaluation device, characterized in that, Comprising: A processor and a communication interface; the communication interface is coupled to the processor, and the processor is configured to run a computer program or instructions to implement the model robustness evaluation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Instructions are stored in the computer-readable storage medium, and when the computer executes the instructions, the computer executes the model robustness evaluation method according to any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes computer instructions, and when the computer instructions run on a computer, the computer is caused to execute the model robustness evaluation method according to any one of claims 1-7.

Images