Robustness testing method and apparatus

Abstract

The present application discloses a robustness testing method and apparatus, used for improving the reliability of robustness testing. The method comprises: acquiring an adversarial sample, wherein the adversarial sample is obtained by adding N types of disturbance to an original sample, N is a positive integer greater than 1, the original sample is perception data or planning and control data, and the N types of disturbance comprise one type of key disturbance and N-1 types of non-key disturbance; and using the adversarial sample to test the robustness of a processing model, wherein the processing model is used for identifying the sample. According to the solution, the robustness of the processing model is tested by constructing the adversarial sample having multi-dimensional disturbance, so that the performance of the processing model in a real environment can be simulated, being more in line with the actual use situation of the processing model, thus improving the reliability of robustness testing.

Description

Robustness testing method and device

[0001] Cross-reference to related applications

[0002] The present application claims priority to the Chinese patent application No. 202410524514.0, filed on April 28, 2024, and entitled "Robustness testing method and device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of computer, and in particular, to a robustness testing method and device. BACKGROUND

[0004] Robustness of a model refers to the ability of the model to adapt to abnormal data. Since it is impossible to traverse all data when designing and training the model, the prediction result for a part of data may fluctuate. The smaller the fluctuation of the prediction result, the stronger the robustness of the model.

[0005] Safety is one of the most important factors for a driving system. Since a vehicle may face various weather and environmental conditions during driving, it is necessary to evaluate the robustness of a perception model in the driving system. Only the model that meets certain robustness requirements can be used in the driving system. For example, when it is raining, the smaller the influence of the rainfall on the perception result, the stronger the robustness of the perception model, and the more suitable the perception model is for the automatic driving system.

[0006] The prior art tests the robustness of a perception model by constructing single-dimensional perturbation (such as light intensity or rainfall) adversarial samples. This method is difficult to simulate real-world conditions, resulting in unreasonable evaluation of the robustness of the model, and little value in actual use.

[0007] How to more reasonably test the robustness of a model is a technical problem to be solved. SUMMARY

[0008] Embodiments of the present application provide a robustness testing method and device for improving the reliability of robustness testing.

[0009] In a first aspect, a robustness testing method is provided, comprising: obtaining an adversarial sample, the adversarial sample being obtained by adding N kinds of perturbations to an original sample, N being a positive integer greater than 1, the original sample being perception data or regulation data, the N kinds of perturbations including one kind of key perturbation and N-1 kinds of non-key perturbations; and testing the robustness of a processing model using the adversarial sample, the processing model being used for identifying the sample.

[0010] The scheme can test the robustness of the processing model by constructing adversarial samples with multi-dimensional perturbations, can simulate the performance of the processing model in a real environment, is more in line with the actual use of the processing model, and therefore can improve the reliability of the robustness test.

[0011] In a possible design, the frequency at which the key perturbation affects the use scenario of the processing model exceeds a first threshold, and the frequency at which the non-key perturbation affects the use scenario of the processing model does not exceed a second threshold.

[0012] This design can further improve the reliability of the robustness test by distinguishing the key perturbation and the non-key perturbation based on the frequency of the influence of the perturbation on the use scenario of the processing model.

[0013] In a possible design, the key perturbation is a perturbation that causes the accuracy of the processing model in the use scenario to decrease by more than a third threshold, and the non-key perturbation is a perturbation that causes the accuracy of the processing model in the use scenario to decrease by not more than a fourth threshold.

[0014] This design can further improve the reliability of the robustness test by distinguishing the key perturbation and the non-key perturbation based on the degree of the influence of the perturbation on the use scenario of the processing model.

[0015] In a possible design, L sets of adversarial samples can be obtained, L is a positive integer greater than 1, the L sets of adversarial samples correspond to L non-key perturbation parameters in a one-to-one manner, the adversarial samples in each set of adversarial samples in the L sets of adversarial samples are added with the non-key perturbation parameter corresponding to each set of adversarial samples, the non-key perturbation parameter is a parameter conforming to N-1 non-key perturbations, and at least two adversarial samples in each set of adversarial samples in the L sets of adversarial samples correspond to the same original sample and are added with different key perturbation parameters, wherein the key perturbation parameter is a parameter conforming to a key perturbation. Correspondingly, the method for testing the robustness of the processing model using the adversarial samples can include: calculating the safety radius of the processing model under the key perturbation using each set of adversarial samples in the L sets of adversarial samples, to obtain L safety radii corresponding to the L non-key perturbation parameters; and determining the robustness of the processing model under the key perturbation according to the L safety radii.

[0016] This design provides a specific scheme for testing the robustness of the processing model using the adversarial samples, can simulate the robustness performance of the processing model under the key perturbation in a real environment, and has strong rationality of the test scheme. Moreover, the scheme has low complexity and is easy to implement.

[0017] In a possible design, each of the L sets of adversarial samples includes M groups of adversarial samples, the M groups of adversarial samples correspond to the M original samples in a one-to-one manner, and each adversarial sample in each group of adversarial samples is obtained by adding a different key perturbation parameter to the original sample corresponding to the group of adversarial samples, where M is a positive integer. Correspondingly, calculating the security radius of the processing model under the key perturbation using each of the L sets of adversarial samples can include: calculating the security radius of the processing model under the key perturbation using each group of adversarial samples in each set of adversarial samples, to obtain M security radii; and determining the security radius corresponding to each set of adversarial samples based on the M security radii.

[0018] For example, M = 1, and the M security radii are the security radii corresponding to each set of adversarial samples; or for example, M > 1, and the expected value of the M security radii is the security radius corresponding to each set of adversarial samples.

[0019] This design provides a scheme for obtaining a security radius based on one or a group of adversarial samples, and has low complexity and is easy to implement.

[0020] In a possible design, the original samples corresponding to each of the L sets of adversarial samples are the same.

[0021] This design generates adversarial samples under different perturbations based on the same original sample, which can further improve the reliability of the test and further reduce the complexity of implementation.

[0022] In a possible design, determining the robustness of the processing model under the key perturbation according to the L security radii can include: mapping the L security radii to an N-dimensional coordinate system according to the L non-key perturbation parameters; wherein a first dimension in the N-dimensional coordinate system corresponds to the security radius, and N-1 dimensions other than the first dimension correspond to N-1 non-key perturbations in a one-to-one manner; calculating the volume of an N-dimensional space formed by the L security radii in the N-dimensional coordinate system; and taking the volume of the N-dimensional space as the robustness of the processing model under the key perturbation.

[0023] This design maps the security radii obtained under different non-key perturbations to a multi-dimensional space, and takes the volume of the multi-dimensional space formed by these security radii in the multi-dimensional coordinate system as a measurement parameter of the robustness size, which is simple to calculate and easy to implement.

[0024] In a possible design, the processing model can be applied to a driving system; and the original sample can be one or more of an image, a point cloud, an audio, a perception result, and a driving trajectory. Of course, the above are only examples, and the actual implementation is not limited thereto.

[0025] In a possible design, the N perturbations can include one or more of the following: light, rain, fog, snow, noise, temperature, humidity, and air conditions. Of course, the above are only examples, and the actual implementation is not limited thereto.

[0026] In a second aspect, a processing apparatus is provided, which comprises a module or unit or means for performing the method according to the first aspect or any possible design of the first aspect.

[0027] In an example, the apparatus can include:

[0028] The obtaining module is configured to obtain an adversarial sample, the adversarial sample being obtained by adding N kinds of perturbations to an original sample, N being a positive integer greater than 1, the original sample being perception data or regulation data, and the N kinds of perturbations including one kind of key perturbation and N-1 kinds of non-key perturbations.

[0029] The testing module is configured to test the robustness of the processing model using the adversarial sample, the processing model being configured to recognize the sample.

[0030] In a third aspect, a processing apparatus is provided, which comprises a processor configured to execute computer-executable instructions stored in a memory to cause the processing apparatus to perform the method according to the first aspect or any possible design of the first aspect.

[0031] Optionally, the memory is located in the processing apparatus.

[0032] Optionally, the memory is located outside the processing apparatus.

[0033] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program or instructions, and when the computer program or instructions are executed by a communication apparatus, the method according to the first aspect or any possible design of the first aspect is implemented.

[0034] In a fifth aspect, a computer program product is provided, which stores instructions, and when the instructions are run on a computer, the computer is caused to perform the method according to the first aspect or any possible design of the first aspect.

[0035] The specific designs and beneficial effects of the second aspect to the fifth aspect are the same as those of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0036] FIG. 1 is a schematic diagram of a model robustness evaluation method;

[0037] FIG. 2 is a schematic diagram of another model robustness evaluation method;

[0038] FIG. 3 is a flowchart of a robustness testing method provided by an embodiment of the present application;

[0039] FIG. 4 is a schematic diagram of an adversarial sample set;

[0040] FIG. 5A and FIG. 5B are schematic diagrams of calculating the robustness size;

[0041] Figure 6 is a schematic diagram of a processing device provided in an embodiment of this application;

[0042] Figure 7 is a schematic diagram of another processing device provided in an embodiment of this application. Detailed Implementation

[0043] Let's first introduce some of the terms used in the following text.

[0044] 1) Processing model.

[0045] The processing model can also be called a model, network model, deep learning model, etc.

[0046] In the embodiments of this application, the processing model includes, but is not limited to, a perception module and a planning and control model. The perception model is used to identify the perceived data to achieve the perception function; the planning and control model is used to identify the planning and control data to achieve the planning and control functions.

[0047] For example, the operation process of an automated driving system (ADS) typically includes several parts such as localization, perception, decision-making and planning, and control.

[0048] Perception, in this context, refers to the process by which an autonomous driving system perceives environmental information based on data collected by sensors such as cameras and lidar, acquiring the positions and attributes of dynamic and static targets around the vehicle. A perception model is the algorithm used by the autonomous driving system to achieve perception. For example, in the field of camera-related visual perception, methods for achieving visual perception fall into two main categories: traditional methods and deep learning methods. Traditional methods extract image features and perform target recognition through designed feature extraction algorithms; deep learning methods first design a neural network, train the neural network using a large amount of labeled data, and then derive a neural network model that can be used for target recognition. Algorithms designed or trained using either of these methods are considered perception models.

[0049] In this context, planning and control refers to the autonomous driving system's planning and control of the vehicle's driving state based on environmental information (such as environmental information perceived by sensors like cameras and LiDAR) and vehicle driving information. This includes planning the vehicle's path and outputting control commands to control the vehicle's driving state. The planning and control model is the algorithm used by the autonomous driving system to achieve planning and control.

[0050] 2) Robustness.

[0051] Robustness refers to the ability of a model to adapt to abnormal data. Since it is impossible to traverse all data when designing and training a model, the detection result for a part of data may fluctuate, and the smaller the fluctuation, the stronger the robustness. For example, when a vehicle is performing automatic driving in a rainy weather, the smaller the rain variation affects the recognition effect of the model, the stronger the robustness of the model, that is, even if the current rain changes greatly, the recognition result of the model can remain unchanged, that is, the model can be applied to automatic driving in the current rainy environment. However, if the rain variation is small, the recognition result of the model changes, which indicates that the robustness of the environment recognition model is poor, and the current rain changes very slightly, which may cause the recognition result of the model to be wrong. In this case, if the model is continued to be used, the vehicle automatic driving process may be wrong due to the inaccurate recognition result of the model, and the safety of vehicle traffic cannot be guaranteed. Therefore, in the field of automatic driving, it is very important to evaluate the robustness of the model to maintain the safety of vehicle traffic.

[0052] 3) Adversarial examples, original examples.

[0053] Adversarial examples refer to samples formed by deliberately adding interference in the data set, which cause the processing model to output an incorrect label with high confidence. The original sample, also known as a "non-adversarial sample", can be understood as a sample without added interference. In some implementations, adversarial examples can be obtained by adding additional information or changing part of the information based on the original sample.

[0054] 4) Perturbation.

[0055] When obtaining adversarial examples based on original samples, the part of information added or the part of information changed on the original sample is the perturbation. In some embodiments, "perturbation" can also be referred to as "interference" or "noise" or "noise perturbation" or "noise interference", etc.

[0056] The above introduces some terms involved in the embodiments of the present application. The following introduces some technical features involved in the embodiments of the present application.

[0057] FIG. 1 illustrates a model robustness evaluation method, as shown in FIG. 1, which mainly relies on a set of specified noise to evaluate the robustness of the model, and the specific implementation process includes:

[0058] Firstly, a number of standard images (i.e. original samples) with very small or even negligible noise are obtained; then, a set of existing specified noises are used to perturb the standard images, respectively obtaining attack images (i.e. adversarial samples) of the standard images perturbed by each noise, wherein the specified noise set can include but is not limited to Gaussian noise, Poisson noise, impulse noise, defocus noise, glass noise, motion noise, zoom noise, snow noise, frost noise, fog noise, brightness noise, contrast noise, elasticity noise, pixel noise and image compression noise, etc. as shown in Fig. 1;

[0059] Then, according to the recognition effect of the model on the attack images, the detection accuracy of the model under the influence of each noise type is determined, the average value of the detection accuracy of the model under the influence of each noise type is calculated, the average accuracy is obtained, and the ratio of the average accuracy to the detection accuracy of the model without noise influence is taken as the robustness measurement index. Obviously, the closer the robustness measurement index is to 1.0, the less obvious the influence of noise on the detection result of the model, and the better the robustness of the model.

[0060] According to the above content, it can be known that the evaluation method shown in Fig. 1 actually tests the robustness of the model under single-dimensional noise disturbance, and then comprehensively measures the robustness corresponding to each dimension of noise. This method is difficult to simulate the noise disturbance in the real environment, and has little value in actual use, because in actual use, the noise disturbance in the environment is multidimensional. For example, in the driving scene, the user is concerned about the scenes of raining in the daytime, in the late afternoon and in the late night, rather than the influence of rain amount on the model.

[0061] Fig. 2 exemplarily shows another model robustness evaluation method, as shown in Fig. 2, this evaluation method does not need to specify a noise set, but uses a neural network to find the maximum distance that makes the identification result of the model not wrong as the maximum safety radius of the model. The specific implementation process is as follows: first, a specified model and a standard image (i.e. an original sample) are obtained, the standard image is identified using the model to obtain the label of the standard image (such as "label 1" in Fig. 2), then the standard image is continuously disturbed according to the order of noise from small to large, and the label of the attack image (i.e. the adversarial sample) after adding the disturbance is identified using the model; as the noise increases, the identified label will gradually deviate from the label of the standard image, until a certain noise disturbance is added to make the model identify the attack image as the remaining label (such as "label 2" or "label 3" in Fig. 2), that is, the decision boundary of the model under the remaining label is found; the minimum value of the noise disturbance amplitude corresponding to the decision boundary of each other label is taken as the maximum safety radius of the model (as shown in Fig. 2, the disturbance amplitude "R2" corresponding to "label 2" is obviously smaller than the disturbance amplitude "R3" corresponding to "label 3", so the maximum safety radius is R2), that is, as long as the noise disturbance added to the standard image is within the maximum safety radius, the model can identify the same identification result as the standard image.

[0062] According to the above content, the evaluation method shown in Fig. 2 actually provides a model robustness authentication means, that is, the robustness of the model is judged by calculating the distance from the standard image to the attack image with the nearest different label, in other words, by finding a minimum noise that distorts the standard image on the basis of the standard image. However, the calculation process of the neural network is very complex, and the calculation amount is relatively large. Solving the minimum distortion on the neural network with an activation function is a non-deterministic polynomial (NP) complete problem with a non-linear complexity, which is very tricky in calculation. At present, although the minimum distortion can be obtained on some small and shallow simple neural networks, when it is extended to medium or large neural networks, the minimum distortion cannot be obtained by direct solving, which leads to the fact that this evaluation method may not be able to analyze the maximum safety radius. In addition, this evaluation method also uses a specified environment image to measure the robustness of the model under single-dimensional noise disturbance, and then measures the robustness corresponding to each dimension of noise, which cannot solve the technical problems existing in the evaluation method of Fig. 1.

[0063] In view of this, the technical solutions of the embodiments of the present application are provided. It can be understood that the embodiments of the present application can be applied to any scenario that needs to evaluate the robustness of a model. For example, a processing model is tested for robustness by a third-party evaluation agency (such as the Ministry of Industry and Information Technology, an automobile research department, etc.) before being deployed to a vehicle, and the processing model can be deployed to the vehicle and put into the market after passing the robustness test by the third-party evaluation agency. Alternatively, the robustness of a model is tested internally by a vehicle manufacturer to improve the safety of the vehicle and the passing rate of the processing model in the third-party evaluation agency.

[0064] The object of the robustness test in the embodiments of the present application can be any processing model, for example, including but not limited to a perception model and a control model applied in an automatic driving system.

[0065] Referring to FIG. 3, a flowchart of a robustness test method provided by the embodiments of the present application is shown. The method can be applied to any electronic device with computing capability, such as a personal computer, a server, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car, or other types of cellular phones, media consumption devices, wearable devices, etc. The method includes S301-S302:

[0066] S301, obtaining an adversarial sample.

[0067] The adversarial sample refers to a sample with added perturbations, and the original sample refers to a sample without added perturbations. In the embodiments of the present application, the adversarial sample can be obtained by adding N kinds of perturbations to the original sample, where N is a positive integer greater than 1. It can be understood that the number of adversarial samples can be multiple, and each of the multiple adversarial samples is added with N kinds of perturbations.

[0068] In the embodiments of the present application, the data types of the original sample and the adversarial sample include but are not limited to one or more of an image, a point cloud, an audio, a perception result, a driving trajectory, etc. For example, when the processing model is a perception model, the input sample can be one or more of an image, a point cloud, an audio, etc. For example, when the processing model is a control model, the input sample can be a perception result or a driving trajectory, etc., where the perception result can be the output result of the perception model.

[0069] The N perturbations include, but are not limited to, one or more of the following: light, weather, noise, temperature, humidity, air conditions, and the like. Optionally, the light can further include one or more of the following: intensity of the light, direction of the light, type of the light (such as natural light, light from a lamp), color temperature, and the like. Optionally, the weather can further include one or more of the following: rain, snow, fog, wind (including wind direction, wind strength, and the like), and the like. Optionally, the noise can further include one or more of the following: direction of the noise, type of the noise (such as noise from a vehicle, noise from a person, and the like), frequency of the noise, intensity of the noise, and the like. Optionally, the air conditions include, for example, concentration of fine particulate matter (such as PM2.5 index), and the like.

[0070] It can be understood that adding a perturbation on an original sample means adding a parameter corresponding to the perturbation in the original sample. For example, for an original sample of the data type of an image, adding rain on the original sample means adding an image parameter corresponding to rain in the image, so that the generated adversarial sample is an image in a rainy day scene.

[0071] For different processing models, the data type of the sample (including the original sample and the adversarial sample) input into the processing model can be different, and the perturbation in the adversarial sample can also be different. It can be understood that the processing model is used to identify the sample (including the original sample and the adversarial sample).

[0072] Taking a perception model as an example:

[0073] Example 1: When the perception model is applied to a vehicle to identify an image collected by a vehicle-mounted camera, the data type of the original sample can be an image, and the added perturbation can be one or more of light, rain, fog, snow, and the like.

[0074] Example 2: When the perception model is applied to a vehicle to identify a point cloud collected by a vehicle-mounted radar, the data type of the original sample can be a point cloud, and the perturbation can be reflection, refraction, scattering, and absorption of electromagnetic waves by non-target objects, and electronic waves emitted by background radiation sources.

[0075] Example 3: When the perception model is applied to a vehicle to identify sound data collected by a vehicle-mounted microphone, the data type of the original sample can be audio, and the added perturbation of the adversarial sample can be noise from a vehicle, noise from a person, and the like.

[0076] Of course, the above several examples are only examples, and the actual application is not limited thereto.

[0077] In the embodiments of the present application, the N perturbations added in the adversarial sample can be divided into key perturbations and non-key perturbations. The key perturbations and the non-key perturbations can be selected according to test requirements.

[0078] In a possible implementation, the critical disturbance can be a disturbance whose frequency of affecting (or interfering with) the use scenario of the processing model exceeds a first threshold, and the non-critical disturbance can be a disturbance whose frequency of affecting (or interfering with) the use scenario of the processing model does not exceed a second threshold. The first threshold and the second threshold can be the same or different. Optionally, the first threshold is greater than or equal to the second threshold.

[0079] For example, in an automatic driving scenario, for an image recognition type processing model, the influence of light on automatic driving is long-term or continuous, and the frequency of occurrence is high, so light can be a critical disturbance. The influence of weather conditions (such as rain, snow, fog, etc.) on driving is short-term or occasional, and the frequency of occurrence is low, so weather conditions (such as rain, snow, fog, etc.) can be a non-critical disturbance.

[0080] In another possible implementation, the critical disturbance can be a disturbance that causes the accuracy of the processing model in the use scenario to decrease by more than a third threshold, and the non-critical disturbance can be a disturbance that causes the accuracy of the processing model in the use scenario to decrease by not more than a fourth threshold. The third threshold and the fourth threshold can be the same or different. Optionally, the third threshold is greater than or equal to the fourth threshold.

[0081] For example, in an automatic driving scenario, for an image recognition type processing model, the accuracy decrease amplitude in a fog scenario is high, and the accuracy decrease amplitude in a rain scenario is low, so fog can be a critical disturbance, and rain can be a non-critical disturbance.

[0082] In a possible design, obtaining the adversarial sample includes: obtaining L adversarial sample sets, L being a positive integer greater than 1.

[0083] The L adversarial sample sets correspond to L non-critical disturbance parameters one by one, and the adversarial samples in each adversarial sample set are added with the non-critical disturbance parameter corresponding to the adversarial sample set. The non-critical disturbance parameter is a parameter conforming to N-1 non-critical disturbances. It can be understood that the number of adversarial samples in different adversarial sample sets can be the same or different, and is not limited.

[0084] For example, the N disturbances include light and rain, where the light is a critical disturbance, and the rain is a non-critical disturbance. Each adversarial sample set of the L adversarial sample sets corresponds to a specific rain intensity (or rainfall), and each sample in each adversarial sample set is added with the rain intensity corresponding to the adversarial sample set. The rain intensities corresponding to different adversarial sample sets are different.

[0085] For example, the N perturbations include light, temperature, and humidity, the light is the key perturbation, and the temperature and humidity are non-key perturbations. Each of the L sets of adversarial samples corresponds to a specific temperature and a specific humidity. Each sample in each set of adversarial samples is added with the temperature and humidity corresponding to the set of adversarial samples. The combinations of the temperatures and humidities corresponding to different sets of adversarial samples are different.

[0086] In a specific example, referring to FIG. 4, a set of adversarial samples is shown. Each dot in FIG. 4 (including black dots and white dots) represents an adversarial sample. There are 90 adversarial samples in total, and the 90 adversarial samples correspond to the same (set of) non-key perturbation parameter.

[0087] Further, at least two adversarial samples in each set of adversarial samples correspond to the same original sample and are added with different key perturbation parameters. In other words, at least two adversarial samples in each set of adversarial samples are obtained by adding the same non-key perturbation parameter and different key perturbation parameters to the same original sample.

[0088] In a possible implementation, each of the L sets of adversarial samples includes M groups of adversarial samples, the M groups of adversarial samples correspond to M original samples one by one, and each group of adversarial samples includes P adversarial samples, the P adversarial samples are obtained by adding P different key perturbation parameters to the same original sample respectively. M and P are positive integers.

[0089] For example, referring to FIG. 4, each row in FIG. 4 corresponds to an original sample, and there are 10 original samples in total (i.e., M = 10). The 10 original samples can all be different. Different adversarial samples in each row correspond to different key perturbation parameters. In FIG. 4, 9 key perturbation parameters are taken as an example (i.e., P = 9).

[0090] It can be understood that the number of adversarial samples included in different groups of adversarial samples in the M groups of adversarial samples can be the same or different. In FIG. 4, the number of adversarial samples included in different groups of adversarial samples in the M groups of adversarial samples is taken as an example (i.e., there are 10 adversarial samples in each row, which correspond to 10 intensity key perturbation parameters respectively).

[0091] Optionally, the original samples corresponding to the L sets of adversarial samples are the same. For example, the L sets of adversarial samples are all generated from the 10 original samples corresponding to FIG. 4. In this way, the complexity of implementation can be reduced.

[0092] The following is an example of generating an adversarial sample:

[0093] Step 1: Obtain M original samples, for example, M = 10 in FIG. 4;

[0094] Step 2, add non-critical perturbation parameters: iterate through the L non-critical perturbation parameters, repeat the following actions until all L non-critical perturbation parameters are iterated: add the currently iterated non-critical perturbation parameter to each of the M original samples, and after all L non-critical perturbation parameters are iterated, a total of L sets of adversarial samples (corresponding to L non-critical perturbation parameters respectively) added with non-critical perturbation parameters are obtained, each set of adversarial samples includes M adversarial samples, the samples in the same set of adversarial samples are added with the same non-critical perturbation parameter, and the samples in different sets of adversarial samples are added with different non-critical perturbation parameters;

[0095] Step 3, add critical perturbation parameters: repeat the following actions for each set of adversarial samples obtained in step 2 until all P critical perturbation parameters are iterated: add the currently iterated critical perturbation parameter to each of the M adversarial samples contained in each set of adversarial samples, and each time a critical perturbation parameter is added, M adversarial samples corresponding to the critical perturbation parameter are obtained, a total of P times (for example, P = 9 in FIG. 4), and finally P * M adversarial samples are obtained.

[0096] It can be understood that the above examples are examples of adding non-critical perturbation parameters first and then adding critical perturbation parameters, and in practice, non-critical perturbation parameters can be added first and then critical perturbation parameters, or non-critical perturbation parameters and critical perturbation parameters can be added at the same time, without limitation.

[0097] S302, test the robustness of the processing model using the adversarial samples.

[0098] For example, following the example of the L sets of adversarial samples above, testing the robustness of the processing model using the adversarial samples can include: calculating the safety radius of the processing model under critical perturbation using each set of adversarial samples in the L sets of adversarial samples, obtaining L safety radii corresponding to the L non-critical perturbation parameters; and determining the robustness of the processing model under critical perturbation according to the L safety radii.

[0099] The following describes an implementation scheme for calculating the safety radius of the processing model under critical perturbation using each set of adversarial samples in the L sets of adversarial samples, taking the processing of one set of adversarial samples as an example:

[0100] Calculate the safety radius of the processing model under critical perturbation using each set of adversarial samples in the M sets of adversarial samples contained in the set of adversarial samples, and obtain M safety radii; determine the safety radius corresponding to each set of adversarial samples based on the M safety radii.

[0101] When M = 1, that is, one set of adversarial samples includes only one set of adversarial samples, the safety radius of the processing model under the key perturbation is calculated using the set of adversarial samples, and the safety radius obtained is the safety radius corresponding to the set of adversarial samples.

[0102] When M > 1, that is, one set of adversarial samples includes multiple sets of adversarial samples, the safety radius of the processing model under the key perturbation is calculated using each set of adversarial samples, and M safety radii are obtained; then the statistical value of the M safety radii is calculated, for example, the expected value is calculated, and the expected value is taken as the safety radius corresponding to the set of adversarial samples. For example, the set of adversarial samples shown in FIG. 4 includes 10 sets of adversarial samples (that is, 10 rows of adversarial samples in FIG. 4), each row of adversarial samples can determine a safety radius, and a total of 10 safety radii are determined, and the expected value of the 10 safety radii can be calculated, and the expected value is taken as the safety radius corresponding to the set of adversarial samples.

[0103] It can be understood that the safety radius refers to the maximum distance (the maximum value of the non-key perturbation parameter) that makes the identification result of the processing model correct. The identification result of the processing model being correct means that the identification result obtained by inputting the adversarial sample into the processing model is consistent with the identification result obtained by inputting the original sample corresponding to the adversarial sample into the processing model (or the difference between the identification results does not exceed a threshold value).

[0104] The safety radius of the processing model under the key perturbation is calculated using a set of adversarial samples, for example: the adversarial samples in the set of adversarial samples are input into the processing model for identification in the order of the key perturbation parameter from small to large, and as the key perturbation parameter increases, the identification result gradually deviates from the identification result corresponding to the original sample, until the adversarial sample added to a certain key perturbation parameter makes the processing model identify the adversarial sample incorrectly, and then the key perturbation parameter can be determined as the safety radius of the processing model under the key perturbation.

[0105] For example, as shown in FIG. 4, the white dot represents that the processing model passes the test using the adversarial sample (that is, the processing model outputs the correct identification result), and the black dot represents that the processing model fails the test using the adversarial sample (that is, the processing model outputs the incorrect identification result or does not output the identification result). It can be seen that the safety radius r obtained based on the first row of adversarial samples is 8, the safety radius r obtained based on the second row of adversarial samples is 9, the safety radius r obtained based on the third row of adversarial samples is 6, and so on.

[0106] In some embodiments, the calculation of the safety radius corresponding to the set of adversarial samples can be written as the following formula: R S = E(r s ); r s = ∫I(f(x), f(D(x, s, m)))dm;

[0107] wherein, R S represents the security radius corresponding to the adversarial sample set, r s represents the security radius corresponding to the adversarial sample; E is a request expectation function, D is a scrambling function, D(x, s, m) represents adding non-critical disturbance s and critical disturbance m on the original sample x; I is an exponential function:

[0108] According to the above method, each adversarial sample set in the L adversarial sample sets is processed iteratively, and L security radii corresponding to the L adversarial sample sets are obtained.

[0109] The following describes an implementation scheme for determining the robustness of the processing model under critical disturbance according to the L security radii:

[0110] In one possible implementation, the L security radii corresponding to the L adversarial sample sets can be mapped into a two-dimensional coordinate system according to the L non-critical disturbance parameters corresponding to the L adversarial sample sets; wherein the first dimension in the two-dimensional coordinate system corresponds to the security radius, and the second dimension corresponds to the non-critical disturbance parameter; the area enclosed by the L security radii in the two-dimensional coordinate system is calculated; and the size of the area is taken as the size of the robustness of the processing model under critical disturbance.

[0111] For example, as shown in FIG. 5A, taking L = 11 as an example, the x-axis represents 11 adversarial sample sets (i.e., 11 (or groups) of non-critical disturbance parameters), and the y-axis represents the security radius (such as the expectation value) of the adversarial sample set. The area enclosed by the dashed line and the x and y axes in the figure can be calculated (optionally, the sum of the areas of the various columnar graphs in FIG. 4 can be calculated to approximate the area enclosed by the dashed line and the x and y axes to reduce the calculation complexity), and the size of the robustness is taken as the size. It can be understood that each (group) of non-critical disturbance parameters can represent a specific value of a type of non-critical disturbance parameter, or a specific value of a combination of multiple types of non-critical disturbance parameters, for example, one horizontal coordinate in FIG. 4 represents one rain intensity, or one horizontal coordinate represents a combination of one rain intensity and one snow intensity.

[0112] In another possible implementation, the L security radii can be mapped into an N-dimensional coordinate system according to the L non-critical disturbance parameters; wherein the first dimension in the N-dimensional coordinate system corresponds to the security radius, and the N-1 dimensions other than the first dimension one-to-one correspond to N-1 non-critical disturbance parameters; the volume of the N-dimensional space formed by the L security radii in the N-dimensional coordinate system is calculated; and the volume of the N-dimensional space is taken as the robustness of the processing model under critical disturbance.

[0113] For example, as shown in FIG. 5B, taking L = 9 as an example, 9 vertices in a three-dimensional space correspond to the safety radius of 9 adversarial sample sets respectively, the x-axis represents a non-critical disturbance parameter A (such as rain intensity), the y-axis represents a non-critical disturbance parameter B (such as snow intensity), and the z-axis represents the safety radius. The volume of the polyhedron surrounded by the 9 vertices in the figure can be calculated as the size of the robustness.

[0114] It can be understood that FIG. 5B is taken as an example of three dimensions, and more dimensions can be further extended.

[0115] The above scheme tests the robustness of the processing model by constructing adversarial samples with multi-dimensional disturbances, can simulate the performance of the processing model in the real environment (that is, the environment with multi-dimensional disturbances), and is more consistent with the actual use of the processing model, so as to improve the reliability of the robustness test. Moreover, the scheme divides the disturbance into critical disturbance and non-critical disturbance, and gives a specific robustness test scheme. In actual application, different disturbances can be selected as critical disturbances according to needs to simulate the robustness performance of the processing model under the influence of critical disturbances in the real environment. The scheme has low complexity and is easy to implement.

[0116] The above introduces the method provided by the embodiments of the application, and the following introduces the device provided by the embodiments of the application.

[0117] Based on the same technical concept, the embodiments of the application provide a processing device, which includes a module or unit or means corresponding to the method steps shown in FIG. 3. The functions or units or means can be realized by software, or by hardware, or by executing corresponding software by hardware.

[0118] For example, referring to FIG. 6, the device can include:

[0119] For example, the device can include:

[0120] The acquisition module 601 is configured to acquire an adversarial sample, the adversarial sample being obtained by adding N disturbances to an original sample, N being a positive integer greater than 1, the original sample being perception data or regulation and control data, and the N disturbances including one critical disturbance and N-1 non-critical disturbances.

[0121] The test module 602 is configured to test the robustness of a processing model using the adversarial sample, the processing model being configured to identify a sample.

[0122] In a possible design, the frequency of the critical disturbance affecting the use scenario of the processing model is greater than a first threshold, and the frequency of the non-critical disturbance affecting the use scenario of the processing model is not greater than a second threshold.

[0123] In a possible design, the key perturbation is a perturbation that causes the accuracy of the processing model under the use scenario to decrease by more than a third threshold, and the non-key perturbation is a perturbation that causes the accuracy of the processing model under the use scenario to decrease by no more than a fourth threshold.

[0124] In a possible design, the obtaining module 601 can be configured to: obtain L sets of adversarial samples, L being a positive integer greater than 1, the L sets of adversarial samples corresponding to L non-key perturbation parameters in a one-to-one manner, the adversarial samples in each set of adversarial samples in the L sets of adversarial samples being added with the non-key perturbation parameter corresponding to the set of adversarial samples, the non-key perturbation parameter being a parameter conforming to N-1 non-key perturbations, and at least two adversarial samples in each set of adversarial samples in the L sets of adversarial samples corresponding to the same original sample and being added with different key perturbation parameters, wherein the key perturbation parameter is a parameter conforming to a key perturbation. Correspondingly, the testing module 602 can be configured to: calculate the safety radius of the processing model under the key perturbation by using each set of adversarial samples in the L sets of adversarial samples, to obtain L safety radii corresponding to the L non-key perturbation parameters in a one-to-one manner; and determine the robustness of the processing model under the key perturbation according to the L safety radii.

[0125] In a possible design, each set of adversarial samples in the L sets of adversarial samples includes M groups of adversarial samples, the M groups of adversarial samples corresponding to M original samples in a one-to-one manner, and each adversarial sample in each group of adversarial samples being obtained by adding a different key perturbation parameter to the original sample corresponding to the group of adversarial samples, M being a positive integer. Correspondingly, the testing module 602 can be configured to: calculate the safety radius of the processing model under the key perturbation by using each group of adversarial samples in each set of adversarial samples, to obtain M safety radii; and determine the safety radius corresponding to each set of adversarial samples based on the M safety radii.

[0126] In a possible design, M=1, and the M safety radii are the safety radii corresponding to each set of adversarial samples; or M>1, and an expected value of the M safety radii is the safety radius corresponding to each set of adversarial samples.

[0127] In a possible design, the original samples corresponding to the respective sets of adversarial samples in the L sets of adversarial samples are the same.

[0128] In a possible design, the testing module 602 can be configured to: map the L safety radii to an N-dimensional coordinate system according to the L non-key perturbation parameters; wherein a first dimension in the N-dimensional coordinate system corresponds to the safety radius, and N-1 dimensions other than the first dimension correspond to the N-1 non-key perturbations in a one-to-one manner; calculate the volume of an N-dimensional space formed by the L safety radii in the N-dimensional coordinate system; and take the volume of the N-dimensional space as the robustness of the processing model under the key perturbation.

[0129] In a possible design, the processing model can be applied to a driving system; and the original sample can be one or more of an image, a point cloud, audio, a perception result, and a driving trajectory. Of course, the above are merely examples, and actual implementations are not limited thereto.

[0130] In a possible design, the N kinds of disturbances can include one or more of the following: light, rain, fog, snow, noise, temperature, humidity, and air conditions. Of course, the above are merely examples, and actual implementations are not limited thereto.

[0131] It should be understood that all relevant content of each step involved in the above method embodiments can be cited from the function description of the corresponding function module, which will not be repeated here.

[0132] Based on the same technical concept, referring to FIG. 7, the embodiment of the present application further provides a processing device, comprising:

[0133] at least one processor 701; the at least one processor 701 executes instructions stored in a memory 702, so that the device performs the method steps shown in FIG. 3.

[0134] Optionally, the memory 702 is located outside the device.

[0135] Optionally, the device comprises the memory 702, the memory 702 is connected to the at least one processor 701, and the memory 702 stores instructions executable by the at least one processor 701. In FIG. 7, the memory 702 is optional for the device, which is represented by a dashed box.

[0136] The processor 701 and the memory 702 can be coupled through an interface circuit or integrated together, which is not limited here.

[0137] The specific connection medium between the processor 701 and the memory 702 is not limited in the embodiment of the present application. In FIG. 7, the processor 701 and the memory 702 are connected through a bus 703, and the connection mode between other components is only schematically illustrated, which is not limited. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience, only one thick line is used to represent the bus in FIG. 7, but it does not mean that there is only one bus or only one type of bus.

[0138] It should be understood that the processor mentioned in the embodiment of the present application can be implemented by hardware or software. When implemented by hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented by software, the processor can be a general-purpose processor, which implements by reading software code stored in a memory.

[0139] The processor can be, for example, a Central Processing Unit (CPU), a general-purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like. The general-purpose processor can be a microprocessor, or the processor can be any conventional processor, etc.

[0140] It should be understood that the memory mentioned in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a Random Access Memory (RAM) used as an external cache. By way of example and not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0141] It should be noted that when the processor is a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, the memory (storage module) can be integrated in the processor.

[0142] It should be noted that the memory described herein is intended to include, but not limited to, these and any other suitable types of memory.

[0143] Based on the same technical concept, the embodiment of the present application also provides a computer readable storage medium, comprising a program or instructions, which, when running on a computer, causes the method steps shown in FIG. 3 to be executed.

[0144] Based on the same technical concept, the embodiment of the present application also provides a chip, which is coupled with a memory, for reading and executing program instructions stored in the memory, so that the method steps shown in FIG. 3 are executed.

[0145] Based on the same technical concept, the embodiment of the present application also provides a computer program product, comprising instructions, which, when running on a computer, causes the method steps shown in FIG. 3 to be executed.

Claims

1. A robustness testing method, characterized by, The method comprises: obtaining an adversarial sample, the adversarial sample being obtained by adding N perturbations to an original sample, the N being a positive integer greater than 1, the original sample being perception data or regulation data, the N perturbations comprising one key perturbation and N-1 non-key perturbations; testing robustness of a processing model using the adversarial sample, the processing model being used for identifying samples.

2. The method of claim 1, wherein the frequency of the key perturbation affecting the use scenario of the processing model exceeds a first threshold, and the frequency of the non-key perturbation affecting the use scenario of the processing model does not exceed a second threshold; or the key perturbation causes the accuracy of the processing model in the use scenario to decrease by an amplitude exceeding a third threshold, and the non-key perturbation causes the accuracy of the processing model in the use scenario to decrease by an amplitude not exceeding a fourth threshold.

3. The method of claim 1 or 2, wherein, The method comprises: obtaining L adversarial sample sets, the L being a positive integer greater than 1, the L adversarial sample sets corresponding to L non-key perturbation parameters one by one, the adversarial samples in each adversarial sample set of the L adversarial sample sets being added with a non-key perturbation parameter corresponding to the each adversarial sample set, the non-key perturbation parameter being a parameter conforming to the N-1 non-key perturbations, at least two adversarial samples in each adversarial sample set of the L adversarial sample sets corresponding to the same original sample and being added with different key perturbation parameters, wherein the key perturbation parameter is a parameter conforming to the key perturbation; The method comprises: calculating a safety radius of the processing model under the key perturbation using each adversarial sample set of the L adversarial sample sets, to obtain L safety radii corresponding to the L non-key perturbation parameters one by one; determining the robustness of the processing model under the key perturbation according to the L safety radii.

4. The method of claim 3, wherein, Each adversarial sample set of the L adversarial sample sets comprises M groups of adversarial samples, the M groups of adversarial samples corresponding to M original samples one by one, each adversarial sample in each group of adversarial samples being obtained by adding a different key perturbation parameter to an original sample corresponding to the each group of adversarial samples, the M being a positive integer; The method comprises: calculating a safety radius of the processing model under the key perturbation using each group of adversarial samples in the each adversarial sample set, to obtain M safety radii; determining a safety radius corresponding to the each adversarial sample set based on the M safety radii.

5. The method of claim 4, wherein the M = 1, and the M safety radii are the safety radii corresponding to the each adversarial sample set; the M > 1, and an expected value of the M safety radii is the safety radius corresponding to the each adversarial sample set.

6. The method according to any one of claims 3 to 5, wherein, The original samples corresponding to each adversarial sample set of the L adversarial sample sets are the same.

7. The method according to any one of claims 3 to 6, wherein, The method comprises: mapping the L safety radii into an N-dimensional coordinate system according to the L non-critical disturbance parameters; wherein a first dimension of the N-dimensional coordinate system corresponds to a safety radius, and N-1 dimensions other than the first dimension one-to-one correspond to the N-1 non-critical disturbances; calculating a volume of an N-dimensional space formed by the L safety radii in the N-dimensional coordinate system; taking the volume of the N-dimensional space as the robustness of the processing model under the critical disturbance.

8. The method according to any one of claims 1 to 7, wherein, The processing model is applied to a driving system; and the original sample is one or more of an image, a point cloud, audio, perception results, and a driving trajectory.

9. The method according to any one of claims 1 to 8, wherein, The N disturbances include one or more of the following: light, rain, fog, snow, noise, temperature, humidity, and air conditions.

10. A processing device, characterized by The method comprises: an acquisition module configured to acquire an adversarial sample, the adversarial sample being obtained by adding N disturbances to an original sample, the N being a positive integer greater than 1, the original sample being perception data or regulation data, and the N disturbances including one critical disturbance and N-1 non-critical disturbances; a test module configured to test the robustness of a processing model using the adversarial sample, the processing model being used for identifying a sample.

11. The apparatus of claim 10, wherein a frequency of the critical disturbance affecting a use scenario of the processing model exceeds a first threshold, and a frequency of the non-critical disturbance affecting the use scenario of the processing model does not exceed a second threshold; or an impact degree of the critical disturbance on the use scenario of the processing model exceeds a third threshold, and an impact degree of the non-critical disturbance on the use scenario of the processing model does not exceed a fourth threshold.

12. The apparatus of claim 10 or 11, wherein, The acquisition module is configured to: acquire L adversarial sample sets, the L being a positive integer greater than 1, the L adversarial sample sets one-to-one corresponding to L non-critical disturbance parameters, adversarial samples in each adversarial sample set of the L adversarial sample sets being added with a non-critical disturbance parameter corresponding to the each adversarial sample set, the non-critical disturbance parameter being a parameter conforming to the N-1 non-critical disturbances, and at least two adversarial samples in each adversarial sample set of the L adversarial sample sets corresponding to a same original sample and being added with different critical disturbance parameters, wherein the critical disturbance parameter is a parameter conforming to the critical disturbance; The test module is configured to: calculate a safety radius of the processing model under the critical disturbance using each adversarial sample set of the L adversarial sample sets, to obtain L safety radii one-to-one corresponding to the L non-critical disturbance parameters; determine the robustness of the processing model under the critical disturbance according to the L safety radii.

13. The apparatus of claim 12, wherein, Each adversarial sample set of the L adversarial sample sets includes M groups of adversarial samples, the M groups of adversarial samples one-to-one corresponding to M original samples, each adversarial sample in each group of adversarial samples being obtained by adding a different critical disturbance parameter to an original sample corresponding to the each group of adversarial samples, and the M being a positive integer; The test module is configured to: calculate a safety radius of the processing model under the critical disturbance using each group of adversarial samples in each adversarial sample set, to obtain M safety radii. determine a safety radius corresponding to each of the L sets of adversarial samples based on the M safety radii. 14.The apparatus of claim 13, wherein, the M is 1, and the M safety radii are safety radii corresponding to each of the L sets of adversarial samples; or the M is greater than 1, and expected values of the M safety radii are safety radii corresponding to each of the L sets of adversarial samples.

15. The apparatus of any one of claims 12-14, wherein, each of the L sets of adversarial samples corresponds to a same original sample.

16. The apparatus of any one of claims 12-15, wherein, The testing module is configured to: map the L safety radii into an N-dimensional coordinate system according to the L non-critical perturbation parameters, wherein a first dimension of the N-dimensional coordinate system corresponds to safety radius, and N-1 dimensions other than the first dimension one-to-one correspond to the N-1 non-critical perturbations; calculate a volume of an N-dimensional space formed by the L safety radii in the N-dimensional coordinate system; use the volume of the N-dimensional space as the robustness of the processing model under the critical perturbation.

17. The apparatus of any one of claims 10-16, wherein, The processing model is applied to a driving system, and the original sample is one or more of an image, a point cloud, an audio, a perception result, and a driving trajectory.

18. The apparatus of any one of claims 10-17, wherein, The N perturbations include one or more of the following: light, rain, fog, snow, noise, temperature, humidity, and air condition.

19. A processing device, comprising: The processing device includes a processor configured to execute computer-executable instructions stored in a memory to cause the processing device to perform the method of any one of claims 1-9.

20. A computer-readable storage medium, characterized in that, The storage medium stores a computer program or instructions, which, when executed by a communication device, implement the method of any one of claims 9.

21. A computer program product, characterised in that, The computer program product stores instructions, which, when executed on a computer, cause the computer to perform the method of any one of claims 1-9.

Images