An adversarial texture generation method and system for automatic driving semantic segmentation

Abstract

The application provides an adversarial texture generation method and system for automatic driving semantic segmentation, and belongs to the technical field of artificial intelligence, comprising: modeling an adversarial texture generation 3D model for a target vehicle, and obtaining a faces id list of adversarial textures to be generated; constructing training data and test data of the adversarial texture generation 3D model based on a preset data set; training the adversarial texture generation 3D model by using the training data and the test data, iteratively optimizing output adversarial textures, and obtaining optimal adversarial textures. The application has strong attack resistance and robustness by making adversarial samples for high-dimensional objects in an automatic driving scene, optimizing and rendering vehicle adversarial textures, and proposing a semantic average loss, guiding model attention to a non-target area in the process of optimizing the semantic average loss, introducing a differentiable renderer to construct an end-to-end training framework, and optimizing adversarial textures by using a back propagation algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an adversarial texture generation method and system for automatic driving semantic segmentation. BACKGROUND

[0002] In an automatic driving system, a semantic segmentation module is located at an upstream position thereof and plays a crucial role in the safety of automatic driving. Since a deep learning model is largely used therein, the deep learning model system is vulnerable to adversarial samples, and therefore, research on adversarial attacks thereon has attracted widespread attention.

[0003] Existing methods generally adopt an adversarial patch (AP) based method, which generally has the problems of poor attack effect and robustness, and is not fully suitable for the scenario of automatic driving.

[0004] Therefore, in view of the limitations of existing methods, a new adversarial texture generation method needs to be proposed. SUMMARY

[0005] The present application provides an adversarial texture generation method and system for automatic driving semantic segmentation, which solves the problem of poor attack effect and robustness of the existing adversarial texture generation method based on adversarial patches in the scenario of automatic driving.

[0006] In a first aspect, the present application provides an adversarial texture generation method for automatic driving semantic segmentation, comprising:

[0007] modeling an adversarial texture generation 3D model for a target vehicle to obtain a faces id list of adversarial textures to be generated;

[0008] constructing training data and test data of the adversarial texture generation 3D model based on a preset data set;

[0009] training the adversarial texture generation 3D model using the training data and the test data, iteratively optimizing the output adversarial texture, and obtaining an optimal adversarial texture.

[0010] According to the adversarial texture generation method for automatic driving semantic segmentation provided by the present application, an adversarial texture generation 3D model is modeled for a target vehicle to obtain a faces id list of adversarial textures to be generated, comprising:

[0011] determining a 3D initial model, obtaining a model file and a material library file of the 3D initial model;

[0012] Import the model file into a preset 3D model editing software, replace the original texture with the adversarial texture at the preset spraying position of the target vehicle according to the preset spraying shape, retain the original texture of the target vehicle except the preset spraying position, and obtain the adversarial texture generation 3D model;

[0013] Export the faces id list corresponding to the adversarial texture generation 3D model through the code runner of the preset 3D model editing software.

[0014] According to the adversarial texture generation method for automatic driving semantic segmentation provided by the application, the training data and test data of the adversarial texture generation 3D model are constructed based on a preset data set, which includes:

[0015] Import the adversarial texture generation 3D model into the CARLA data set, and collect a plurality of pictures according to different random locations, different distances, different camera pitch angles and different yaw angles;

[0016] According to a preset proportion, the plurality of pictures are divided into the training data and the test data, and the semantic segmentation labels corresponding to the training data and the test data are determined.

[0017] According to the adversarial texture generation method for automatic driving semantic segmentation provided by the application, the training data and the test data are used to train the adversarial texture generation 3D model, and the output adversarial texture is iteratively optimized to obtain the optimal adversarial texture, which includes:

[0018] Determine the maximum number of iterations and the minibatch size of each epoch;

[0019] Obtain the initialized adversarial texture by random initialization;

[0020] Using a neural renderer, based on the adversarial texture generation 3D model, the initialized adversarial texture and the camera parameters of the collected data, a rendered picture is obtained;

[0021] Determine the binary mask matrix of the training data, and fuse the binary mask matrix, the training data and the rendered picture to obtain an adversarial sample, the binary mask matrix including vehicle elements with a value of 1 and background elements with a value of 0;

[0022] Input the adversarial sample into a target attack model to obtain a semantic segmentation output prediction classification probability, determine the adversarial picture segmentation result from the prediction classification probability, and the semantic segmentation output prediction classification probability includes any pixel point classification probability;

[0023] Calculate a plurality of loss functions of the adversarial picture segmentation result, perform back propagation optimization of the adversarial texture by comprehensively synthesizing the plurality of loss functions through a preset optimization algorithm, and determine the original texture of the target vehicle according to a mask of drawing an adversarial sample region.

[0024] Re-execute the step of obtaining a rendered picture by the neural renderer, and obtain the optimal adversarial texture when all training rounds end.

[0025] According to the adversarial texture generation method for automatic driving semantic segmentation provided by the application, a plurality of loss functions of the adversarial picture segmentation result are calculated, including:

[0026] The vehicle prediction classification probability in the adversarial picture segmentation result is obtained from the prediction classification probability output by the semantic segmentation, and the vehicle classification score in the vehicle prediction classification probability is determined, and the semantic average loss function is obtained from any connected component, any connected component pixel average value, the total number of pixels of the picture and the number of pixels of any connected component of the vehicle classification score.

[0027] The target semantic segmentation model is attacked based on the adversarial picture segmentation result, and the adversarial attack query value of the adversarial picture segmentation result is determined, and the cross-entropy loss function is obtained from the adversarial attack query value, the adversarial sample label and the total number of pixels of the picture.

[0028] The adjacent pixels of any pixel in the adversarial picture segmentation result are obtained, and the smoothing loss function is obtained based on all adjacent pixels.

[0029] According to the adversarial texture generation method for automatic driving semantic segmentation provided by the application, the back propagation optimization of the adversarial texture is performed by comprehensively synthesizing the plurality of loss functions through a preset optimization algorithm, including:

[0030] A control semantic average loss size hyperparameter is determined, and the overall loss function is obtained based on the control semantic average loss size hyperparameter, the semantic average loss function, the cross-entropy loss function and the smoothing loss function.

[0031] An Adam optimizer is used to calculate the overall loss function and the original texture of the adversarial texture generation 3D model, and the final generated adversarial texture is obtained.

[0032] According to the adversarial texture generation method for automatic driving semantic segmentation provided by the application, the original texture of the target vehicle is determined according to a mask of drawing an adversarial sample region, including:

[0033] The drawing adversarial sample region is obtained.

[0034] If it is determined that any of the preset spraying shape regions is not in the drawn adversarial sample region, the original texture of the any of the preset spraying shape regions is determined;

[0035] The original texture of the target vehicle except the preset spraying position is acquired, and the original texture of the any of the preset spraying shape regions and the original texture of the target vehicle except the preset spraying position constitute the original texture of the target vehicle.

[0036] According to the automatic driving semantic segmentation-oriented adversarial texture generation method provided by the application, the training data and the test data are used to train the adversarial texture generation 3D model, the output adversarial texture is iteratively optimized, and after the optimal adversarial texture is obtained, the method further comprises:

[0037] The preset evaluation index is used to test and evaluate the attack effect of the optimal adversarial texture.

[0038] In the second aspect, the application further provides an automatic driving semantic segmentation-oriented adversarial texture generation system, comprising:

[0039] The generation module is configured to model the target vehicle by using the adversarial texture generation 3D model, and acquire a faces id list of the adversarial texture to be generated.

[0040] The construction module is configured to construct training data and test data of the adversarial texture generation 3D model based on a preset data set.

[0041] The training module is configured to train the adversarial texture generation 3D model by using the training data and the test data, iteratively optimize the output adversarial texture, and acquire the optimal adversarial texture.

[0042] In the third aspect, the application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the automatic driving semantic segmentation-oriented adversarial texture generation method according to any of the above aspects when executing the program.

[0043] The automatic driving semantic segmentation-oriented adversarial texture generation method and system provided by the application have strong attack resistance and robustness by making adversarial samples and optimizing and rendering vehicle adversarial textures for high-dimensional objects in an automatic driving scene, and propose a semantic average loss, guide model attention to a non-target region in the process of optimizing the semantic average loss, and introduce a differentiable renderer to construct an end-to-end training framework, and optimize the adversarial texture by using a back propagation algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort.

[0045] Figure 1 is a flowchart of the adversarial texture generation method for automatic driving semantic segmentation provided by the present application;

[0046] Figure 2 is a principle diagram of the adversarial texture generation method for automatic driving semantic segmentation provided by the present application;

[0047] Figure 3 is a structural schematic diagram of the adversarial texture generation system for automatic driving semantic segmentation provided by the present application;

[0048] Figure 4 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0049] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort belong to the protection scope of the present application.

[0050] In view of the problems of poor effect and poor adaptability of the existing adversarial attack method based on adversarial patches in the automatic driving scene, the embodiment of the present application proposes an adversarial texture generation method for automatic driving semantic segmentation based on neural rendering, which involves using a neural rendering method to control adversarial texture generation. Here, the neural renderer (Neural Render) is proposed by Kato et al. In the rasterization rendering framework, the gradient is approximated, the traditional rendering method based on physical laws, discretization and non-differentiable becomes differentiable, the scene environment and 3D model are taken as parameters, and different perspectives, different distances and different textures of 3D objects can be rendered into 2D pictures. The embodiment of the present application uses the neural renderer as a differentiable renderer to draw the adversarial texture on the attack carrier. In the automatic driving scene, the embodiment of the present application first proposes a 3D adversarial texture generation method for semantic segmentation based on the vehicle as an attack vector, and proposes a semantic average loss function to increase the discrete degree of context features, and mislead the judgment basis of the model.

[0051] Figure 1is a flowchart of an adversarial texture generation method for automatic driving semantic segmentation provided by an embodiment of the present application, as shown in Figure 1 , comprising:

[0052] Step 100: modeling the target vehicle with an adversarial texture generation 3D model, and obtaining a faces id list of the adversarial texture to be generated;

[0053] Step 200: constructing training data and test data of the adversarial texture generation 3D model based on a preset data set;

[0054] Step 300: training the adversarial texture generation 3D model using the training data and the test data, iteratively optimizing the output adversarial texture, and obtaining the optimal adversarial texture.

[0055] Specifically, as shown in Figure 2 , first, according to the constructed 3D ModelM, a differentiable renderer R is used to render model pictures I under a specific camera perspective, height, and yaw angle r Then, combined with the vehicle mask (Car Mask), the environmental information of the original training sample is obtained, and after querying the target attack model (represented by Feature Labels), the comprehensive loss function L of three loss functions (combined with the data set Dataset(x,c,y)) is calculated adv , and finally, the optimization T of the adversarial texture is completed through backpropagation (Backpropagation) and the pre-set non-spraying area. adv Specifically, the following steps are included:

[0056] Modeling the object with a 3D model, and defining a faces id list of the adversarial texture to be generated;

[0057] Prepare training data and test data for experiments, which are usually collected using common data sets;

[0058] Iteratively optimize the generated adversarial texture, and save the optimal adversarial texture after the training converges.

[0059] The embodiment of the application aims to propose a 3D adversarial texture generation method based on a vehicle as an attack vector in an automatic driving scene. The vehicle containing the adversarial texture to be optimized is rendered through a neural renderer, and the background information of the vehicle is obtained by combining a scene fusion function. In the loss function design aspect, according to the characteristics of the semantic segmentation task and the characteristics of the semantic segmentation model with the help of the context information classification, a semantic average loss is proposed, and the attention of the model is guided to the non-target area in the process of optimizing the semantic average loss. After querying the target model, the semantic average loss function representing the discrete degree of the characteristic is calculated, and then the cross-entropy loss function representing the similarity between the real distribution and the predicted distribution and the smooth loss function constraining the naturalness of the adversarial texture are combined to form a whole differentiable calculation framework. Finally, the problem is solved by the Adam optimizer, and the adversarial texture is continuously optimized. A large number of experimental results show that the attack method proposed in the application is superior to the existing method in attack effect, especially in the interference of the vehicle category to the model.

[0060] On the basis of the above embodiment, the target vehicle is subjected to adversarial texture generation 3D model modeling, and a faces id list of the adversarial texture to be generated is obtained, including:

[0061] A 3D initial model is determined, and a model file and a material library file of the 3D initial model are obtained;

[0062] The model file is imported into a preset 3D model editing software, the original texture is replaced with an adversarial texture at a preset spraying position of the target vehicle according to a preset spraying shape, the original texture of the target vehicle except the preset spraying position is retained, and an adversarial texture generation 3D model is obtained;

[0063] The faces id list corresponding to the adversarial texture generation 3D model is exported through a code runner of the preset 3D model editing software.

[0064] Specifically, according to the attack vector in the automatic driving scene, which is usually a target vehicle, a 3D model is constructed or selected, denoted as M, and the corresponding 3D model file (.obj) and material library file (.mtl) are retained.

[0065] In a commonly used 3D model editing software, such as 3ds MAX, the built 3D model is opened, and the editable state is entered. By selecting the area (according to the preset spraying shape, usually a triangle) that needs to be sprayed with an adversarial texture, the original texture is replaced with an adversarial texture only at the preset spraying position (such as the vehicle door, roof, hood, etc.) of the vehicle, and the original texture at other positions, such as the vehicle window and tire, is retained, so that the adversarial vehicle is more realistic and natural. Finally, the corresponding faces id list Area is exported through the code runner of 3ds MAX. The faces id list here is usually a model format list output by the 3D model software.

[0066] On the basis of the above-mentioned embodiments, the training data and the test data of the adversarial texture generation 3D model are constructed based on a preset data set, comprising:

[0067] The adversarial texture generation 3D model is imported into the CARLA data set, and a plurality of pictures are collected according to different random locations, different distances, different camera pitch angles and different yaw angles;

[0068] The plurality of pictures are divided into the training data and the test data according to a preset proportion, and the semantic segmentation labels corresponding to the training data and the test data are determined.

[0069] Specifically, the embodiment of the present application uses CARLA (CAR Learning to Act) to collect the data set. CARLA is a famous automatic driving simulation software based on Unreal4 engine. CARLA can simulate various environments in the real world (such as vehicles, buildings and pedestrians, etc.) with high precision and photorealism. The model constructed in the foregoing embodiment is imported into CARLA. The embodiment of the present application collects data under different environments and different camera parameters. Specifically, 12500 training pictures, 3000 test pictures and their corresponding semantic segmentation labels are collected through different combinations of 155 random locations, 4 distances, 4 camera pitch angles and 8 yaw angles.

[0070] Compared with the adversarial attack based on the adversarial patch of the plane, the present application selects an object (vehicle) with higher dimension as the attack vector, uses the vehicle as the medium, draws the adversarial sample on the vehicle, which is more natural and hidden. At the same time, due to the 3D nature of the vehicle, by optimizing the 3D adversarial texture, the adversarial sample after rendering to the vehicle is less affected by the perspective transformation of the attacker, and the attack effect is better, which overcomes the disadvantage of poor robustness of the adversarial patch.

[0071] On the basis of the above-mentioned embodiments, the training data and the test data are used to train the adversarial texture generation 3D model, and the output adversarial texture is iteratively optimized to obtain the optimal adversarial texture, comprising:

[0072] The maximum number of iterations and the minibatch size of each epoch are determined;

[0073] An initialized adversarial texture is obtained by random initialization;

[0074] A neural renderer is adopted to obtain a rendered picture based on the adversarial texture generation 3D model, the initialized adversarial texture and the camera parameters of the collected data;

[0075] determine a binary mask matrix of the training data, the binary mask matrix including a vehicle element assigned a value of 1 and a background element assigned a value of 0, fuse the binary mask matrix, the training data and the rendered picture to obtain an adversarial sample;

[0076] input the adversarial sample into a target attack model to obtain semantic segmentation output prediction classification probability, the semantic segmentation output prediction classification probability including any pixel point classification probability;

[0077] calculate a plurality of loss functions of the adversarial picture segmentation result, perform back propagation optimization of adversarial texture by a preset optimization algorithm based on the plurality of loss functions, and determine the original texture of the target vehicle according to a mask drawn on an adversarial sample region;

[0078] re-execute the step of obtaining the rendered picture by the neural renderer, and obtain the optimal adversarial texture after all training rounds are completed.

[0079] The calculating the plurality of loss functions of the adversarial picture segmentation result includes:

[0080] obtain vehicle prediction classification probability in the adversarial picture segmentation result from the semantic segmentation output prediction classification probability, determine a vehicle classification score in the vehicle prediction classification probability, and obtain a semantic average loss function based on any connected component, any connected component pixel average value, a total number of picture pixels and any connected component pixel number of the vehicle classification score;

[0081] query a target semantic segmentation model based on the adversarial picture segmentation result to determine an adversarial attack query value of the adversarial picture segmentation result, and obtain a cross-entropy loss function based on the adversarial attack query value, an adversarial sample label and the total number of picture pixels;

[0082] obtain adjacent pixels of any pixel in the adversarial picture segmentation result, and obtain a smoothing loss function based on all adjacent pixels.

[0083] The performing back propagation optimization of adversarial texture by the preset optimization algorithm based on the plurality of loss functions includes:

[0084] determine a control semantic average loss size hyperparameter, obtain an overall loss function based on the control semantic average loss size hyperparameter, the semantic average loss function, the cross-entropy loss function and the smoothing loss function;

[0085] use an Adam optimizer to calculate the overall loss function and the original texture of the adversarial texture to generate a 3D model, and obtain a final generated adversarial texture.

[0086] The original texture of the target vehicle is determined according to the mask of the drawn adversarial sample region, and the original texture of the target vehicle comprises:

[0087] The drawn adversarial sample region is obtained.

[0088] If it is determined that any preset spraying shape region is not in the drawn adversarial sample region, the original texture of the any preset spraying shape region is determined.

[0089] The original texture of the target vehicle except the preset spraying position is obtained, and the original texture of the any preset spraying shape region and the original texture of the target vehicle except the preset spraying position constitute the original texture of the target vehicle.

[0090] Specifically, the training process of the embodiment of the present application specifically comprises the following steps:

[0091] (1) setting the maximum number of iterations E;

[0092] (2) setting the minibatch batch size of each epoch, for example, setting it to 8; here, the epoch represents a complete process of training once using all samples in the training set, and the minibatch represents the minimum sample quantity selected before each adjustment of parameters;

[0093] (3) initializing the adversarial texture:

[0094] In the overall attack framework, the adversarial texture needs to be initialized first, and different initialization methods will have different effects on the attack effect. The embodiment of the present application compares experiments by considering three initialization methods: original texture initialization (Origin), all-zero initialization (Zeros), and random initialization (Random). It is found through experiments that, for most models, random initialization achieves the best attack effect, and original texture initialization has the worst attack effect. Therefore, the embodiment of the present application selects random initialization as the initialization method, which guarantees the attack effect of ACS while ensuring the naturalness of the adversarial texture.

[0095] (4) using a neural renderer to render an object containing an adversarial texture:

[0096] According to the constructed model M and the camera parameters c when collecting data, the embodiment of the present application uses a neural renderer to render a 3D object under the condition of a specific view angle, distance and yaw angle, to obtain a rendered picture I r , that is, r I adv = R((m, T adv ), c), wherein T adv represents the adversarial texture, and R is a differentiable renderer.

[0097] (5) obtaining environment information according to an object mask to generate an adversarial picture:

[0098] Since neural renderers cannot render the background, in order to capture the contextual semantic information of the scene in which the vehicle is located, this embodiment of the invention proposes a scene fusion function F to fuse the adversarial vehicle rendered by the renderer with the original background, thereby obtaining more realistic attack scene data, namely: F=(1-m)*x+m* adv , where m is the binary mask matrix corresponding to sample x, with the same size as the sample. The portion of the mask matrix containing vehicles is 1, and the background portion is 0. The application scenario uses a fusion function to obtain the adversarial sample I to be input into the target semantic segmentation model. adv :I adv =F(I r ,m,x).

[0099] (6) Query the segmentation results of the adversarial image by the target attack model.

[0100] An adversarial example (an image with height H, width W, and number of channels C) is fed into the target attack model, and the corresponding semantic segmentation is queried to obtain the output predicted classification probability. Where N c f represents the number of categories. i j (x)∈[0,1] represents the probability that the i-th pixel in the image is classified into the j-th class. Then, the class with the highest score among all predicted classes is taken as the final predicted class. in:

[0101] (7) Calculate the loss function, optimize the adversarial texture through backpropagation, and preserve the original texture based on the region mask of the texture:

[0102] The attack proposed in this invention aims to reduce the segmentation performance of the target semantic segmentation model (especially the vehicle category), misclassifying images containing adversarial vehicles into other categories at the pixel level. This is achieved through a loss function based on semantic averaging, a loss function based on cross-entropy, and a smoothing loss function, respectively.

[0103] First, the semantic average loss L is calculated. avg :

[0104] The output predicted classification probability is obtained by querying the target semantic segmentation model. Subsequently, embodiments of the present invention focus on the classification probability score (I) for the vehicle category. adv )∈[0,1] (H×W) That is, for each pixel in the image, the score given to the vehicle during model classification is: score ij =∑ k f(x) ijk ,k∈S vehicle Svehicle The classification label set is a category of vehicles. Since these feature labels determine the final classification result, and most models today will converge other associated region information to classify pixels, it is proposed to average the semantic labels to interfere with the context acquisition of the target model. Specifically, the context relationship is represented using connected components, and the context acquisition area of the model is shifted by increasing the number of connected components, averaging the weights of each connected component, and the weights of each pixel within the connected component. The calculation method is as follows:

[0105]

[0106] where G k is the kth connected component of score, N is the total number of pixels of the picture, N k is the number of pixels of the kth connected component, is the average value of the kth connected component. By optimizing L avg , the attention of the model to vehicle classification becomes average, that is, the confidence score of the vehicle region in the picture is reduced, and the confidence score of other regions is improved.

[0107] Secondly, the cross-entropy loss L cr is calculated:

[0108]

[0109] where N = {1,…,HxW} represents the set of all pixels in I adv , SS is the target semantic segmentation model queried during attack, SS i is any element in the model, y i is any adversarial sample label, and y is all adversarial sample labels.

[0110] Thirdly, the smooth loss L smooth is calculated:

[0111] In order to make the adversarial texture more natural, the smooth loss proposed by Eykholt et al. is adopted, that is, for the rendered adversarial image I adv , the adjacent pixels in the adversarial image are as close as possible:

[0112] L smooth =∑(x i,j -x i+1,j ) 2 +(x i,j -x i,j+1 ) 2

[0113] where x i,j is the pixel value of the adversarial image I adv at coordinate (i,j), and xi+1,j and x i,j+1 are the adversarial image I adv is the value of the neighboring pixel at coordinates (i,j).

[0114] Finally, the adversarial texture is optimized by Adam, and the overall loss function is as follows:

[0115] L adv = αL avg -L ce +L smooth

[0116] wherein α is a hyperparameter that controls the size of the semantic average loss.

[0117] Therefore, the generation of the adversarial texture can be formalized as solving the following optimization problem:

[0118]

[0119] For the finally generated adversarial texture, T is the original texture of the 3D model, and the problem is solved using the Adam optimizer.

[0120] Further, the original texture of the object is set according to the Area mask:

[0121] Area is the area that needs to draw the adversarial sample, and in order to make the 3D object natural, the original texture is retained in the area such as the window, so the area of all triangles of the adversarial texture is judged, and if it is not in the Area set, the original texture is set back.

[0122] (8) reacquire step (4);

[0123] (9) after all training rounds are completed, save the model and output the optimal adversarial texture.

[0124] In the design of the loss function, in view of the characteristics that the existing segmentation model will converge the information of other related areas to classify the pixel points, the present application proposes to average the semantic label, interfere with the context acquisition of the target model, and shift the context acquisition area of the model.

[0125] On the basis of the above embodiment, the training data and the test data are used to train the adversarial texture generated 3D model, the output adversarial texture is iteratively optimized, and after obtaining the optimal adversarial texture, the method further comprises:

[0126] The attack effect of the optimal adversarial texture is tested and evaluated by using a preset evaluation index.

[0127] Optionally, the embodiment of the present application tests and evaluates the attack according to the test set by defining evaluation indexes such as mean Intersection over-Union, mean Accuracy and mean Intersection over-Union of vehicle categories.

[0128] (1) Mean Intersection over-Union (mIoU), mIoU represents the average Intersection over-Union on the class, and the calculation method is as follows:

[0129]

[0130] Wherein, k represents the class, k+1 represents adding the background class, i represents the true value, j represents the predicted value, p ij represents predicting i as j, which is a false negative value, p ji represents predicting j as i, which is a false positive value, p ii represents predicting i as i, which is a true value.

[0131] (2) Mean Accuracy (mAcc), mAcc is the average accuracy at the pixel level, and the calculation method is as follows:

[0132]

[0133] (3) mIoU of vehicle categories, only the mean Intersection over-Union of vehicle categories (such as small car, truck, bus, on rails, motorcycle, bicycle, caravan and trailer).

[0134] The present application tests and verifies the proposed adversarial texture generation method, and verifies the advantages of the model in mean Intersection over-Union, mean Accuracy and mean Intersection over-Union of vehicle categories.

[0135] The adversarial texture generation system for automatic driving semantic segmentation provided by the present application is described below, and the adversarial texture generation system for automatic driving semantic segmentation described below can be correspondingly referred to the adversarial texture generation method for automatic driving semantic segmentation described above.

[0136] Figure 3 The adversarial texture generation system for automatic driving semantic segmentation provided by the present application is described below, and the adversarial texture generation system for automatic driving semantic segmentation described below can be correspondingly referred to the adversarial texture generation method for automatic driving semantic segmentation described above. Figure 3 As shown in FIG. 1, the adversarial texture generation system for automatic driving semantic segmentation provided by the present application comprises a generation module 31, a construction module 32 and a training module 33, wherein:

[0137] The generation module 31 is configured to model an adversarial texture generation 3D model of a target vehicle, and obtain a faces id list of adversarial textures to be generated. The construction module 32 is configured to construct training data and test data of the adversarial texture generation 3D model based on a preset data set. The training module 33 is configured to train the adversarial texture generation 3D model by using the training data and the test data, iteratively optimize the output adversarial texture, and obtain an optimal adversarial texture.

[0138] Figure 4 An example of a schematic diagram of an entity structure of an electronic device is shown in Figure 4 As shown, the electronic device can include a processor 410, a communications interface 420, a memory 430, and a communications bus 440, wherein the processor 410, the communications interface 420, and the memory 430 can communicate with each other through the communications bus 440. The processor 410 can invoke a logical instruction in the memory 430 to execute an adversarial texture generation method for automatic driving semantic segmentation, which includes modeling an adversarial texture generation 3D model of a target vehicle, obtaining a faces id list of adversarial textures to be generated, constructing training data and test data of the adversarial texture generation 3D model based on a preset data set, training the adversarial texture generation 3D model by using the training data and the test data, iteratively optimizing the output adversarial texture, and obtaining an optimal adversarial texture.

[0139] In addition, the logical instruction in the memory 430 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0140] In another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the method for generating adversarial textures for semantic segmentation of autonomous driving provided by the above method, the method comprising: modeling an adversarial texture generation 3D model for a target vehicle, and obtaining a list of faces ids to be generated adversarial textures; constructing training data and test data of the adversarial texture generation 3D model based on a preset data set; training the adversarial texture generation 3D model using the training data and the test data, iteratively optimizing the output adversarial textures, and obtaining optimal adversarial textures.

[0141] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0142] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and necessary general hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in terms of contribution to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0143] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An adversarial texture generation method for autonomous driving semantic segmentation, characterized in that, The method comprises the steps of: modeling an adversarial texture generation 3D model of a target vehicle, obtaining a model format faces id list of an adversarial texture to be generated; constructing training data and test data of the adversarial texture generation 3D model based on a preset data set; training the adversarial texture generation 3D model using the training data and the test data, iteratively optimizing the output adversarial texture, and obtaining an optimal adversarial texture, comprising: determining the maximum number of iterations and the minimum sample size minibatch of each complete training process epoch; obtaining an initialized adversarial texture using random initialization; using a neural renderer to generate a rendered picture based on the adversarial texture generation 3D model, the initialized adversarial texture, and the camera parameters of the collected data; determining a binary mask matrix of the training data, fusing the binary mask matrix, the training data, and the rendered picture to obtain an adversarial sample, and the binary mask matrix comprising vehicle elements with a value of 1 and background elements with a value of 0; inputting the adversarial sample into a target attack model to obtain a semantic segmentation output prediction classification probability, determining an adversarial picture segmentation result from the prediction classification probability, and the semantic segmentation output prediction classification probability comprising any pixel point classification probability; calculating a plurality of loss functions of the adversarial picture segmentation result, comprehensively optimizing the plurality of loss functions through a preset optimization algorithm, and determining the original texture of the target vehicle according to a mask drawn on the adversarial sample region; re-executing the step of obtaining a rendered picture using a neural renderer, and obtaining the optimal adversarial texture after all training rounds are completed; wherein the plurality of loss functions of the adversarial picture segmentation result comprises: obtaining vehicle prediction classification probability in the adversarial picture segmentation result from the semantic segmentation output prediction classification probability, determining vehicle classification scores in the vehicle prediction classification probability, and obtaining a semantic average loss function from any connected component, any connected component pixel average value, total pixel number of the picture, and any connected component pixel number of the vehicle classification scores; querying a target semantic segmentation model based on the adversarial picture segmentation result to determine an adversarial attack query value of the adversarial picture segmentation result, and obtaining a cross-entropy loss function from the adversarial attack query value, an adversarial sample label, and the total pixel number of the picture; obtaining adjacent pixels of any pixel in the adversarial picture segmentation result, and obtaining a smoothing loss function based on all adjacent pixels.

2. The method of claim 1, wherein, The method comprises the steps of: modeling an adversarial texture generation 3D model of a target vehicle, obtaining a model format faces id list of an adversarial texture to be generated; determining a 3D initial model, obtaining a model file and a material library file of the 3D initial model; importing the model file into a preset 3D model editing software, replacing the original texture with an adversarial texture at a preset spraying position of the target vehicle according to a preset spraying shape, retaining the original texture of the target vehicle except at the preset spraying position, and obtaining the adversarial texture generation 3D model; export the faces id list corresponding to the adversarial texture generated 3D model through a code runner of the preset 3D model editing software.

3. The method of claim 1, wherein, construct training data and test data of the adversarial texture generated 3D model based on a preset data set, including: import the adversarial texture generated 3D model into a CARLA data set, and collect a plurality of pictures according to different random locations, different distances, different camera pitch angles and different yaw angles; divide the plurality of pictures into the training data and the test data according to a preset proportion, and determine semantic segmentation labels corresponding to the training data and the test data.

4. The method of claim 1, wherein, comprehend the plurality of loss functions through a preset optimization algorithm to perform back propagation optimization of the adversarial texture, including: determine a control semantic average loss size hyperparameter, and obtain an overall loss function based on the control semantic average loss size hyperparameter, the semantic average loss function, the cross-entropy loss function and the smoothness loss function; adopt an Adam optimizer to calculate the overall loss function and an original texture of the adversarial texture generated 3D model, and obtain a final generated adversarial texture.

5. The method of claim 4, wherein, determine the original texture of the target vehicle according to a mask of a drawn adversarial sample region, including: obtain the drawn adversarial sample region; if it is determined that any preset spraying shape region is not in the drawn adversarial sample region, then determine an original texture of the any preset spraying shape region; obtain an original texture of the target vehicle except a preset spraying position, and combine the original texture of the any preset spraying shape region to obtain the original texture of the target vehicle.

6. The method of claim 1, wherein, train the adversarial texture generated 3D model by using the training data and the test data, iteratively optimize the output adversarial texture, and after obtaining the optimal adversarial texture, further include: test and evaluate the attack effect of the optimal adversarial texture by using a preset evaluation index.

7. An adversarial texture generation system for autonomous driving semantic segmentation based on the adversarial texture generation method for autonomous driving semantic segmentation of any of claims 1 to 6. including: a generation module configured to model a target vehicle by using an adversarial texture generated 3D model, and obtain a faces id list of an adversarial texture to be generated; a construction module configured to construct training data and test data of the adversarial texture generated 3D model based on a preset data set; a training module configured to train the adversarial texture generated 3D model by using the training data and the test data, iteratively optimize the output adversarial texture, and obtain an optimal adversarial texture.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, the processor implements the automatic driving semantic segmentation facing adversarial texture generation method according to any one of claims 1 to 6 when executing the program.

Images