Adversarial sample generation method based on automatic driving multi-sensor fusion

Abstract

The invention provides an adversarial sample generation method based on automatic driving multi-sensor fusion, which can be applied to the technical field of adversarial networks. The method comprises the steps that for each view angle in a plurality of view angles, a three-dimensional object model is perceived to obtain perception data, and the perception data comprises visual perception data and radar perception data; obtaining visual loss data based on the visual perception data and disturbed visual perception data obtained by disturbing the visual perception data, and obtaining radar loss data based on the radar perception data and disturbed radar perception data obtained by disturbing the radar perception data; based on candidate adversarial samples generated by the disturbance perception data of the multiple visual angles and the visual perception data, respectively calculating alignment loss and smooth loss to obtain alignment loss data and smooth loss data; the candidate adversarial sample is adjusted based on the visual loss data, the radar loss data, the alignment loss data, and the smoothing loss data to generate a target adversarial sample.

Description

Technical Field

[0001] The present invention relates to the field of adversarial network technology, and more specifically to an adversarial sample generation method based on multi-sensor fusion for autonomous driving, and an adversarial testing method and device for an autonomous driving system. Background Art

[0002] With the development of deep learning technology, autonomous driving technology has gradually matured. Neural networks have achieved great success in image classification, target detection and other fields. Deep neural networks are widely used in vision and radar perception. However, deep neural networks are vulnerable to adversarial attacks.

[0003] In related technologies, when testing autonomous driving technology, adversarial testing is often only performed on image recognition in the autonomous driving system. However, current autonomous driving systems often use multiple sensors for recognition. Testing the autonomous driving system only with adversarial samples generated by image recognition tests will result in poor adversarial testing results. Summary of the invention

[0004] In view of the above problems, the present invention provides an adversarial sample generation method based on multi-sensor fusion for autonomous driving.

[0005] According to a first aspect of the present invention, a method for generating adversarial samples based on multi-sensor fusion for autonomous driving is provided, comprising: for each of a plurality of viewpoints, respectively perceiving a three-dimensional object model to obtain perception data, wherein the three-dimensional object model is used to simulate obstacles encountered during autonomous driving, and the perception data comprises visual perception data and radar perception data; obtaining visual loss data based on the visual perception data and perturbed visual perception data obtained by perturbing the visual perception data, and obtaining radar loss data based on the radar perception data and perturbed radar perception data obtained by perturbing the radar perception data; calculating alignment loss and smoothing loss based on candidate adversarial samples generated from the perturbed perception data of the plurality of viewpoints and the visual perception data to obtain alignment loss data and smoothing loss data; adjusting the candidate adversarial samples based on the visual loss data, the radar loss data, the alignment loss data and the smoothing loss data to generate target adversarial samples, wherein the target adversarial samples are used to simulate obstacles after the disturbance is added.

[0006] According to an embodiment of the present invention, the above-mentioned candidate adversarial samples are adjusted based on the above-mentioned visual loss data, the above-mentioned radar loss data, the above-mentioned alignment loss data and the above-mentioned smoothing loss data to generate a target adversarial sample, including: obtaining a total loss function value based on the above-mentioned visual loss data, the above-mentioned radar loss data, the above-mentioned alignment loss data and the above-mentioned smoothing loss data; using a stochastic gradient descent algorithm to solve the target perturbation data that minimizes the above-mentioned total loss function value; adjusting the above-mentioned candidate adversarial samples based on the above-mentioned target perturbation data to obtain a target adversarial sample.

[0007] According to an embodiment of the present invention, the above-mentioned three-dimensional object model is constructed by the following method: obtaining respective images and sampling attribute data of the target object at multiple viewing angles, wherein the above-mentioned sampling attribute data includes the acquisition positions of the image acquisition device corresponding to different viewing angles and the acquisition angles relative to the target object; inputting the above-mentioned multiple images and the above-mentioned sampling attribute data into a trained volume radiation field model to obtain the volume radiation field of the above-mentioned target object; and obtaining the above-mentioned three-dimensional object model based on the above-mentioned volume radiation field and the above-mentioned sampling attribute data.

[0008] According to an embodiment of the present invention, the sampling attribute data also includes transmittance, opacity and color of each of the multiple sampling points. The multiple sampling points are distributed on multiple rays from the optical center of the image acquisition device to the target object. The three-dimensional object model is obtained based on the volume radiation field and the sampling attribute data, including: for each ray, a ray rendering color is obtained based on the transmittance of the sampling point, the opacity of the sampling point and the color of the sampling point; and the three-dimensional object model is obtained by rendering the volume radiation field based on the ray rendering colors of each of the multiple rays.

[0009] According to an embodiment of the present invention, the method further includes: using a mean square error loss function to obtain a mean square loss value based on the pixel values ​​of the three-dimensional object model and the pixel values ​​of the target object; and adjusting the three-dimensional object model based on the mean square loss value.

[0010] According to an embodiment of the present invention, the above-mentioned visual loss data is calculated by the following formula: in, represents visual loss data, N represents the total number of viewing angles, represents the color value of the perturbed visual perception data at the i-th viewing angle, represents the color value of the visual perception data at the i-th viewing angle. The above radar loss data is calculated by the following formula: in, represents radar loss data, represents the cross entropy loss function, Represents the point cloud prediction mask of the perturbed radar perception data, Point cloud ground truth mask representing radar perception data.

[0011] According to an embodiment of the present invention, the above alignment loss data is calculated by the following formula: ,in, represents the above alignment loss data, n represents the total number of points in the point cloud formed by the candidate adversarial sample, represents the weight of the kth point, represents the feature of the kth point in the candidate adversarial sample, represents the feature of the kth point in the visual perception data; the above smoothing loss data is calculated by the following formula: ,in, represents the above smoothed loss data, Represents the pixel value at position (i, j) on the above visual perception data, represents the pixel value at position (i+1, j) in the above visual perception data, Represents the pixel value at position (i, j+1) in the above visual perception data.

[0012] The second aspect of the present invention provides an adversarial testing method for an autonomous driving system, comprising: obtaining a target adversarial sample, wherein the target adversarial sample is obtained using the method of the first aspect; using the target adversarial sample to test the autonomous driving system to obtain an adversarial testing result of the autonomous driving system, wherein the adversarial testing result is used to improve the safety of the autonomous driving system.

[0013] The third aspect of the present invention provides an adversarial sample generation device based on multi-sensor fusion for autonomous driving, comprising: a perception module, for sensing a three-dimensional object model for each of a plurality of perspectives to obtain perception data, wherein the three-dimensional object model is used to simulate obstacles encountered during autonomous driving, and the perception data includes visual perception data and radar perception data; a perturbation module, for obtaining visual loss data based on the visual perception data and the perturbed visual perception data obtained by perturbing the visual perception data, and for obtaining radar loss data based on the radar perception data and the perturbed radar perception data obtained by perturbing the radar perception data; a loss data calculation module, for calculating alignment loss and smoothing loss based on candidate adversarial samples generated by the perturbed perception data of the plurality of perspectives and the visual perception data, respectively, to obtain alignment loss data and smoothing loss data; a target adversarial sample generation module, for adjusting the candidate adversarial samples based on the visual loss data, the radar loss data, the alignment loss data and the smoothing loss data to generate target adversarial samples, wherein the target adversarial samples are used to simulate obstacles after the disturbance is added.

[0014] The fourth aspect of the present invention provides an adversarial testing device for an autonomous driving system, comprising: an acquisition module for acquiring a target adversarial sample, wherein the target adversarial sample is obtained using the method of the first aspect; a testing module for testing the autonomous driving system using the target adversarial sample to obtain an adversarial testing result of the autonomous driving system, wherein the adversarial testing result is used to improve the safety of the autonomous driving system.

[0015] A fifth aspect of the present invention provides an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0016] The sixth aspect of the present invention further provides a computer-readable storage medium on which a computer program or instruction is stored, and the steps of the above method are implemented when the above computer program or instruction is executed by a processor.

[0017] The seventh aspect of the present invention also provides a computer program product, including a computer program or instructions, which implement the steps of the above method when executed by a processor.

[0018] The present invention perceives a three-dimensional object model through multiple perspectives to obtain visual perception data and radar perception data, and simultaneously perturbs the visual perception data and radar perception data to obtain candidate adversarial samples, calculates alignment loss and smoothing loss for the candidate adversarial samples and the perception data, and adjusts the candidate adversarial samples through the visual loss data, radar loss data, alignment loss data and smoothing loss data to generate target adversarial samples, thereby ensuring the correspondence between the generated target adversarial samples and the three-dimensional object model, and adjusting the candidate adversarial samples through the smoothing loss data to reduce the difference in adjacent pixel values, thereby ensuring the naturalness and robustness of the generated target adversarial samples in the physical scene, thereby improving the effect of the adversarial test. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0020] Figure 1 An application scenario diagram of an adversarial sample generation method for multi-sensor fusion of autonomous driving, an adversarial testing method for an autonomous driving system, and an apparatus according to an embodiment of the present invention is shown.

[0021] Figure 2 A flowchart of a method for generating adversarial samples based on multi-sensor fusion for autonomous driving according to an embodiment of the present invention is shown.

[0022] Figure 3 A flowchart of generating a target adversarial sample according to an embodiment of the present invention is shown.

[0023] Figure 4 A flow chart of an adversarial testing method for an autonomous driving system according to an embodiment of the present invention is shown.

[0024] Figure 5 A structural block diagram of an adversarial sample generation device based on multi-sensor fusion for autonomous driving according to an embodiment of the present invention is shown.

[0025] Figure 6 A structural block diagram of an adversarial testing device for an autonomous driving system according to an embodiment of the present invention is shown.

[0026] Figure 7 A block diagram of an electronic device suitable for implementing an adversarial sample generation method based on multi-sensor fusion for autonomous driving according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0027] Below, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of concepts of the present invention.

[0028] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0029] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0030] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0031] With the development of deep learning technology, neural networks have achieved great success in image classification, object detection and other fields. However, neural networks also have some defects, one of the most important of which is adversarial sample attacks. Adversarial sample attacks refer to making minor modifications to the original input to cause errors in the output of the neural network. This type of attack may lead to serious security issues.

[0032] With the rapid development of deep learning technology, autonomous driving technology has gradually matured, among which DNN (Deep Neural Network) is particularly widely used in vision and radar perception. However, studies have shown that DNN models are susceptible to adversarial attacks, that is, by applying small perturbations to the input data, the model's prediction results can be significantly changed. This poses a serious safety hazard to autonomous driving systems that rely on precise perception.

[0033] Existing 2D adversarial attack methods mainly target image classification, but these methods have certain limitations in actual 3D scenes. In addition, current 3D adversarial attack methods mostly focus on modifying the texture or geometry of 3D models, but these methods have poor consistency and transferability under different viewing angles, making it difficult to effectively test multimodal perception systems.

[0034] An embodiment of the present invention provides an adversarial sample generation method based on multi-sensor fusion for autonomous driving, comprising: perceiving a three-dimensional object model for each of multiple perspectives to obtain perception data, the three-dimensional object model being used to simulate obstacles encountered during autonomous driving, the perception data comprising visual perception data and radar perception data; obtaining visual loss data based on the visual perception data and perturbed visual perception data obtained by perturbing the visual perception data, and obtaining radar loss data based on the radar perception data and perturbed radar perception data obtained by perturbing the radar perception data; calculating alignment loss and smoothing loss based on candidate adversarial samples generated from the perturbed perception data of multiple perspectives and the visual perception data to obtain alignment loss data and smoothing loss data; adjusting the candidate adversarial samples based on the visual loss data, the radar loss data, the alignment loss data and the smoothing loss data to generate a target adversarial sample, the target adversarial sample being used to simulate obstacles after the disturbance is added.

[0035] Figure 1 An application scenario diagram of an adversarial sample generation method for multi-sensor fusion of autonomous driving, an adversarial testing method for an autonomous driving system, and an apparatus according to an embodiment of the present invention is shown.

[0036] like Figure 1As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0037] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0038] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0039] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0040] It should be noted that the adversarial sample generation method based on multi-sensor fusion of autonomous driving and the adversarial test method of the autonomous driving system provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the adversarial sample generation device based on multi-sensor fusion of autonomous driving and the adversarial test device of the autonomous driving system provided in the embodiment of the present invention can generally be set in the server 105. The adversarial sample generation method based on multi-sensor fusion of autonomous driving and the adversarial test method of the autonomous driving system provided in the embodiment of the present invention can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the adversarial sample generation device based on multi-sensor fusion of autonomous driving and the adversarial test device of the autonomous driving system provided in the embodiment of the present invention can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0041] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0042] The following will be based on Figure 1 The scene described by Figure 2~Figure 3 The adversarial sample generation method based on multi-sensor fusion for autonomous driving according to an embodiment of the present invention is described in detail.

[0043] Figure 2 A flowchart of a method for generating adversarial samples based on multi-sensor fusion for autonomous driving according to an embodiment of the present invention is shown.

[0044] like Figure 2 As shown, the adversarial sample generation method based on multi-sensor fusion for autonomous driving of this embodiment includes operations S210 to S240.

[0045] In operation S210, for each of the multiple perspectives, the three-dimensional object model is sensed to obtain perception data.

[0046] Among them, the three-dimensional object model is used to simulate obstacles encountered during autonomous driving, and the perception data includes visual perception data and radar perception data.

[0047] According to an embodiment of the present invention, the above-mentioned multi-sensor may include a visual perception sensor and a radar perception sensor.

[0048] According to an embodiment of the present invention, the above-mentioned three-dimensional object can be obtained by modeling the target object based on the neural radiation field, or by modeling the target object based on grid, voxel grid and other methods and further rendering. The present invention does not impose any restrictions on this.

[0049] According to an embodiment of the present invention, the above-mentioned viewing angle includes the angle, lighting and background of the three-dimensional object. The angle, lighting and background of the three-dimensional object are adjusted and perceived to obtain perception data under multiple viewing angles, thereby ensuring robustness in a real physical environment.

[0050] In operation S220, visual loss data is obtained based on the visual perception data and the disturbed visual perception data obtained by disturbing the visual perception data, and radar loss data is obtained based on the radar perception data and the disturbed radar perception data obtained by disturbing the radar perception data.

[0051] According to an embodiment of the present invention, the three-dimensional object model may be disturbed by the following formula (1), thereby obtaining disturbed visual perception data and disturbed radar perception data.

[0052]

[0053] in, represents the set of disturbed visual perception data and disturbed radar perception data, Represents a three-dimensional object model, represents a randomly initialized adversarial perturbation.

[0054] According to an embodiment of the present invention, by perturbing the three-dimensional object model, visual perception data and radar perception data can be perturbed at the same time to obtain perturbed visual perception data and perturbed radar perception data.

[0055] According to an embodiment of the present invention, visual loss data can be obtained based on visual perception data and perturbed visual perception data through a first preset loss function, and radar loss data can be obtained based on radar perception data and perturbed radar perception data through a second preset loss function.

[0056] In operation S230, alignment loss and smoothness loss are respectively calculated based on candidate adversarial samples generated from perturbation perception data of multiple perspectives and visual perception data to obtain alignment loss data and smoothness loss data.

[0057] In operation S240 , the candidate adversarial sample is adjusted based on the vision loss data, the radar loss data, the alignment loss data, and the smoothing loss data to generate a target adversarial sample.

[0058] Among them, the target adversarial sample is used to simulate obstacles after adding disturbances.

[0059] The present invention perceives a three-dimensional object model through multiple perspectives to obtain visual perception data and radar perception data, and simultaneously perturbs the visual perception data and radar perception data to obtain candidate adversarial samples, calculates alignment loss and smoothing loss for the candidate adversarial samples and the perception data, and adjusts the candidate adversarial samples through the visual loss data, radar loss data, alignment loss data and smoothing loss data to generate target adversarial samples, thereby ensuring the correspondence between the generated target adversarial samples and the three-dimensional object model, and adjusting the candidate adversarial samples through the smoothing loss data to reduce the difference in adjacent pixel values, thereby ensuring the naturalness and robustness of the generated target adversarial samples in the physical scene, thereby improving the effect of the adversarial test.

[0060] According to an embodiment of the present invention, the above-mentioned adjustment of candidate adversarial samples based on visual loss data, radar loss data, alignment loss data and smoothing loss data to generate target adversarial samples includes: obtaining a total loss function value based on visual loss data, radar loss data, alignment loss data and smoothing loss data; using a stochastic gradient descent algorithm to solve the target perturbation data that minimizes the total loss function value; adjusting the candidate adversarial samples based on the target perturbation data to obtain the target adversarial samples.

[0061] According to an embodiment of the present invention, the total loss function may be calculated using the following formula (2).

[0062]

[0063] in, represents the total loss function, represents visual loss data, represents radar loss data, represents the alignment loss data, represents the smoothed loss data, is a hyperparameter.

[0064] According to an embodiment of the present invention, appropriate hyperparameters may be adjusted according to the scenario to generate target adversarial samples that can be fully tested against.

[0065] According to an embodiment of the present invention, the total loss value can be iteratively optimized by using an optimization algorithm such as a stochastic gradient descent algorithm or a batch gradient descent algorithm, and the disturbance to the three-dimensional object model can be adjusted to minimize the total loss.

[0066] According to an embodiment of the present invention, the effectiveness of the generated target adversarial samples can be verified on a data set, and the aforementioned effectiveness includes the attack success rate on visual perception sensors and radar perception sensors.

[0067] According to an embodiment of the present invention, by integrating visual loss data, radar loss data, alignment loss data and smoothing loss data, the performance of the model on different perception tasks and the consistency and smoothness of the model output are comprehensively considered, and the target perturbation data is obtained by the stochastic gradient descent algorithm to adjust the candidate adversarial samples, so as to obtain the target adversarial samples that can maximize the adversarial testing of the autonomous driving system.

[0068] According to an embodiment of the present invention, a three-dimensional object model is constructed by the following method: obtaining respective images and sampling attribute data of a target object at multiple viewing angles, wherein the sampling attribute data includes acquisition positions of an image acquisition device corresponding to different viewing angles and acquisition angles relative to the target object; inputting the multiple images and sampling attribute data into a trained volume radiation field model to obtain a volume radiation field of the target object; and obtaining a three-dimensional object model based on the volume radiation field and the sampling attribute data.

[0069] According to the embodiment of the present invention, images of the target object from multiple perspectives can be collected in the scene to establish a three-dimensional data set. ,in, Represents images from different perspectives. The volume radiation field of a three-dimensional object can be generated by a trained volume radiation field model, and then the volume radiation field is rendered based on the volume radiation field and the sampled attribute data to obtain a three-dimensional object model.

[0070] According to an embodiment of the present invention, the above-mentioned sampling attribute data also includes the transmittance, opacity and color of each of the multiple sampling points. The multiple sampling points are distributed on multiple rays from the optical center of the image acquisition device to the target object. A three-dimensional object model is obtained based on the volume radiation field and the sampling attribute data, including: for each ray, a ray rendering color is obtained based on the transmittance of the sampling point, the opacity of the sampling point and the color of the sampling point; and the volume radiation field is rendered based on the ray rendering colors of each of the multiple rays to obtain the three-dimensional object model.

[0071] According to an embodiment of the present invention, a target object in a three-dimensional scene may be sampled along a line of sight or a light direction, and the density, color, and transparency of the sampling points jointly determine the final pixel color of the ray.

[0072] The ray rendering color can be calculated using the following formula (3).

[0073]

[0074] in, represents the final pixel color rendered along ray r, N represents the number of sampling points on the ray, represents the transmittance from the starting point of the ray to the i-th sampling point, is the volume density or absorption coefficient of the ith sampling point, represents the distance between adjacent sampling points, represents the opacity of the ith sampling point, Indicates the color of the i-th sampling point.

[0075] According to the embodiment of the present invention, the above formula (3) can be used to calculate each ray firstly render the color according to its own ray and then further render the volume radiation field to obtain the above three-dimensional object model.

[0076] According to an embodiment of the present invention, the volume density and color can be predicted by a neural radiation field model, which receives an input five-dimensional coordinate (x, y, z, θ, ψ), where x, y, z are position coordinates in three-dimensional space, θ, ψ represent observation angles (azimuth and elevation, respectively), outputs the volume density of each sampling point and the emission amplitude value (color) related to the viewing angle at that position, and predicts the transmittance and opacity of the sampling point through the volume density, thereby rendering the volume radiation field to obtain a three-dimensional object model.

[0077] According to an embodiment of the present invention, images and sampling attribute data of a target object are acquired through multiple viewing angles and input into a trained volume radiation field model to obtain a volume radiation field of the target object, so that the details of the target object can be accurately captured, thereby generating a more realistic three-dimensional model; the ray rendering color of each ray is obtained by calculating the sampling attributes, and finally the volume radiation field is rendered based on the ray rendering color to obtain a three-dimensional object model, comprehensively considering the scattering and absorption of light when passing through the object, and calculating the transmittance, opacity and color of the sampling points on each ray to obtain a more accurate rendering result, thereby better simulating the lighting and material effects in the real world.

[0078] According to an embodiment of the present invention, the method further includes: using a mean square error loss function to obtain a mean square loss value based on pixel values ​​of the three-dimensional object model and pixel values ​​of the target object; and adjusting the three-dimensional object model based on the mean square loss value.

[0079] According to an embodiment of the present invention, the volume radiation field can be rendered using a neural radiation field model, and the neural radiation field model can be adjusted using the following formula (4).

[0080]

[0081] in, Represents the mean square error value of pixel color, Represents the number of rays involved in the neural radiation field training process, r represents a single ray, and R represents the set of all sampled rays. represents the final pixel color rendered along ray r, Represents the actual pixel color on ray r.

[0082] According to an embodiment of the present invention, the mean square error value in the above formula (4) can be minimized to adjust the neural radiation field model, thereby adjusting the three-dimensional object model.

[0083] According to an embodiment of the present invention, the above-mentioned visual loss data is calculated by the following formula (5):

[0084]

[0085] in, represents visual loss data, N represents the total number of viewing angles, represents the color value of the perturbed visual perception data at the i-th viewing angle, Represents the color value of visual perception data at the i-th viewing angle.

[0086] Radar loss data is calculated using the following formula (6):

[0087] (6)

[0088] in, represents radar loss data, represents the cross entropy loss function, Represents the point cloud prediction mask of the perturbed radar perception data, Point cloud ground truth mask representing radar perception data.

[0089] According to an embodiment of the present invention, the alignment loss data is calculated by the following formula (7):

[0090] (7)

[0091] in, represents the alignment loss data, n represents the total number of points in the point cloud formed by the candidate adversarial sample, represents the weight of the kth point, represents the feature of the kth point in the candidate adversarial sample, Represents the features of the kth point in the visual perception data.

[0092] The smoothed loss data is calculated using the following formula (8):

[0093] (8)

[0094] in, represents the smoothed loss data, Represents the pixel value at position (i, j) in the visual perception data, Represents the pixel value at position (i+1, j) in the visual perception data, Represents the pixel value at position (i, j+1) in the visual perception data.

[0095] According to an embodiment of the present invention, the above-mentioned alignment loss data is used to ensure the point-to-point correspondence between the generated radar point cloud and the real point cloud. The above-mentioned smoothing loss data adopts a total variation strategy to reduce the difference in adjacent pixel values ​​to ensure the naturalness of the generated target adversarial samples in the physical scene. By jointly optimizing the alignment loss and the smoothing loss, a smoother and more natural target adversarial sample can be generated.

[0096] Figure 3 A flowchart of generating a target adversarial sample according to an embodiment of the present invention is shown.

[0097] First, a data set is constructed by inputting multi-view images, and a volumetric radiation field of a three-dimensional object is generated. Then, sampling is performed along rays, and the volumetric radiation field is rendered to obtain a three-dimensional object model. The visual perception data and radar perception data are perturbed, and the visual loss data, radar loss data, alignment loss data, and smoothing loss data are calculated to adjust the candidate adversarial samples. Finally, the target adversarial sample is obtained to achieve a joint attack on visual and radar sensors, and to ensure robustness and naturalness under physical conditions. Finally, the relevant data can be input into the neural radiation field model to generate a new perspective and perform the next round of iteration.

[0098] Figure 4 A flow chart of an adversarial testing method for an autonomous driving system according to an embodiment of the present invention is shown.

[0099] like Figure 4 As shown, the adversarial sample generation method based on multi-sensor fusion for autonomous driving of this embodiment includes operations S410 to S420.

[0100] In operation S410, a target adversarial sample is obtained.

[0101] Among them, the target adversarial sample is obtained according to the above-mentioned adversarial sample generation method based on multi-sensor fusion of autonomous driving.

[0102] In operation S420, the autonomous driving system is tested using the target adversarial sample to obtain an adversarial test result of the autonomous driving system.

[0103] Among them, the adversarial test results are used to improve the safety of the autonomous driving system.

[0104] Based on the above-mentioned adversarial sample generation method based on multi-sensor fusion for autonomous driving, the present invention also provides an adversarial sample generation device based on multi-sensor fusion for autonomous driving. Figure 5The device is described in detail.

[0105] Figure 5 A structural block diagram of an adversarial sample generation device based on multi-sensor fusion for autonomous driving according to an embodiment of the present invention is shown.

[0106] like Figure 5 As shown, the adversarial sample generation device 500 based on multi-sensor fusion for autonomous driving of this embodiment includes a perception module 510, a disturbance module 520, a loss data calculation module 530 and a target adversarial sample generation module 540.

[0107] The perception module 510 is used to perceive the three-dimensional object model for each of the multiple perspectives to obtain perception data, the three-dimensional object model is used to simulate obstacles encountered during the autonomous driving process, and the perception data includes visual perception data and radar perception data. In one embodiment, the perception module 510 can be used to perform the operation S210 described above, which will not be repeated here.

[0108] The perturbation module 520 is used to obtain visual loss data based on the visual perception data and the perturbed visual perception data obtained by perturbing the visual perception data, and to obtain radar loss data based on the radar perception data and the perturbed radar perception data obtained by perturbing the radar perception data. In one embodiment, the perturbation module 520 can be used to perform the operation S220 described above, which will not be repeated here.

[0109] The loss data calculation module 530 is used to calculate the alignment loss and the smoothing loss based on the candidate adversarial samples and the visual perception data generated by the perturbation perception data of multiple perspectives, respectively, to obtain the alignment loss data and the smoothing loss data. In one embodiment, the loss data calculation module 530 can be used to perform the operation S230 described above, which will not be repeated here.

[0110] The target adversarial sample generation module 540 is used to adjust the candidate adversarial sample based on the visual loss data, the radar loss data, the alignment loss data, and the smoothing loss data to generate a target adversarial sample, and the target adversarial sample is used to simulate the obstacle after the disturbance is added. In one embodiment, the target adversarial sample generation module 540 can be used to perform the operation S240 described above, which will not be repeated here.

[0111] According to an embodiment of the present invention, any multiple modules of the perception module 510, the perturbation module 520, the loss data calculation module 530 and the target adversarial sample generation module 540 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the perception module 510, the perturbation module 520, the loss data calculation module 530 and the target adversarial sample generation module 540 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in an appropriate combination of any of them. Alternatively, at least one of the perception module 510, the perturbation module 520, the loss data calculation module 530 and the target adversarial sample generation module 540 may be at least partially implemented as a computer program module, which may perform corresponding functions when executed.

[0112] Based on the above-mentioned confrontation test method of the autonomous driving system, the present invention also provides a confrontation test device for the autonomous driving system. Figure 6 The device is described in detail.

[0113] Figure 6 A structural block diagram of an adversarial testing device for an autonomous driving system according to an embodiment of the present invention is shown.

[0114] like Figure 6 As shown, the adversarial testing device 600 of the autonomous driving system of this embodiment includes an acquisition module 610 and a testing module 620 .

[0115] The acquisition module 610 is used to acquire a target adversarial sample, which is obtained by using an adversarial sample generation method based on multi-sensor fusion for autonomous driving. In one embodiment, the acquisition module 610 can be used to perform the operation S410 described above, which will not be repeated here.

[0116] The test module 620 is used to test the autonomous driving system using the target adversarial sample to obtain the adversarial test result of the autonomous driving system, and the adversarial test result is used to improve the safety of the autonomous driving system. In one embodiment, the test module 620 can be used to perform the operation S420 described above, which will not be repeated here.

[0117] According to an embodiment of the present invention, any multiple modules in the acquisition module 610 and the test module 620 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the acquisition module 610 and the test module 620 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the acquisition module 610 and the test module 620 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0118] Figure 7 A block diagram of an electronic device suitable for implementing an adversarial sample generation method based on multi-sensor fusion for autonomous driving according to an embodiment of the present invention is shown.

[0119] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 705 to a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0120] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present invention by executing the programs in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present invention by executing the programs stored in the one or more memories.

[0121] According to an embodiment of the present invention, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the input / output (I / O) interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 705 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 705 as needed.

[0122] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0123] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0124] The embodiment of the present invention also includes a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the adversarial sample generation method based on multi-sensor fusion for autonomous driving provided by the embodiment of the present invention.

[0125] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 701. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0126] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0127] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0128] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0129] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0130] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, without departing from the spirit and teachings of the present invention, the features described in the various embodiments of the present invention may be combined and / or combined in various ways. All of these combinations and / or combinations fall within the scope of the present invention.

[0131] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination advantageously. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for generating adversarial samples based on multi-sensor fusion for autonomous driving, characterized in that: The method comprises: For each of the multiple perspectives, the three-dimensional object model is sensed to obtain perception data, where the three-dimensional object model is used to simulate obstacles encountered during the autonomous driving process, and the perception data includes visual perception data and radar perception data; Obtaining visual loss data based on the visual perception data and perturbed visual perception data obtained by perturbing the visual perception data, and obtaining radar loss data based on the radar perception data and perturbed radar perception data obtained by perturbing the radar perception data; Based on the candidate adversarial samples generated by the perturbed perception data of the multiple perspectives and the visual perception data, respectively calculating the alignment loss and the smoothness loss to obtain the alignment loss data and the smoothness loss data; The candidate adversarial sample is adjusted based on the visual loss data, the radar loss data, the alignment loss data, and the smoothness loss data to generate a target adversarial sample, where the target adversarial sample is used to simulate an obstacle after adding disturbance.

2. The method according to claim 1, characterized in that The adjusting the candidate adversarial sample based on the visual loss data, the radar loss data, the alignment loss data, and the smoothing loss data to generate a target adversarial sample comprises: Obtaining a total loss function value based on the visual loss data, the radar loss data, the alignment loss data, and the smoothing loss data; Using a stochastic gradient descent algorithm, solving the target perturbation data that minimizes the total loss function value; The candidate adversarial sample is adjusted based on the target perturbation data to obtain a target adversarial sample.

3. The method according to claim 1, characterized in that The three-dimensional object model is constructed by the following method: Acquire images and sampling attribute data of the target object at the multiple viewing angles, wherein the sampling attribute data includes a capture position of an image capture device corresponding to different viewing angles and a capture angle relative to the target object; Inputting the multiple images and the sampled attribute data into a trained volume radiation field model to obtain the volume radiation field of the target object; The three-dimensional object model is obtained based on the volume radiation field and the sampled attribute data.

4. The method according to claim 3, characterized in that The sampling attribute data also includes transmittance, opacity and color of each of the plurality of sampling points, the plurality of sampling points being distributed on a plurality of rays from the optical center of the image acquisition device to the target object, and obtaining the three-dimensional object model based on the volume radiation field and the sampling attribute data includes: For each ray, obtaining a ray rendering color based on the transmittance of the sampling point, the opacity of the sampling point and the color of the sampling point; The volume radiation field is rendered based on the ray rendering colors of each of the multiple rays to obtain the three-dimensional object model.

5. The method according to claim 4, characterized in that The method further comprises: Using a mean square error loss function, a mean square loss value is obtained based on the pixel values ​​of the three-dimensional object model and the pixel values ​​of the target object; The three-dimensional object model is adjusted based on the mean square loss value.

6. The method according to claim 1, characterized in that The visual loss data is calculated by the following formula: in, represents visual loss data, N represents the total number of viewing angles, represents the color value of the perturbed visual perception data at the i-th viewing angle, Represents the color value of visual perception data at the i-th viewing angle; The radar loss data is calculated by the following formula: in, represents radar loss data, represents the cross entropy loss function, Represents the point cloud prediction mask of the perturbed radar perception data, Point cloud ground truth mask representing radar perception data.

7. The method according to claim 6, characterized in that The alignment loss data is calculated by the following formula: in, represents the alignment loss data, n represents the total number of points in the point cloud formed by the candidate adversarial sample, represents the weight of the kth point, represents the feature of the kth point in the candidate adversarial sample, Represents the features of the kth point in the visual perception data; The smoothed loss data is calculated by the following formula: in, represents the smoothed loss data, represents the pixel value at position (i, j) on the visual perception data, represents the pixel value at the position (i+1, j) on the visual perception data, Represents the pixel value at position (i, j+1) on the visual perception data.

8. A confrontation test method for an autonomous driving system, characterized in that: The method comprises: Obtain a target adversarial sample, wherein the target adversarial sample is obtained by using the method described in any one of claims 1 to 7; The target adversarial sample is used to test the autonomous driving system to obtain an adversarial test result of the autonomous driving system.

9. A device for generating adversarial samples based on multi-sensor fusion for autonomous driving, characterized in that: The device comprises: a perception module, configured to perceive the three-dimensional object model for each of the multiple perspectives to obtain perception data, wherein the three-dimensional object model is used to simulate obstacles encountered during the autonomous driving process, and the perception data includes visual perception data and radar perception data; a perturbation module, configured to obtain visual loss data based on the visual perception data and perturbed visual perception data obtained by perturbing the visual perception data, and to obtain radar loss data based on the radar perception data and perturbed radar perception data obtained by perturbing the radar perception data; A loss data calculation module, configured to calculate alignment loss and smoothing loss based on candidate adversarial samples generated by the perturbation perception data of the multiple perspectives and the visual perception data, respectively, to obtain alignment loss data and smoothing loss data; A target adversarial sample generation module is used to adjust the candidate adversarial sample based on the visual loss data, the radar loss data, the alignment loss data and the smoothing loss data to generate a target adversarial sample, wherein the target adversarial sample is used to simulate an obstacle after adding disturbance.

10. A confrontation test device for an automatic driving system, characterized in that: The device comprises: An acquisition module, used to acquire a target adversarial sample, wherein the target adversarial sample is obtained by using the method described in any one of claims 1 to 7; The testing module is used to test the autonomous driving system using the target adversarial sample to obtain the adversarial test result of the autonomous driving system.

Images