Adversarial patch generation method and device for monocular depth estimation method

Abstract

The invention discloses an adversarial patch generation method and device for a monocular depth estimation method. The adversarial patch generation method comprises the following steps: generating a patch-free scene image according to a scene image, a target non-image and a target object mask; generating a benign depth map corresponding to the patch-free scene image based on the target MDE model; determining a target depth map according to the benign depth map; reversely calculating an adversarial patch image and an adversarial patch mask according to the target depth map and the overall loss function; according to the method, the target depth map is determined according to the benign depth map, and the anti-patch image and the anti-patch mask are obtained through reverse calculation in combination with the target depth map and the target MDE model, so that the attack effect of the anti-patch image can be expanded to the whole target, and the attack effect is greatly improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and computer vision technology, and in particular relates to a method and device for generating adversarial patches for a monocular depth estimation method. Background Art

[0002] Significant advances in deep learning have driven the development of the field of computer vision, enabling neural networks to perceive and understand visual data with unprecedented accuracy and efficiency. A fundamental task in computer vision is monocular depth estimation (MDE), which aims to infer the depth information of a scene from a single 2D image.

[0003] The depth estimation task is critical in a variety of applications, including drones, robotics, augmented reality, virtual reality, and scene understanding. However, despite significant progress in MDE, the vulnerability of deep learning models to adversarial attacks remains a major issue. These attacks pose a serious threat to the reliability and robustness of deep learning systems and can cause severe failures in safety-critical applications. In the field of autonomous driving, vision-based models are relied upon to achieve autonomous driving capabilities. If these visual sensors, which provide important distance information, misinterpret depth due to adversarial attacks, the consequences can be dire, including vehicle collisions and other serious accidents.

[0004] Existing attacks against MDE can be divided into digital space attacks and physical world attacks. Physical world attacks are more challenging and meaningful than digital space attacks because they need to consider more real-world constraints and factors. Factors such as lighting conditions, object surface reflection characteristics, camera position and angle will affect the performance of depth estimation models in real scenes, and attackers need to fully consider these factors when designing effective attack methods. Although physical world attacks are more challenging, they also have a more direct and serious impact on the performance and reliability of depth estimation models.

[0005] The patch-based adversarial attack method is a common physical attack method at present. However, when this method is applied, the attack effect of the patch only affects the overlapping part of the target object and the patch, and cannot cover the entire target object, resulting in poor attack effect. Summary of the invention

[0006] The purpose of the present invention is to provide a method and device for generating an adversarial patch for a monocular depth estimation method, so that the attack effect of the patch is extended to the entire target object, thereby improving the attack effect.

[0007] The present invention adopts the following technical solution: a method for generating adversarial patches for a monocular depth estimation method, comprising the following steps:

[0008] Generate a patch-free scene image according to the scene image, the target-free image and the target object mask;

[0009] Generate a benign depth map corresponding to the patch-free scene image based on the target MDE model;

[0010] Determine a target depth map according to the benign depth map;

[0011] The adversarial patch image and adversarial patch mask are reversely calculated based on the target depth map and the overall loss function.

[0012] Furthermore, the overall loss function includes a depth loss function;

[0013] The depth loss function consists of the loss function within the mask and the loss function of the target object outside the mask.

[0014] Furthermore, the in-mask loss function is:

[0015] L1=[(D adv -D t )⊙M p ] 2 ,

[0016] Among them, L1 represents the loss function within the mask, D adv Denotes the adversarial depth map, D t represents the target depth map, M p Denotes the adversarial scene patch mask.

[0017] Furthermore, the loss function of the target outside the mask is:

[0018] L2=exp{|D adv -D t |⊙(M O -M p )},

[0019] Among them, L2 represents the loss function of the target outside the mask, M O Represents the target object mask.

[0020] Furthermore, the overall loss function also includes a smoothness loss function, an unprintable score loss function, and an area loss function.

[0021] Furthermore, reversely calculating the adversarial patch image and the adversarial patch mask according to the target depth map and the overall loss function includes:

[0022] Calculate the adversarial depth map based on the target depth map and the overall loss function;

[0023] The adversarial patch image and adversarial patch mask are calculated based on the adversarial depth map and the target MDE model.

[0024] Furthermore, the adversarial patch image is a rectangle, a trapezoid, a circle, or an ellipse.

[0025] Furthermore, when the adversarial patch image is an irregular quadrilateral, the adversarial patch mask is calculated as:

[0026]

[0027] Among them, m p [i,j] represents the value of the pixel in the i-th row and j-th column in the adversarial patch mask, t represents the upper edge boundary value, and s t represents the slope of the upper edge boundary value, l represents the left edge boundary value, b represents the lower edge boundary value, s b represents the slope of the lower edge boundary value, r represents the right edge boundary value, s l Indicates the slope of the left edge boundary value, s l Indicates the slope of the left edge boundary value.

[0028] Furthermore, when the adversarial patch image is elliptical, the adversarial patch mask is formed based on the circular adversarial patch mask variation, and the adversarial patch mask is calculated as:

[0029]

[0030] Among them, m p [i,j] represents the value of the pixel in the i-th row and j-th column in the adversarial patch mask, F B {·} represents the binary function of gradient back propagation, (x, y) represents the coordinates of the circle center, R represents the circle radius, s x Represents the slope of the horizontal coordinate of the center of the circle, s y Represents the slope of the vertical coordinate of the center of the circle.

[0031] Another technical solution of the present invention: an adversarial patch generation device for a monocular depth estimation method, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the above method is implemented when the processor executes the computer program.

[0032] The beneficial effect of the present invention is: the present invention determines the target depth map based on the benign depth map, and then reversely calculates the adversarial patch image and the adversarial patch mask by combining the target depth map and the target MDE model, so that the attack effect of the adversarial patch image can be extended to the entire target object, thereby greatly improving the attack effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A schematic diagram of a method for generating an adversarial patch for a monocular depth estimation method according to an embodiment of the present invention;

[0034] Figure 2is a schematic diagram of a trapezoidal mask in an embodiment of the present invention;

[0035] Figure 3 Schematic diagram of a circular mask in an embodiment of the present invention.

[0036] Figure 4 This is a schematic diagram of the attack effect in the verification embodiment of the present invention;

[0037] Figure 5 This is a diagram showing the effect of the patch area on the attack effect in the embodiment of the present invention;

[0038] Figure 6 This is a verification result diagram of the universality of the method in the embodiment of the present invention;

[0039] Figure 7 This is a schematic diagram of the physical attack effect in the verification embodiment of the present invention. DETAILED DESCRIPTION

[0040] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] The present invention provides a physics-based adversarial attack method, which maximizes the interference effect on the output of a monocular depth estimation sensor through shape-variable adversarial patch optimization, thereby revealing the vulnerability of the monocular depth estimation algorithm.

[0042] Specifically, the present invention discloses an adversarial patch generation method for a monocular depth estimation method, comprising the following steps: generating a patchless scene image based on a scene image, a target image and a target object mask; generating a benign depth map corresponding to the patchless scene image based on a target MDE model; determining a target depth map based on the benign depth map; and reversely calculating an adversarial patch image and an adversarial patch mask based on the target depth map and an overall loss function.

[0043] The present invention determines the target depth map according to the benign depth map, and then reversely calculates the adversarial patch image and adversarial patch mask by combining the target depth map and the target MDE model, so that the attack effect of the adversarial patch image can be extended to the entire target object, thereby greatly improving the attack effect.

[0044] like Figure 1 As shown, the patch p and the patch mask m p Represents the content to be optimized. Taking a car as an example, the present invention inputs a target car image O, a target car mask m O and random scene RS. In order to enhance the diversity of the data set and the adaptability of the model, the method of the present invention embeds the car image into various background images. It should be noted that when the target object is other things, the above car content can be directly replaced.

[0045] As shown in ②, patch p and patch mask m p and transform the target car image O to generate the adversarial object O adv As shown in ④, the adversarial scene image X adv is to transform the antagonist O adv Pasted onto the random scene RS. In ⑤, the adversarial scene image X adv Input into the MDE model (i.e., monocular depth estimation model) to obtain the adversarial depth map D adv Then proceed to step ⑥, and set the target depth map of the car to D according to different attack modes. t Finally, in ⑦, the adversarial loss function is used based on D t and adversarial depth map D adv To reversely optimize the patch p and the patch mask m p .

[0046] In addition, according to the benign depth map D b To determine the target depth map D t About the benign depth map D b It can be obtained through the following steps.

[0047] As shown in ①, random transformations are applied to the car, including size, position, rotation, and brightness. The size of the car mask is adjusted to match the size of the random scene RS in the dataset to form the scene car mask M O Next, in ③, the target car image O is transformed through the car mask M O Embedded into the random scene RS to construct the benign scene image X b . The benign scene image X b Input into the MDE model to obtain the benign depth map D b .

[0048] In the embodiment of the present invention, the adversarial patch image is in the shape of a rectangle, a trapezoid, a circle or an ellipse.

[0049] In the prior art, few people pay attention to the impact of mask shape and position on the attack effect. To solve this problem, the present invention aims to explore the optimization method of different patch shapes and positions. Specifically, strategies for generating rectangular, irregular quadrilateral and circular patches are formulated.

[0050] Regarding rectangular patches, since patch masks are usually binary 0-1 masks, they are not differentiable and cannot be directly optimized. In order to solve this problem, it is necessary to establish a mask optimization framework so that patches can generate gradients and be optimized accordingly. On a mask with a width of w and a height of h (the size of the mask and the vehicle image is the same in the present invention), a boundary parameter θ1 = [l, r, t, b] is defined, where 0≤l≤r≤w and 0≤t≤b≤h, see the figure below, l represents the left edge boundary value, that is, the distance between the upper left pixel of the mask area and the left boundary of the mask; r represents the right edge boundary value, that is, the distance between the lower right pixel of the mask area and the left boundary of the mask; t represents the upper edge boundary value, that is, the distance between the upper left pixel of the mask area and the upper boundary of the mask; b represents the lower edge boundary value, that is, the distance between the lower right pixel of the mask area and the upper boundary of the mask.

[0051] The formula for calculating a rectangular patch mask is as follows:

[0052]

[0053] Among them, m p [i,j] represents the value of the pixel in the i-th row and j-th column in the mask, where the boundary parameter θ1=[l,r,t,b] is the parameter that needs to be optimized.

[0054] However, the above formula can only generate rectangular patches with fixed shapes. Therefore, the present invention also proposes a new patch generation method that can generate any irregular trapezoidal patch shape. This is because the present invention defines four slope parameters S = [s l ,s r ,s t ,s b ], the slope parameter S is in the range of [-1,1], s l Indicates the slope of the left edge boundary value, s r Represents the slope of the right edge boundary value, s t Represents the slope of the upper edge boundary value, s b Represents the slope of the lower edge boundary value. Therefore, when the adversarial patch image is an irregular quadrilateral, the calculation method of the adversarial patch mask is:

[0055]

[0056] Among them, t+s t(j-1) is taken as an example and regarded as a whole. Compared with t in formula (1), except for the pixel at (t, l), linear changes will occur at all other pixels, which actually causes the value of t in the mask to change linearly with the movement of the pixel, thus forming a tilted boundary. By introducing the distance from the boundary, the angle of the mask corner point is kept within a reasonable range, which is conducive to optimization. In this embodiment, the boundary parameter θ1 = [l, r, t, b] and the slope parameter S = [s l ,s r ,s t ,s b ] as the optimization parameter, so that the desired irregular trapezoid can be obtained.

[0057] In addition, the present invention also discloses a design method for a circular patch. The difference between a circular mask and a rectangular mask is that a circular mask only requires three parameters: the coordinates of the center of the circle θ2 = [x, y] and the radius R. When generating a mask, a value is assigned based on the distance from each pixel to the center of the circle. The formula is as follows:

[0058]

[0059] Among them, F B (·) is a binarization function designed for gradient back propagation. In the forward propagation process, F B (·) Performs a non-differentiable binarization operation. Specifically, it rounds each element in the input tensor to the nearest integer value. During the backward pass, the backward method simply passes the gradients to the downstream nodes without any modification. This means that for this method, the gradient propagation remains unchanged, that is, the input gradient is equal to the output gradient.

[0060] like Figure 3 As shown, Figure 3 (a) is a circular mask without binary, Figure 3 (b) is a binary circular mask; Figure 3 (c) is an elliptical mask with the circular mask slope added.

[0061] In fact, when one needs to create an elliptical mask, one can do this by introducing an offset of the center coordinates based on the pixel position, thus converting a circular mask into an elliptical mask.

[0062] That is, when the adversarial patch image is elliptical, the adversarial patch mask is formed based on the circular adversarial patch mask variation, and the adversarial patch mask is calculated as:

[0063]

[0064] Among them, m p [i,j] represents the value of the pixel in the i-th row and j-th column in the adversarial patch mask, FB {·} represents the binary function of gradient back propagation, (x, y) represents the coordinates of the circle center, R represents the circle radius, s x Represents the slope of the horizontal coordinate of the center of the circle, s y Represents the slope of the vertical coordinate of the center of the circle.

[0065] Compared with the traditional formula for generating a standard elliptical mask, this embodiment solves the limitation that the focus of the ellipse is always located on the coordinate axis, thereby making the shape and position of the mask more varied and thus applicable to a variety of usage environments.

[0066] In the prior art, although the depth of the patch area is effectively modified, the target outside the patch area is almost unaffected. To solve this problem, the present invention proposes a new loss function and conducts experiments, and observes that the depth within the patch area has a significant impact, while the area outside the patch changes slowly.

[0067] The present invention aims to synchronize the depth change rate of the target object outside the patch area with the depth change rate of the patch area. This synchronization will enable the depth of the target object outside the patch area to be optimized at the same time as the depth of the patch area.

[0068] In one embodiment, an adversarial depth map is first calculated based on the target depth map and the overall loss function; and then an adversarial patch image and an adversarial patch mask are calculated based on the adversarial depth map and the target MDE model.

[0069] Specifically, the overall loss function includes a depth loss function; the depth loss function is composed of an in-mask loss function and an out-of-mask target loss function. Exemplarily, the in-mask loss function is defined as:

[0070] L1=[(D adv -D t )⊙M p ] 2 (5)

[0071] Among them, M p represents the adversarial scene patch mask, which is composed of the patch mask m p It is formed by combining with random scene RS.

[0072] In addition, the loss function of the target outside the mask is:

[0073] L2=exp{|D adv -D t |⊙(M O -M p )} (6)

[0074] Therefore, the depth loss function of the depth information is calculated as follows:

[0075] Ldepth =L1+L2 (7)

[0076] The loss function value is an exponential loss outside the patch boundary, and a quadratic function inside the patch area. The gradient of the exponential function is always greater than the gradient of the quadratic function. This setting ensures that during the back propagation process (i.e., inverse operation), the gradient outside the target object mask area is always greater than the gradient inside the mask area, so that it approaches the target depth as quickly as possible.

[0077] To improve the effectiveness of patches, the overall loss function also includes smoothness loss function, unprintable score loss function and area loss function.

[0078] In order to optimize the patch content, L TV is a smoothness loss function that mitigates sudden pixel changes in the generated image, thereby ensuring the smoothness of the entire image, and is defined as follows:

[0079]

[0080] Among them, (i, j) represents the pixel coordinates within the patch area, p i,j Represents the pixel value of point (i,j), p i+1,j Represents the pixel value of point (i+1,j), p i,j+1 Represents the pixel value of point (i,j+1).

[0081] In addition, L NPS represents the unprintable score loss function, which is given by the following formula:

[0082]

[0083] Wherein, C represents a set of colors that can be truly reproduced on a printer, and c is a specific color of C in the set.

[0084] Regarding the limitation of patch size, an area loss function L is introduced area , which adds the binary masks to calculate the total area of ​​the patch relative to the object:

[0085]

[0086] in, Represents the area of ​​point (i,j) in the mask.

[0087] In summary, the overall loss function of patch optimization is:

[0088] L=λ1L depth +λ2L TV +λ3L NPS +λ4L area (11)

[0089] Among them, λ1, λ2, λ3, and λ4 are all set hyperparameters.

[0090] Through the above overall loss function, L is minimized and back propagation is performed to calculate the pixel value of each pixel in the patch and the corresponding mask information.

[0091] In addition, in order to verify the effectiveness of the method of the present invention, the following verification experiments were carried out.

[0092] 1) Attack model and dataset.

[0093] When selecting the target model, we referred to the models used in previous studies and selected three representative CNN-based self-supervised MDE models: Monodepth2, Depthhints, and Manydepth, as well as the self-supervised MDE model Lite-Mono based on the Transformer architecture. The random scenes used in training are derived from various outdoor scene images in the KITTI dataset.

[0094] 2) Data preparation.

[0095] Real-world factors such as lighting conditions, car size, and distance variations are taken into account. Different brightness levels are applied to the objects, their sizes are adjusted, slight rotations are added, and real-world cars are simulated. For car size and distance variations, car images are randomly scaled between 50% and 100% of the original size. Rotations are applied within the ±3° range to simulate the roll of the car while driving. Random adjustments to the color of the car images are limited to 30% brightness, 10% contrast, and 10% saturation.

[0096] 3) Attack mode.

[0097] Since the target depth of the present invention is set flexibly, the attack strategy can be diversified. In addition to the traditional method of maximizing (i.e., moving the car "further" than normal) the distance of the target car, in this case, the depth value of the target area is set to zero.

[0098] Another attack is to make the target car “disappear”, that is, to make the depth of the car the same as the background. This attack directly sets the random scene depth to the target depth.

[0099] At the same time, the car can be made closer by setting the depth of the target car part to 1, which is called the "closer" attack.

[0100] In addition, by increasing or decreasing D tTo control the distance of the car. In the physical world, "further" attack and "disappearance" attack are more serious threats to the autonomous driving system and have greater research significance and value. t Smaller. When attacking with "disappearance", you can let D t is equal to the depth of the random scene RS. Moreover, these attacks are more difficult to implement than the "closer" attacks. In the "closer" attack, D t Bigger.

[0101] In this embodiment, the depths are all normalized depths, that is, the difference between the minimum and maximum values ​​in the pixel matrix is ​​found, the minimum value is subtracted from each pixel value, and then the result is divided by the difference to obtain the normalized depth value of the pixel.

[0102] 4) Evaluation indicators.

[0103] This embodiment mainly uses five evaluation indicators:

[0104] The first evaluation metric, adversarial depth map D adv With the target depth map D t Mean Squared Error (MSE) between:

[0105]

[0106] The second evaluation metric, adversarial depth map D adv With the benign depth map D b The mean square error between:

[0107]

[0108] The third evaluation indicator is the ratio of affected pixels in the depth area of ​​the target object α:

[0109]

[0110] Where I(x) is an indicator function, and the count is 1 only when x is true. Specifically, in order to verify the validity of the results, the pixel depth change value exceeding 0.01 is regarded as a successfully attacked pixel.

[0111] The fourth evaluation indicator is the disparity map ε of the target area disp Degree of impact:

[0112]

[0113] The fifth evaluation indicator is the average error of the actual depth of the target area ε depth :

[0114]

[0115] in, and Represent the real depth maps of adversarial samples and benign samples in the physical world respectively.

[0116] Among the five parameters mentioned above, the first parameter MSE t Indicates the closeness to the target depth map. The smaller the value, the better the attack effect. For the other evaluation indicators, the larger the value, the better the performance.

[0117] 5) Main results of the experiment.

[0118] like Figure 4 As shown, Figure 4 (a) is a benign sample image, i.e., an image without patches added and its corresponding depth map, where the average depth of the car is measured to be 6.77 meters.

[0119] Figure 4 (b) The image and corresponding depth map after adding the patch using the DisM model. This patch brings an average depth error of 1.45 meters to the car.

[0120] Figure 4 (c) The image and corresponding depth map after adding the patch using the StylePatch model. This patch affects the average depth error of the car by 7.19 meters.

[0121] Figure 4 (d) Image and corresponding depth map after adding a patch for the “further” attack using the method of the present invention, where the patch is trained to maximize the distance to the car, with an average depth error of 19.17 meters.

[0122] Figure 4 (e) The image and the corresponding depth map after adding the patch using the "disappearing" attack method of the method of the present invention, the goal is to make the car appear to disappear in the depth map, rather than the depth map simply lacking part of its content.

[0123] Figure 4 (f) The image and corresponding depth map after adding patches using the “closer” attack method of the method of the present invention, with the goal of making the car closer. As a result, the average depth of the car is reduced to 5.05 meters.

[0124] In summary, the patch of the prior art only affects the area where the patch overlaps with the car. In contrast, the patch of the present invention under different attack modes extends its impact to the entire car.

[0125] In addition, as shown in Table 1 below, the effectiveness of the attack of the present invention is compared with other models. In order to facilitate comparison with previous studies, a rectangular mask that can be autonomously optimized in terms of position and shape is used in this embodiment, and the table contains the results of three commonly used models in previous studies: APARATE, StylePatch, and DisM. The results of the APARATE model were obtained by copying the settings in the original paper, while the results of StylePatch were obtained by directly running the source code. In addition, the results of DisM were obtained by integrating the patches trained by the original authors into the experimental scenario of this embodiment. The best results in the table are highlighted in bold. In this experiment, the Monodepth2 model was attacked, and the patch size was consistent with previous studies, which was 11% of the image area. Analysis of the data shows that the model of the present invention consistently outperforms previous models in all indicators and shows excellent effectiveness in patch-based attacks.

[0126] Table 1

[0127]

[0128] In addition, the effectiveness of a two-stage optimization method using differential evolution and a method that uses Gaussian kernel convolution to aggregate scattered pixels are compared. These results correspond to the results in "DE" and "Gaussian" in the table.

[0129] First, a patch equal to the size of the entire target car is optimized. Second, the content of the patch is frozen and several points are selected on it. Then, using cubic spline interpolation, these points are connected to form a smooth closed curve. The values ​​inside the curve are set to 1, while the values ​​outside the curve are set to 0, forming a patch mask. Adversarial loss and patch area loss are used as the fitness function of differential evolution. The differential evolution method is used to optimize and update the position of these points, aiming to find the best patch shape and position for the attack.

[0130] In the Gaussian method, the entire mask m p Each pixel of is considered as an optimization parameter. Subsequently, gradient-based optimization is first employed for refinement and then binarization is performed. Then, Gaussian kernel convolution is applied to blur the edges and promote the aggregation of scattered pixels. Another binarization step is then performed to form a cohesive patch.

[0131] In DE experiments, it is found that optimizing the patch content first and then determining the patch shape significantly reduces the effectiveness of the attack, because the optimization in the first step targets the entire target car, while the patch shape optimized in the second step does not match the patch content. In Gaussian experiments, difficulties in aggregating noise into closed graphs are observed, which indicates the challenges of translating digital spatial noise attacks to physical world implementations. Ultimately, consistent with all research results, the proposed method performs well in achieving adversarial goals.

[0132] Table 2

[0133]

[0134] Table 2 above gives a comparison of patches of different shapes and different attack methods. It can be seen from the data that in the "disappearing" attack, the attack effect of using any irregular trapezoidal patch is better than that of rectangular and circular patches. However, in the "further" attack, the attack effects of the three shapes of patches are different. Regarding these results, the present invention believes that in the "disappearing" attack, the target depth is more complex, and the irregularity of the irregular trapezoidal patch shape allows for more diverse depth settings.

[0135] In addition, the present invention also studies the influence of patch size on the attack effect. Specifically, the patches used for attack are all irregular quadrilaterals, and their shapes and positions are optimized. The target model of the attack is Monodepth2. Figure 5 As shown in the figure, the horizontal axis represents the percentage of patch area to the target area. As the patch area increases, the improvement rate of patch attack effect gradually slows down. When the patch area accounts for about 18% of the mask area, patch attack can achieve the best balance between area coverage and attack effect.

[0136] We also evaluate the generalizability of our attack framework against four pre-selected self-supervised MDE models. Specifically, we attack using patches of various sizes. depth (like Figure 6 (a)) and MSE t (like Figure 6 (b)) is used as the evaluation metric for verification, and a rectangular patch is used. The experimental results are shown in Figure 6 As shown in Figure 2, the patch of the present invention shows better attack performance on three CNN-based MDE models compared to the Transformer-based Lite-Mono network. These results show that the Transformer architecture can better resist patch attacks. Nevertheless, the patch of the present invention successfully attacks the Lite-Mono network, confirming the broad effectiveness of the attack method of the present invention.

[0137] The results of the experiments provide strong evidence of the effectiveness of a wide range of attacks. Across the various models and evaluation metrics employed, the proposed attack consistently demonstrates the ability to undermine the integrity and reliability of the target self-supervised MDE system. Such strong performance highlights the universality and effectiveness of the proposed attack framework, highlighting its potential utility in adversarial settings.

[0138] In addition, physical experiments were conducted. In the physical experiments, the target car was systematically placed at a distance of 7 meters from the camera setup. This distance is consistent with the typical braking distance observed at a speed of 25 mph, which is a common operating speed in traditional driving environments. The experimental results are displayed graphically, highlighting the significant effect of the patch in different scenarios. Whether in single-car scenarios or more complex scenarios involving multiple cars, the patch of the present invention consistently demonstrates its effectiveness. Notably, both types of simulated attacks produced errors of more than 7 meters, further confirming the robustness and reliability of the present method.

[0139] More specifically, Figure 7 (a) shows the recognition effect of the single-bike experiment without adding patches. Figure 7 (b) shows the recognition effect diagram of the single-bike experiment with the patch added. Figure 7 (c) shows the recognition effect diagram of the multi-vehicle experiment without adding patches. Figure 7 (d) shows the recognition effect diagram when the multi-vehicle experiment is performed and the patch is added. According to the actual test results, both types of attacks cause the error of the self-supervised MDE system to exceed 7m, regardless of whether it is a single-vehicle experiment or a multi-vehicle experiment.

[0140] In summary, the present invention studies physical world adversarial patch attacks against MDE, proposes a new attack framework for physical objects, and introduces patches whose shapes and positions can be optimized according to the attack effect. In addition, different adversarial patches can be used to control the distance of the car in the depth map, and even cause the depth map of the attacked car to disappear in the background. The experimental results demonstrate the effectiveness and robustness of the patches designed using the method of the present invention. In addition, the effect of patch size on the attack effect is also explored, and the adversarial patches successfully attack various MDE models and induce multiple types of adversarial deep outputs.

[0141] The present invention discloses an adversarial patch generation device for a monocular depth estimation method, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the method is implemented when the processor executes the computer program.

[0142] The present invention also discloses an embodiment that provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0143] The present invention also provides a computer program product. When the computer program product runs on a data storage device, the data storage device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0144] If the integrated unit module is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to a storage device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0145] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0146] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed in the present invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0147] In the embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0148] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

Claims

1. A method for generating adversarial patches for a monocular depth estimation method, characterized in that: The following steps are involved: Generate a patch-free scene image according to the scene image, the target-free image and the target object mask; Generate a benign depth map corresponding to the patch-free scene image based on the target MDE model; Determine a target depth map according to the benign depth map; The adversarial patch image and the adversarial patch mask are reversely calculated based on the target depth map and the overall loss function.

2. The method for generating an adversarial patch for a monocular depth estimation method according to claim 1, wherein: The overall loss function includes a depth loss function; The depth loss function consists of an in-mask loss function and an out-of-mask target loss function.

3. The method for generating an adversarial patch for a monocular depth estimation method according to claim 2, wherein: The intra-mask loss function is: L1=[(Dadv-Dt)⊙Mp] 2 , Among them, L1 represents the loss function within the mask, D adv Denotes the adversarial depth map, D t represents the target depth map, M p Represents the adversarial scene patch mask.

4. The method for generating an adversarial patch for a monocular depth estimation method according to claim 3, wherein: The loss function of the target object outside the mask is: L2=exp{|D adv -D t |⊙(M O -M p )}, Among them, L2 represents the loss function of the target outside the mask, M O Represents the target object mask.

5. The method for generating an adversarial patch for a monocular depth estimation method according to claim 2, wherein: The overall loss function also includes a smoothness loss function, an unprintable fraction loss function and an area loss function.

6. The method for generating an adversarial patch for a monocular depth estimation method according to claim 2 or 5, characterized in that: Reversely calculating the adversarial patch image and the adversarial patch mask according to the target depth map and the overall loss function includes: Calculating an adversarial depth map based on the target depth map and the overall loss function; An adversarial patch image and an adversarial patch mask are calculated based on the adversarial depth map and the target MDE model.

7. The method for generating an adversarial patch for a monocular depth estimation method according to claim 6, wherein: The adversarial patch image is in the shape of a rectangle, an irregular trapezoid, a circle or an ellipse.

8. The method for generating an adversarial patch for a monocular depth estimation method according to claim 7, wherein: When the adversarial patch image is an irregular quadrilateral, the adversarial patch mask is calculated as follows: Among them, m p [i,j] represents the value of the pixel in the i-th row and j-th column in the adversarial patch mask, t represents the upper edge boundary value, and s t represents the slope of the upper edge boundary value, l represents the left edge boundary value, b represents the lower edge boundary value, s b represents the slope of the lower edge boundary value, r represents the right edge boundary value, s l Indicates the slope of the left edge boundary value, s l Indicates the slope of the left edge boundary value.

9. The method for generating an adversarial patch for a monocular depth estimation method according to claim 7, wherein: When the adversarial patch image is elliptical, the adversarial patch mask is formed based on a circular adversarial patch mask variation, and the adversarial patch mask is calculated as follows: Among them, m p [i,j] represents the value of the pixel in the i-th row and j-th column in the adversarial patch mask, F B {·} represents the binary function of gradient back propagation, (x, y) represents the coordinates of the circle center, R represents the circle radius, s x Represents the slope of the horizontal coordinate of the center of the circle, s y Represents the slope of the vertical coordinate of the center of the circle.

10. An adversarial patch generation device for a monocular depth estimation method, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 9 when executing the computer program.

Images