An adversarial patch generation method and device for monocular depth estimation method

Abstract

The application discloses an anti-patch generation method and device for a monocular depth estimation method, generates a patch-free scene image according to a scene image, a target image and a target mask, generates a benign depth map corresponding to the patch-free scene image based on a target MDE model, determines a target depth map according to the benign depth map, and reversely calculates an anti-patch image and an anti-patch mask according to the target depth map and an overall loss function. According to the application, the target depth map is determined according to the benign depth map, the anti-patch image and the anti-patch mask are reversely calculated in combination with the target depth map and the target MDE model, the attack effect of the anti-patch image can be expanded to the entire target object, and thus the attack effect is greatly improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of artificial intelligence and computer vision, and particularly relates to an adversarial patch generation method and device for a monocular depth estimation method. BACKGROUND

[0002] The major progress of deep learning has driven the development of the field of computer vision, enabling neural networks to perceive and understand visual data with unprecedented accuracy and efficiency. One of the fundamental tasks of computer vision is monocular depth estimation (MDE), which aims to infer depth information of a scene from a single 2D image.

[0003] The depth estimation task is crucial in various applications, including drones, robots, augmented reality, virtual reality, and scene understanding. However, despite the significant progress in MDE, the vulnerability of deep learning models to adversarial attacks remains a major problem. These attacks pose a serious threat to the reliability and robustness of deep learning systems, which can lead to serious malfunctions in safety-critical applications. In the field of autonomous driving, vision-based models are relied upon to enable autonomous driving capabilities. If these vision sensors, which provide critical distance information, are misinterpreted in depth due to adversarial attacks, the consequences can be dire, including vehicle collisions and other serious accidents.

[0004] Existing attacks against MDE can be divided into digital space attacks and physical world attacks. Physical world attacks are more challenging and meaningful than digital space attacks because they need to consider more real-world constraints and factors. Factors such as lighting conditions, object surface reflection characteristics, camera position and angle, etc. will affect the performance of depth estimation models in real scenes, and attackers need to fully consider these factors when designing effective attack methods. Although physical world attacks are more challenging, they have a more direct and serious impact on the performance and reliability of depth estimation models.

[0005] Patch-based adversarial attack methods are currently a common physical attack method, but the attack effect of the patch only affects the overlapping part of the target object and the patch when applied, and cannot cover the entire target object, resulting in poor attack effect. SUMMARY

[0006] The purpose of the present application is to provide an adversarial patch generation method and device for a monocular depth estimation method, which expands the attack effect of the patch to the entire target object and improves the attack effect.

[0007] The present application adopts the following technical solution: an adversarial patch generation method for a monocular depth estimation method, comprising the following steps:

[0008] generate the patch-free scene image according to the scene image, the target object image and the target object mask;

[0009] generate the benign depth map corresponding to the patch-free scene image based on the target MDE model;

[0010] determine the target depth map according to the benign depth map;

[0011] calculate the adversarial patch image and the adversarial patch mask reversely according to the target depth map and the overall loss function.

[0012] Further, the overall loss function comprises a depth loss function;

[0013] The depth loss function is composed of an in-mask loss function and an out-of-mask target object loss function.

[0014] Further, the in-mask loss function is:

[0015] L1=[(D adv -D t )⊙M p ] 2 ,

[0016] Wherein, L1 represents the in-mask loss function, D adv represents the adversarial depth map, D t represents the target depth map, and M p represents the adversarial scene patch mask.

[0017] Further, the out-of-mask target object loss function is:

[0018] L2=exp{|D adv -D t |⊙(M O -M p )},

[0019] Wherein, L2 represents the out-of-mask target object loss function, and M O represents the target object mask.

[0020] Further, the overall loss function further comprises a smoothness loss function, a non-printable score loss function and an area loss function.

[0021] Further, the calculation of the adversarial patch image and the adversarial patch mask reversely according to the target depth map and the overall loss function comprises:

[0022] calculate the adversarial depth map according to the target depth map and the overall loss function;

[0023] calculate the adversarial patch image and the adversarial patch mask based on the adversarial depth map and the target MDE model.

[0024] Further, the adversarial patch image is a rectangle, an irregular quadrilateral, a circle or an ellipse.

[0025] Further, when the adversarial patch image is an irregular quadrilateral, the calculation method of the adversarial patch mask is:

[0026]

[0027] Wherein, m p [i,j] represents the value of the i-th row and j-th column pixel in the adversarial patch mask, t represents the upper edge boundary value, s t represents the slope of the upper edge boundary value, l represents the left edge boundary value, b represents the lower edge boundary value, s b represents the slope of the lower edge boundary value, r represents the right edge boundary value, s l represents the slope of the left edge boundary value, s l represents the slope of the left edge boundary value.

[0028] Further, when the adversarial patch image is an ellipse, the adversarial patch mask is formed based on the circular adversarial patch mask, and the calculation method of the adversarial patch mask is:

[0029]

[0030] Wherein, m p [i,j] represents the value of the i-th row and j-th column pixel in the adversarial patch mask, F B {·} represents a binary function of gradient back propagation, (x, y) represents the circular center coordinates, R represents the circular radius, s x represents the slope of the horizontal center coordinates, s y represents the slope of the vertical center coordinates.

[0031] Another technical solution of the present application: an adversarial patch generation device for monocular depth estimation method, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to realize the method described above.

[0032] The beneficial effects of the present application are: the present application determines the target depth map according to the benign depth map, and then reversely calculates the adversarial patch image and the adversarial patch mask in combination with the target depth map and the target MDE model, so that the attack effect of the adversarial patch image can be expanded to the entire target object, thereby greatly improving the attack effect. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 It is a schematic diagram of an adversarial patch generation method for monocular depth estimation method according to an embodiment of the present application;

[0034] Figure 2A schematic diagram of an irregular quadrilateral mask in an embodiment of the present application.

[0035] Figure 3 A schematic diagram of a circular mask in an embodiment of the present application.

[0036] Figure 4 A schematic diagram of an attack effect in a verification embodiment of the present application.

[0037] Figure 5 A graph showing the influence of patch area on attack effect in a verification embodiment of the present application.

[0038] Figure 6 A graph showing the verification result of the generality of the method in a verification embodiment of the present application.

[0039] Figure 7 A schematic diagram of a physical attack effect in a verification embodiment of the present application. DETAILED DESCRIPTION

[0040] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0041] The present application provides a physical-based adversarial attack method, which optimizes the shape-variable adversarial patch to maximize the interference effect on the output of a monocular depth estimation sensor, thereby revealing the vulnerability of the monocular depth estimation algorithm.

[0042] Specifically, the present application discloses an adversarial patch generation method for a monocular depth estimation method, comprising the following steps: generating a patch-free scene image according to a scene image, a target object image and a target object mask; generating a benign depth map corresponding to the patch-free scene image based on a target MDE model; determining a target depth map according to the benign depth map; and reversely calculating an adversarial patch image and an adversarial patch mask according to the target depth map and an overall loss function.

[0043] The present application determines the target depth map according to the benign depth map, and reversely calculates the adversarial patch image and the adversarial patch mask in combination with the target depth map and the target MDE model, so that the attack effect of the adversarial patch image can be expanded to the entire target object, thereby greatly improving the attack effect.

[0044] As shown in Figure 1 , the patch p and the patch mask m p represent the content to be optimized. The present application takes a car as an example, and inputs are composed of a target car image O, a target car mask m O and a random scene RS. In order to enhance the diversity of the data set and the adaptability of the model, the car image is embedded into various background images in the method of the present application. It should be noted that when the target object is replaced by other things, the above-mentioned car content can be directly replaced.

[0045] As shown in ②, the patch p, the patch mask m p and the target car image O are transformed to generate the adversarial object O adv . As shown in ④, the adversarial scene image X adv is formed by pasting the adversarial object O adv onto the random scene RS. In ⑤, the adversarial scene image X adv is input into the MDE model (i.e., monocular depth estimation model) to obtain the adversarial depth map D adv . Then in ⑥, according to different attack modes, the target depth map of the car is set as D t . Finally, in ⑦, the adversarial loss function is used to optimize the patch p and the patch mask m t based on D adv and the adversarial depth map D p .

[0046] In addition, the target depth map D b may be determined according to the benign depth map D t . The benign depth map D b may be obtained by the following steps.

[0047] As shown in ①, random transformations are applied to the car, including size, position, rotation and brightness. The size of the car mask is adjusted to match the size of the random scene RS in the dataset, forming the scene car mask M O . Next in ③, the target car image O is embedded into the random scene RS through the car mask M O , thereby constructing the benign scene image X b . The benign scene image X b is input into the MDE model to obtain the benign depth map D b .

[0048] In an embodiment of the present application, the adversarial patch image is rectangular, irregular quadrilateral, circular or elliptical.

[0049] In the prior art, few people pay attention to the influence of mask shape and position on attack effect. In order to solve this problem, the present application aims to explore the optimization method of different patch shapes and positions. Specifically, strategies for generating rectangular, irregular quadrilateral and circular patches are developed.

[0050] Regarding the rectangular patch, since the patch mask is usually a binary 0-1 mask, they are not differentiable and cannot be directly optimized. To solve this problem, a mask optimization framework needs to be established to make the patch able to generate gradients and make corresponding optimization. On a mask with a width of w and a height of h (the size of the mask and the vehicle image in the present application), the boundary parameters θ1 = [l, r, t, b] are defined, where 0≤l≤r≤w and 0≤t≤b≤h, as shown in the following figure, l represents the left edge boundary value, that is, the distance between the top-left pixel of the mask region and the left boundary of the mask; r represents the right edge boundary value, that is, the distance between the bottom-right pixel of the mask region and the left boundary of the mask; t represents the upper edge boundary value, that is, the distance between the top-left pixel of the mask region and the upper boundary of the mask; b represents the lower edge boundary value, that is, the distance between the bottom-right pixel of the mask region and the upper boundary of the mask.

[0051] The formula for calculating the rectangular patch mask is as follows:

[0052]

[0053] where m p [i,j] represents the value of the i-th row and j-th column pixel in the mask, and the boundary parameter θ1 = [l, r, t, b] in the formula is the parameter to be optimized.

[0054] However, the above formula can only generate rectangular patches with fixed shapes. Therefore, a new patch generation method is also proposed in the present application, which can generate arbitrary irregular quadrilateral patch shapes, because four slope parameters S = [s l ,s r ,s t ,s b ] are defined for irregular quadrilaterals in the present application, and the slope parameter S is in the range of [-1, 1], s l represents the slope of the left edge boundary value, s r represents the slope of the right edge boundary value, s t represents the slope of the upper edge boundary value, and s b represents the slope of the lower edge boundary value. Thus, when the adversarial patch image is an irregular quadrilateral, the calculation method of the adversarial patch mask is:

[0055]

[0056] where t + s t(j-1) is taken as a whole, compared with t in formula (1), linear changes occur at all pixels except the pixel at (t, l), which actually leads to the value of t in the mask changing linearly with the movement of the pixel, thereby forming a tilted boundary. By introducing the distance from the boundary, the angle of the mask corner point is kept within a reasonable range, which is conducive to optimization. In this embodiment, the boundary parameters θ1=[l,r,t,b] and the slope parameters S=[s l r t b ] are taken as optimization parameters, so that the required irregular quadrilateral can be obtained.

[0057] In addition, the present application also discloses a design method of a circular patch. The circular mask is different from the rectangular mask in that only three parameters are needed for the circular mask: the center coordinates θ2=[x,y] and the radius R. When generating the mask, the value is assigned according to the distance of each pixel to the center. The formula is as follows:

[0058]

[0059] where F B (·) is a binary function specially designed for gradient backpropagation. In the forward transmission process, F B (·) performs a non-differentiable binary operation. Specifically, it rounds each element in the input tensor to the nearest integer value. In the backward transmission process, the backward method only transmits the gradient to the downstream node without any modification. This means that for this method, the gradient propagation remains unchanged, that is, the input gradient is equal to the output gradient.

[0060] As Figure 3 shown, Figure 3 (a) is a circular mask without binary, Figure 3 (b) is a circular mask with binary; and Figure 3 (c) is an elliptical mask with a slope of the circular mask.

[0061] In fact, when an elliptical mask needs to be created, this can be achieved by introducing an offset of the center coordinates based on the pixel position, thereby converting the circular mask into an elliptical mask.

[0062] That is, when the adversarial patch image is elliptical, the adversarial patch mask is changed based on the circular adversarial patch mask, and the calculation method of the adversarial patch mask is:

[0063]

[0064] where m p [i,j] represents the value of the pixel at the i-th row and the j-th column in the adversarial patch mask, and F​​​B {·} denotes the binarization function for gradient backpropagation, (x, y) represents the coordinates of the center of the circle, R represents the radius of the circle, and s x The slope s represents the x-coordinate of the circle's center. y This represents the slope of the ordinate of the circle's center.

[0065] Compared to the traditional formula for generating standard elliptical masks, this embodiment solves the limitation that the foci of the ellipse are always located on the coordinate axes, resulting in more variable mask shapes and positions, thus making it suitable for a variety of usage environments.

[0066] In existing technologies, although the depth of the patch region is effectively modified, the target object outside the patch region is hardly affected. To address this issue, this invention proposes a new loss function, and experiments show that the depth within the patch region is significantly affected, while the change in the region outside the patch is slow.

[0067] The objective of this invention is to synchronize the depth change rate of the target object outside the patch region with the depth change rate of the patch region. This synchronization will allow the depth of the target object outside the patch region to be optimized simultaneously with the depth of the patch region.

[0068] In one embodiment, an adversarial depth map is first calculated based on the target depth map and the overall loss function; then, an adversarial patch image and an adversarial patch mask are calculated based on the adversarial depth map and the target MDE model.

[0069] Specifically, the overall loss function includes a depth loss function; the depth loss function consists of an in-mask loss function and an out-of-mask target loss function. For example, the in-mask loss function is defined as:

[0070] L1 = [(D adv -D t )⊙M p ] 2 (5)

[0071] Among them, M p This represents the adversarial scenario patch mask, which consists of patch mask m. p It is formed by combining with random scene RS.

[0072] In addition, the loss function for targets outside the mask is:

[0073] L2=exp{|D adv -D t |⊙(M O -M p (6)

[0074] Therefore, the depth loss function for depth information is calculated as follows:

[0075] Ldepth = L1+ L2 (7)

[0076] The loss function value is exponential outside the patch boundary and quadratic inside the patch region. The gradient of the exponential function is always greater than that of the quadratic function. With this setting, it is ensured that the gradient outside the target mask region is always greater than that inside the mask region during the backpropagation process (i.e., the inverse operation), so that it approaches the target depth as soon as possible.

[0077] To improve the effectiveness of the patch, the overall loss function also includes a smoothness loss function, a non-printable score loss function, and an area loss function.

[0078] To optimize the content of the patch, L TV For the smoothness loss function, sudden pixel changes in the generated image can be reduced, thereby ensuring the smoothness of the entire image, which is defined as follows:

[0079]

[0080] where (i, j) represents the pixel coordinates inside the patch region, p i,j represents the pixel value of the (i, j) point, p i+1,j represents the pixel value of the (i+1, j) point, p i,j+1 represents the pixel value of the (i, j+1) point.

[0081] In addition, L NPS represents the non-printable score loss function, which is given by the following formula:

[0082]

[0083] where C represents a set of colors that can be realistically reproduced on a printer, and c is a specific color in the set.

[0084] Regarding the limitation of patch size, an area loss function L area is introduced, which adds binary masks to calculate the total area of the patch relative to the target object:

[0085]

[0086] where, represents the area of the (i, j) point in the mask.

[0087] In summary, the overall loss function for patch optimization is:

[0088] L = λ1L depth + λ2L TV + λ3L NPS + λ4L area (11)

[0089] where λ1, λ2, λ3, λ4 are all set hyperparameters.

[0090] Through the above overall loss function, the pixel value of each pixel in the patch and the corresponding mask information can be calculated by minimizing L and back propagation.

[0091] In addition, in order to verify the effectiveness of the method of the present application, the following verification experiments are also carried out.

[0092] 1) Attack model and dataset.

[0093] When selecting the target model, reference was made to the models used in previous studies, and three representative self-supervised MDE models based on CNN were selected: Monodepth2, Depthhints and Manydepth, as well as a self-supervised MDE model based on the Transformer architecture, Lite-Mono. The random scene used for training was derived from various outdoor scene images in the KITTI dataset.

[0094] 2) Data preparation.

[0095] Real-world factors such as lighting conditions, changes in car size and distance were considered. Different brightness levels were applied to the target objects, their sizes were adjusted, slight rotations were added, and real-world cars were simulated. For changes in car size and distance, car images were randomly scaled within a range of 50% to 100% of the original size. Rotations within a range of ±3° were applied to simulate the roll of the car during driving. Random adjustments to the color of the car image were limited to a range of 30% brightness, 10% contrast, and 10% saturation.

[0096] 3) Attack mode.

[0097] Due to the flexibility of the target depth setting of the present application, the attack strategy can be diversified. In addition to the traditional method of maximizing (i.e. moving the target car "further" than the normal state) the distance of the target car, in this case, the depth value of the target area is set to zero.

[0098] Another attack is to make the target car "disappear", i.e. aiming to make the depth of the car the same as the depth of the background. This attack directly sets the depth of the random scene to the target depth.

[0099] At the same time, the car can be made closer by setting the depth of the target car part to 1, which is called "closer" attack.

[0100] In addition, the depth of the target car can be increased or decreased by D tto control the distance of the car. In the physical world, the "farther" attack and the "vanish" attack are more serious threats to the autonomous driving system and have more research significance and value. In the "farther" attack, D t is made smaller. In the "vanish" attack, D t is made equal to the depth of the random scene RS. Moreover, compared with the "closer" attack, these attacks are more difficult to implement. In the "closer" attack, D t is made larger.

[0101] In this embodiment, the depth is normalized depth, that is, find the minimum value and the maximum value difference in the pixel matrix, subtract the minimum value from each pixel value, and then divide by the difference to obtain the normalized depth value of the pixel.

[0102] 4) Evaluation index.

[0103] In this embodiment, five evaluation indexes are mainly used:

[0104] The first evaluation index is the mean squared error (MSE) between the adversarial depth map D adv and the target depth map D t :

[0105]

[0106] The second evaluation index is the mean squared error between the adversarial depth map D adv and the benign depth map D b :

[0107]

[0108] The third evaluation index is the proportion of affected pixels in the target object depth region, denoted as α:

[0109]

[0110] where I(x) is an indicator function, which is only 1 when x is true. Specifically, in order to verify the effectiveness of the results, the pixel depth change value exceeding 0.01 is regarded as a successful attack pixel.

[0111] The fourth evaluation index is the influence degree of the target region disparity map ε disp :

[0112]

[0113] The fifth evaluation index is the average error of the actual depth of the target object region, denoted as ε depth :

[0114]

[0115] wherein, and respectively represent the real depth map of the adversarial sample and the benign sample in the physical world.

[0116] Among the above five parameters, the first parameter MSE t represents the closeness to the target depth map, the smaller the value, the better the attack effect. The rest of the evaluation indexes, the larger the value, the better the performance.

[0117] 5) Main results of the experiment.

[0118] As Figure 4 shown, Figure 4 (a) is a benign sample image, i.e. an image without added patches and its corresponding depth map, in which the average depth measurement of the car is 6.77 meters.

[0119] Figure 4 (b) is an image and its corresponding depth map after adding a patch using the DisM model, which brings an average depth error of 1.45 meters to the car.

[0120] Figure 4 (c) is an image and its corresponding depth map after adding a patch using the StylePatch model, which affects the car with an average depth error of 7.19 meters.

[0121] Figure 4 (d) is an image and its corresponding depth map after adding a patch using the "further" attack method of the present application, which is trained to maximize the distance of the car, and the average depth error reaches 19.17 meters.

[0122] Figure 4 (e) is an image and its corresponding depth map after adding a patch using the "disappearance" attack method of the present application, the goal is to make the car appear to disappear in the depth map, rather than simply missing part of the content of the depth map.

[0123] Figure 4 (f) is an image and its corresponding depth map after adding a patch using the "closer" attack method of the present application, the goal is to make the car closer, and as a result, the average depth of the car is reduced to 5.05 meters.

[0124] In summary, the patches of the prior art only affect the area where the patch overlaps the car. In contrast, the patches of the present application under different attack modes extend their influence to the entire car.

[0125] Additionally, the effectiveness of the attack of the present invention is compared with other models as shown in Table 1 below. To facilitate comparison with previous studies, a rectangular mask that can be autonomously optimized in terms of position and shape is used in this example, and the results of three commonly used models in previous studies are included in the table: APARATE, StylePatch, and DisM. The results of the APARATE model are obtained by replicating the settings in the original paper, while the results of StylePatch are obtained by directly running the source code. In addition, the results of DisM are obtained by integrating the patch set trained by the original authors into the experimental scenario of this example. The best results in the table are highlighted in bold. In this experiment, the Monodepth2 model is attacked, and the patch size is consistent with previous studies, which is 11% of the image area. Analysis of the data shows that the model of the present invention is always superior to previous models in all indicators, and performs outstanding effectiveness in patch-based attacks.

[0126] Table 1

[0127]

[0128] In addition, the effectiveness of the two-stage optimization method using differential evolution and the method of aggregating scattered pixels using Gaussian kernel convolution is compared. These results correspond to the results in the table under “DE” and “Gaussian”.

[0129] First, a patch equal in size to the entire target car is optimized. Second, the contents of the patch are frozen and a few points are selected on it. Then, using cubic spline interpolation, these points are connected to form a smooth closed curve. The values inside the curve are set to 1, while the values outside the curve are set to 0, forming a patch mask. The adversarial loss and patch area loss are used as the fitness function of differential evolution. The differential evolution method is used to optimize and update the positions of these points, aiming to find the best patch shape and position of the attack.

[0130] In the Gaussian method, each pixel of the entire mask m p is considered as an optimization parameter. Subsequently, a gradient-based optimization is first adopted to refine, and then binarized. Then, a Gaussian kernel convolution is applied to blur the edges, promoting the aggregation of scattered pixels. Subsequently, another binarization step is performed to form a cohesive patch.

[0131] In the DE experiments, it was found that optimizing the patch content first and then determining the patch shape significantly reduced the effectiveness of the attack, because the optimization in the first step was directed at the entire target car, and the patch shape optimized in the second step did not match the patch content. In the Gaussian experiments, the difficulty of aggregating noise into a closed map was observed, which indicates that there are challenges in translating digital space noise attacks into physical world implementations. Finally, consistent with all the research results, the method of the present application performs well in achieving the adversarial target.

[0132] Table 2

[0133]

[0134] Table 2 above gives a comparison of patches of different shapes and different attack methods. From the data, it can be seen that in the "vanish" attack, the attack effect of using an arbitrary irregular quadrilateral patch is better than that of a rectangular and circular patch. But in the "further" attack, the attack effects of patches of the three shapes are different. Regarding these results, the present application believes that in the "vanish" attack, the target depth is more complex, and the irregularity of the irregular quadrilateral patch shape allows more diverse depth settings.

[0135] In addition, the present application also studies the effect of patch size on attack effectiveness. Specifically, the patches used for attack are all irregular quadrilaterals and are optimized in shape and position. The target model of the attack is Monodepth2. As shown in Figure 5 , the horizontal coordinate represents the percentage of patch area to target area, and as the patch area increases, the speed of improvement of patch attack effect gradually slows down. When the patch area accounts for about 18% of the mask area, the patch attack can achieve the best balance between area coverage and attack effect.

[0136] The present application also evaluates the universality of the attack framework of the present application for four pre-selected self-supervised MDE models. Specifically, patches of various sizes are used for attack. The selected ε depth (as shown in Figure 6 (a)) and MSE t (as shown in Figure 6 (b)) are used as evaluation indicators for verification, and rectangular patches are used. The experimental results are shown in Figure 6 , compared with the Lite-Mono network based on Transformer, the patches of the present application show better attack performance on three CNN-based MDE models. These results show that the Transformer architecture can better resist patch attacks. Nevertheless, the patches of the present application still successfully attack the Lite-Mono network, confirming the widespread effectiveness of the attack method of the present application.

[0137] The results of the experiments provide strong evidence of the effectiveness of the attacks. Across the various models and evaluation metrics employed, the attacks of the present invention consistently demonstrated the ability to disrupt the integrity and reliability of the target self-supervised MDE system. Such strong performance highlights the universality and efficacy of the attack framework of the present invention, underscoring its potential utility in adversarial environments.

[0138] In addition to this, physical experiments were conducted. In the physical experiments, the target car was systematically placed at a distance of 7 meters from the camera setup. This distance is consistent with the typical braking distance observed at a speed of 25 miles per hour, which is a common operating speed in traditional driving environments. The experimental results were displayed graphically, highlighting the significant effect of the patches in different scenarios. Regardless of whether it was a single-car scenario or a more complex scenario involving multiple cars, the patches of the present invention consistently demonstrated their effectiveness. Notably, both types of simulated attacks resulted in errors exceeding 7 meters, further attesting to the robustness and reliability of the method of the present invention.

[0139] More specifically, Figure 7 (a) illustrates the recognition effect diagram of a single-car experiment without adding patches, Figure 7 (b) illustrates the recognition effect diagram of a single-car experiment with added patches, Figure 7 (c) illustrates the recognition effect diagram of a multi-car experiment without adding patches, Figure 7 (d) illustrates the recognition effect diagram of a multi-car experiment with added patches. According to the experimental results, both types of attacks resulted in errors of the self-supervised MDE system exceeding 7m, regardless of whether it was a single-car experiment or a multi-car experiment.

[0140] In summary, the present invention investigates physical world adversarial patch attacks against MDE, proposes a new attack framework for physical objects, and introduces patches that can be optimized in shape and position based on attack effectiveness. Additionally, different adversarial patches can be used to control the distance of cars in depth maps, and even cause the depth map of the attacked car to disappear into the background. Experimental results demonstrate the effectiveness and robustness of the patches designed using the method of the present invention. Furthermore, the impact of patch size on attack effectiveness is explored, and adversarial patches successfully attack various MDE models and induce multiple types of adversarial depth outputs.

[0141] The present invention discloses an adversarial patch generation device for monocular depth estimation method, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the method described above.

[0142] The application further discloses a computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement the steps in the above method embodiments.

[0143] The application further provides a computer program product, which, when running on a data storage device, enables the data storage device to implement the steps in the above method embodiments.

[0144] The integrated unit module can be stored in a computer readable storage medium if it is realized in the form of a software function unit and sold or used as an independent product. Based on this understanding, the application can implement all or part of the processes in the above embodiments by a computer program to instruct related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can implement the steps in the above method embodiments when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable medium at least includes any entity or device capable of carrying the computer program code to a storage device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal and a software distribution medium. For example, a U disk, a mobile hard disk, a magnetic disk or an optical disk.

[0145] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0146] Those skilled in the art can realize that the units and algorithm steps of each example described in combination with the disclosed embodiments of the application can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0147] In the embodiments of the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the embodiments of the apparatus described above are merely schematic, and the division of the modules or units is merely a logical function division. In actual implementation, another division manner can be adopted, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0148] The modules described as separated components can or can not be physically separated, and the components displayed as modules can or can not be physical units, that is, can be located in one place, or can be distributed on a plurality of network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

Claims

1. An adversarial patch generation method for monocular depth estimation method, characterized in that, The method comprises the following steps: generating a patch-free scene image according to a scene image, a target object image and a target object mask; generating a benign depth map corresponding to the patch-free scene image based on a target MDE model; determining a target depth map according to the benign depth map; calculating an adversarial patch image and an adversarial patch mask reversely according to the target depth map and an overall loss function; the overall loss function comprises a depth loss function; the depth loss function is composed of an in-mask loss function and an out-of-mask target object loss function; the in-mask loss function is: L1 = [(D adv - D t ) ⊙ M p ] 2 , wherein L1represents a mask-in loss function, D adv represents an adversarial depth map, D t represents a target depth map, M p represents an adversarial scene patch mask; the out-of-mask target object loss function is: L2 = exp{D adv - D t |⊙(M O - M p )}, wherein L2 represents the loss function of the target outside the mask, M O represents the target mask; the overall loss function further comprises a smoothness loss function, an unprintable score loss function and an area loss function; calculating an adversarial depth map according to the target depth map and the overall loss function; calculating an adversarial patch image and an adversarial patch mask based on the adversarial depth map and the target MDE model. The adversarial patch image is a rectangle, an irregular quadrilateral, a circle or an ellipse.

2. The method of claim 1, wherein the method of generating adversarial patches for monocular depth estimation is characterized by, When the adversarial patch image is an irregular quadrilateral, the calculation method of the adversarial patch mask is:

3. The method of claim 2, wherein the method of generating adversarial patches for monocular depth estimation is characterized by, When the adversarial patch image is an ellipse, the adversarial patch mask is formed by changing a circular adversarial patch mask, and the calculation method of the adversarial patch mask is: where m p [i,j] represents the value of the pixel in the ith row and jth column of the adversarial patch mask, t represents the upper edge boundary value, s t represents the slope of the upper edge boundary value, l represents the left edge boundary value, b represents the lower edge boundary value, s b represents the slope of the lower edge boundary value, r represents the right edge boundary value, s l represents the slope of the left edge boundary value, s l represents the slope of the left edge boundary value.

4. The method of claim 3, wherein the method of generating adversarial patches for monocular depth estimation is characterized by, The computer program product comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1-4 when executing the computer program. where m p [i,j] represents the value of the i-th row and j-th column pixel in the adversarial patch mask, F B {·} represents a binarization function of gradient backpropagation, (x, y) represents the circular center coordinates, R represents the circular radius, s x represents the slope of the horizontal coordinate of the center, s y represents the slope of the vertical coordinate of the center.

5. An adversarial patch generation device for monocular depth estimation method, characterized in that, ​

Images