Adversarial sample generation method for six-degree-of-freedom attitude estimation method based on monocular camera

By designing the U6DA method to generate adversarial examples and using SegGradCAM to interpret the segmentation problem, the problem of monocular camera 6DoF pose estimation being susceptible to interference is solved, and the transferability of adversarial examples between different models and the stability of pose estimation are improved.

CN118918408APending Publication Date: 2024-11-08CHANGSHA UNIVERSITY OF SCIENCE AND TECHNOLOGY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410576696.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-05-10
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing 6DoF pose estimation methods based on monocular cameras are susceptible to adversarial examples, which can lead to decreased pose estimation accuracy or failure. Existing adversarial example generation methods are mainly for classification tasks, and their applicability and effectiveness in 6DoF pose estimation still need further research.

Method used

A unified 6-DOF pose estimation attack (U6DA) method is designed. Adversarial examples are generated by segmenting the network, the segmentation problem is explained by SegGradCAM, and adversarial examples that can deceive the deep 6-DOF pose estimation network are generated by lp-norm regularization perturbation.

Benefits of technology

The robustness of the 6-DOF pose estimation method based on a monocular camera is improved, enabling adversarial examples to migrate between different models, effectively enhancing the stability and accuracy of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918408A_ABST
    Figure CN118918408A_ABST
Patent Text Reader

Abstract

The invention discloses an adversarial sample generation method for a six-degree-of-freedom attitude estimation method based on a monocular camera. The adversarial sample generation method comprises the following steps: step 1, giving an RGB image I and a target 3D model M; step 2, estimating a target attitude of the target model; step 3, designing an image segmentation network for six-degree-of-freedom attitude estimation attack and deception of a depth six-degree-of-freedom attitude estimation network; the method has the advantages that the U6DA loss function is designed to guide generation of the adversarial sample, the purpose of the loss function is to transfer the segmentation attention map from the original position, and the confrontation sample is generated through the segmentation network which is designed in advance. Therefore, the method can be used for migrating the black box attack to a mainstream six-degree-of-freedom attitude estimation network. As a PnP solver based on RANSAC may be affected by U6DA, a generated adversarial sample often has aggressiveness to a mainstream six-degree-of-freedom attitude estimation network.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method for generating adversarial samples for a 6-DOF posture estimation method based on a monocular camera. Background Art

[0002] In the field of computer vision, 6-degree-of-freedom (6DoF) pose estimation is a fundamental and critical problem, which involves determining the position and orientation of an object in three-dimensional space. This technology is widely used in many fields such as augmented reality, virtual reality, robot navigation, autonomous driving, and interactive games. With the development of deep learning, 6DoF pose estimation methods based on monocular cameras have attracted the attention of researchers due to their simple equipment and low cost. However, such methods are easily affected by adversarial samples, resulting in a decrease in the accuracy of pose estimation or complete failure. Therefore, studying and developing effective adversarial sample generation methods is of great significance to improve the robustness of 6DoF pose estimation methods based on monocular cameras.

[0003] 6DoF attitude estimation aims to determine the three rotation angles (pitch, yaw and roll) and three translations (displacements along the X, Y and Z axes) of an object relative to the camera. Traditional methods rely on multi-camera systems or sensors such as inertial measurement units (IMUs) to obtain accurate 6DoF information. However, multi-camera systems are expensive and bulky, while sensors such as IMUs have cumulative error problems.

[0004] Using a monocular camera for 6DoF pose estimation usually requires relying on specific algorithm models, such as convolutional neural networks (CNN), recurrent neural networks (RNN), or optical flow methods. These methods extract features from a single or continuous image sequence, and then use the trained model to predict the pose of the object. Although some progress has been made, these methods still face challenges when facing complex scenes and different perspectives.

[0005] Adversarial examples are carefully designed input data that can mislead machine learning models into making incorrect judgments. In 6DoF pose estimation, adversarial examples may cause the model to output incorrect pose information. For example, by adding imperceptible perturbations to the image, the model can be unable to correctly identify the feature points of the object, thereby affecting the accuracy of pose estimation.

[0006] In order to improve the robustness of the model, researchers began to explore how to generate adversarial samples. Existing adversarial sample generation methods mainly include fast gradient sign attack (FGSM), projected gradient descent (PGD) and C&W attack. These methods generate adversarial samples by calculating the gradient of the loss function with respect to the input image, and then making slight adjustments along the gradient direction in the image space. However, these methods are mainly aimed at classification tasks, and their applicability and effectiveness in 6DoF pose estimation still need further research.

[0007] The 6DoF pose estimation method based on monocular camera has shown great potential in practical applications, but its susceptibility to adversarial samples cannot be ignored. Therefore, studying adversarial sample generation technology specifically for such methods can not only evaluate and improve the robustness of existing models, but also provide an important reference for the design of more stable and reliable pose estimation algorithms in the future. With the deepening of adversarial sample research, more technologies are expected to be developed to cope with increasingly complex security threats and ensure the reliability and accuracy of the 6DoF pose estimation method based on monocular camera in various application scenarios. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides an adversarial sample generation method for a 6-DOF posture estimation method based on a monocular camera, comprising the following steps:

[0009] S1: Given an RGB image I and a target 3D model M;

[0010] S2: Estimate the target pose of the target model (target instance);

[0011] S3: Design a segmentation network, namely the Unified 6-DOF Pose Estimation Attack (U6DA) to deceive the deep 6-DOF pose estimation network;

[0012] Input: original RGB image I, trained UNet model F θ (), θ is the network parameter, perturbation shear factor ∈, and the maximum number of iterations M.

[0013] Output: Perturbed RGB image I adv ;

[0014] S4: Generate adversarial samples: Generate adversarial samples through a pre-designed segmentation network. adv =I+∈.

[0015] The target pose in step S2 can be expressed as a rotation R∈SO(3) and a translation T∈R3w.rt camera,

[0016] The specific tasks are described as follows:

[0017] [R|T]=F{[I,M|θ}.

[0018] Preferably: in step 3, the design steps of the segmentation network are:

[0019] 1): Set the initial g0=0,α=∈ / M

[0020] 2): Calculate the attention map

[0021] 3): Get the attention map M sh

[0022] 4): When i<M, execute:

[0023] {Calculate the attention map M 0ri of }

[0024] 5):

[0025] 6):

[0026] 7):

[0027] 8): i=i+1

[0028] 9) At the end

[0029] 10): Return

[0030] Preferably: in step 4, the generation of adversarial samples is as follows:

[0031]

[0032] And through l p -norm to regularize the perturbation.

[0033] The attention map uses SegGradCAM to explain the segmentation problem. SegGradCAM is an extension of GradCAM designed to explain classification models.

[0034] Preferably, the GradCAM calculation method in step 5 is as follows:

[0035]

[0036] in is a set of selected feature maps of interest (the K kernels in the last convolutional layer of a classification network), and y cis the logit of the selected class c. Grad-CAM is the average of the gradients of yc for each feature map A k All N pixels (indexed by u, v) are used to generate a weight Indicates its importance.

[0037] SegGradCAM. SegGradCAM is an extension of GradCAM that can explain the segmentation problem. Its definition is as follows:

[0038] and

[0039] Where M is a set of pixel indices of interest in the output mask.

[0040] let represents the attention map of input I and a specified class t. Push the attention map away from its original position. Therefore, the loss of u6da will be:

[0041] L u6da (I)=||M adv -M sh ||1;

[0042] M adv is the attention map from the adversarial sample, i.e., M sh is the attention map transferred from the clean sample.

[0043] Preferably, the verification means of the method include the following three aspects:

[0044] 1) Set up the experiment: first determine the dataset and model, then determine the evaluation metrics and the specific details of the experiment;

[0045] 2) Sensitivity analysis: determine the shift position and analyze the impact of different shift positions on U6DA: analyze the impact of different segmentation models on transferability;

[0046] 3) Comparison with SOTA cross-task attack;

[0047] 4) Analyze the transferability results. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 : Pose estimation results of clean images and adversarial images generated by U6DA;

[0049] Figure 2 : Visualization of attention maps for clean and adversarial images;

[0050] Figure 3 : The influence of different displacement distances;

[0051] Figure 4 : The impact of different disturbance sizes;

[0052] Figure 5 : Visualization of visible object masks of GDRNet:

[0053] Figure 6 : Visualization of PVNet’s vector field maps (bottom row) and segmentation masks (top row). DETAILED DESCRIPTION

[0054] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are provided for the purpose of illustration and description, and are not intended to be exhaustive or to limit the present invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention, and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

[0055] In this embodiment, a method for generating adversarial samples for a 6-DOF pose estimation method based on a monocular camera is provided. Given an RGB image I, the goal is to generate an adversarial image I adv This can fool deep learning models for 6DOF pose estimation.

[0056] Step 1: Build a surrogate model based on the segmentation network

[0057] The purpose of the deep 6-DOF pose estimation network is to estimate the target pose of a target instance given an RGB image I and a target 3D model M. The 6D pose can be represented as a rotation R∈SO(3) and a translation T∈R3 w.rt camera. The whole task can be described as follows:

[0058] [R|T]=F{[I,M|θ}.

[0059] The following is the algorithm of Unified 6DOF Pose Estimation Attack (U6DA) to deceive the deep 6DOF pose estimation network:

[0060] Input: original RGB image I, trained UNet model F θ (), θ is the network parameter, perturbation shear factor ∈, and the maximum number of iterations M.

[0061] Output: perturbed RGB image Iadv.

[0062] 1: Set the initial g0=0,α=∈ / M

[0063] 2: Calculate the attention map

[0064] 3: Obtain the attention map M sh

[0065] 4: When i < M, execute:

[0066] {Calculate the attention map M 0ri of }

[0067] 5):

[0068] 6):

[0069] 7:

[0070] 8: i = i + 1

[0071] 9: At the end

[0072] 10: Return

[0073] where F is a specific depth 6D pose estimation network and θ are the parameters of the network. The goal is to generate an adversarial example I adv = I + ∈, which is perturbed by a carefully designed perturbation ∈ but misleads the 6 - degree - of - freedom pose estimation network. Usually, the lp - norm is used to regularize the perturbation. Here, the generation of adversarial samples is defined as:

[0074]

[0075] where ADD is the pose accuracy metric and p = ∞. Due to the two - stage scheme of the depth 6 - degree - of - freedom pose estimation network, it is impossible to solve the above problem through the well - known Fast Gradient Sign Method (FGSM) or other white - box attacks. For this reason, the present invention generates adversarial examples through a pre - trained segmentation network. Since the depth 6 - degree - of - freedom pose estimation network relies heavily on the semantic segmentation information of the target instance to locate the object, the present invention uses the classic segmentation network UNet. To pursue the high transferability of U6DA to three mainstream depth 6 - degree - of - freedom pose estimation networks (According to the convention in the literature, the state - of - the - art deep - learning estimation methods for the monocular 6 - degree - of - freedom object pose of monocular RGB images can be roughly divided into three categories: direct methods, keypoint - based methods, and dense - coordinate - based methods), it is necessary to find the common features of these networks.

[0076] Step 2: Design the U6DA loss function to shift the segmentation attention map away from its original position

[0077] This application designs the U6DA loss to guide the generation of adversarial samples. The purpose of the loss is to shift the segmentation attention map away from its original position. The key idea of ​​U6DA is that if general features are shifted away from their original positions, the following RANSAC-based PnP solver will make wrong estimates. The present invention uses SegGradCAM as the attention map because it has impressive performance in explaining segmentation networks. Therefore, GradCAM and SegGradCAM are first introduced as follows:

[0078] and

[0079] in is a set of selected feature maps of interest (the K kernels in the last convolutional layer of a classification network), and y c is the logit of the selected class c. Grad-CAM is the average of the gradients of yc for each feature map A k All N pixels (indexed by u, v) are used to generate a weight Indicates its importance.

[0080] SegGradCAM. SegGradCAM is an extension of GradCAM that can explain the segmentation problem. Its definition is as follows:

[0081] and

[0082] Where M is a set of pixel indices of interest in the output mask.

[0083] let represents the attention map of input I and a specified class t. Push the attention map away from its original position. Therefore, the loss of U6DA will be:

[0084] L u6da (I)=||M adv -M sh ||1;

[0085] M adv is the attention map from the adversarial sample, i.e., M sh is the attention map transferred from the clean sample.

[0086] Step 3: Use the gradient attack scheme to attack the segmentation-based alternative model and generate transferable adversarial samples.

[0087] Adversarial samples are samples generated during the update process guided by the U6DA loss. As shown in the algorithm, it is set to and are updated by the momentum iteration method, and the attack ∞ distance is limited by l. Since the attention map of U6DA has been transferred from the original position, it can be used for black-box attacks on the mainstream 6-DOF pose estimation network.

[0088] Step 4: Migrate the adversarial samples to different 6-DOF pose estimation models to verify the effectiveness of the attack

[0089] Select multiple different 6DoF pose estimation models as migration targets. These models should be structurally different from the original target model to test whether the adversarial examples can work across different model architectures. These models can be from different research teams, use different training datasets, or have different network structures. Make sure that each model has not seen these specific adversarial examples before applying the adversarial examples, that is, these examples have not been used in the training phase of any model. Evaluate the effect of each migrated model. Quantitative evaluation criteria can include success rate (i.e., the proportion of models that are misled and make incorrect predictions) and error size (i.e., the difference between the pose predicted by the model and the actual pose). Record the misclassification rate or error increase of each model when facing the same adversarial example. Analyze the transferability of the adversarial examples based on the evaluation results. If it is found that the adversarial examples can cause a significant increase in error rate or error between different models, it indicates that the adversarial examples have good transferability. At the same time, this also verifies the effectiveness of the adversarial attack. To further verify the transferability, the performance changes of each model before and after the adversarial examples can be compared. In addition, the effects of adversarial examples generated by different attack algorithms on the same model can be compared to evaluate the transfer attack capabilities of different attack methods.

[0090] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field and related fields without creative work should fall within the scope of protection of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention are implemented according to the conventional means in the field unless otherwise specified and limited.

Claims

1. A method for generating adversarial samples for a 6-DOF pose estimation method based on a monocular camera, characterized in that: The steps include: S1: Given an RGB image I and a target 3D model M; S2: Estimate the target pose of the target model; S3: Design a segmentation network, i.e., a unified 6-DOF pose estimation attack to deceive a deep 6-DOF pose estimation network; Input: original RGB image I, trained UNet model F θ (), θ is the network parameter, perturbation shear factor ∈, maximum number of iterations M; Output: Perturbed RGB image I adv ; S4: Generate adversarial samples: Generate adversarial samples through a pre-designed segmentation network. adv =I+∈.

2. According to claim 1, a method for generating adversarial samples for a 6-DOF posture estimation method based on a monocular camera, characterized in that: The target posture in step S2 can be expressed as a rotation R∈SO(3) and a translation T∈R3w.rt camera. The specific task description is as follows: [R|T]=F{[I,M|θ}.

3. The method for generating adversarial samples for a 6-DOF posture estimation method based on a monocular camera according to claim 1, characterized in that: In step 3, the design steps of the segmentation network are: 1): Set the initial g0=0,α=∈ / M 2): Calculate the attention map M ori of 3): Get the attention map M sh 4): When i<M, execute: {Calculate the attention map M ori of } 8): i=i+1 9) At the end 10): Return 4. The method for generating adversarial samples for a 6-DOF posture estimation method based on a monocular camera according to claim 1, characterized in that: In step 4, the generation of adversarial samples is as follows: And through l p -norm to regularize the perturbation.

5. The method for generating adversarial samples for a 6-DOF posture estimation method based on a monocular camera according to claim 3, characterized in that: The attention map uses SegGradCAM to explain the segmentation problem. SegGradCAM is an extension of GradCAM designed to explain classification models.

6. The method for generating adversarial samples for a 6-DOF pose estimation method based on a monocular camera according to claim 5, characterized in that: The GradCAM calculation method in step 5 is as follows: and in is a set of selected feature maps of interest (the K kernels in the last convolutional layer of a classification network), and y c is the logit of the selected class c, and Grad-CAM is the average of the gradients of yc for each feature map A. k All N pixels (indexed by u, v) are used to generate a weight To express its importance; SegGradCAM,SegGradCAM is an extension of GradCAM that can explain the segmentation problem and is defined as follows: and Where M is a set of pixel indices of interest in the output mask, let represents the attention map for input I and a specified class t.

7. The method for generating adversarial samples for a 6-DOF posture estimation method based on a monocular camera according to claim 1, characterized in that: The loss amount in step S5 is: L u6da (I)=||M adv -M sh ||1; M adv is the attention map from the adversarial sample, i.e., M sh is the attention map transferred from the clean sample.

8. A verification method according to any one of claims 1 to 7, characterized in that: It includes the following three aspects: 1) Set up the experiment: first determine the dataset and model, then determine the evaluation metrics and the specific details of the experiment; 2) Sensitivity analysis: determine the mobile position and analyze the impact of different mobile positions on U6DA: analyze the impact of different segmentation models on transferability; 3) Comparison with the most advanced cross-task attacks; 4) Analyze the transferability results.