Adversarial sample generation method, device and equipment and medium suitable for target tracking task

Abstract

The application relates to an adversarial sample generation method, device and equipment and medium suitable for a target tracking task. The method comprises the following steps: obtaining a target three-dimensional model of a target to be tracked, a set of target and camera operation rendering parameters and an original digital adversarial sample image, performing color correction on the original digital adversarial sample image to obtain a physical adversarial sample image, fusing the physical adversarial sample image and the target three-dimensional model through a neural renderer, combining target and camera pose information to generate a two-dimensional adversarial sample video; performing binarization on the two-dimensional adversarial sample video to obtain a target mask video, generating a structured prompt word, inputting a visual language large model to generate a reconstructed video, fusing the three to obtain a real video containing an adversarial sample, iteratively optimizing the adversarial sample based on a joint loss function, and updating an original texture image to generate a physical adversarial sample. The method can improve the physical authenticity and scene adaptability of the adversarial sample, and effectively reduce the positioning accuracy of the target tracking task in a real scene.

Description

Technical Field

[0001] This application relates to the fields of computer vision and deep learning technology, and in particular to a method, apparatus, device and medium for generating adversarial examples adapted to target tracking tasks. Background Technology

[0002] Single-target trackers based on deep convolutional neural networks have been widely applied in intelligent traffic monitoring, UAV environmental perception, and android visual perception due to their high positioning accuracy, high operating efficiency, and strong robustness. However, deep learning models are susceptible to carefully crafted adversarial examples, which can lead to erroneous tracking predictions. In application scenarios, such perceptual biases can easily cause equipment malfunctions and distorted monitoring results, affecting the practical safety and reliability of single-target tracking systems. Therefore, researching physical adversarial example generation methods for single-target trackers in specific scenarios is of great significance for discovering model vulnerabilities and improving the stability of the system in practical applications.

[0003] Existing methods for generating adversarial examples for single-target trackers typically involve first optimizing perturbations in the digital domain, then creating physical entities through printing, spraying, or other methods to reduce the localization accuracy of target trackers in physical scenes. Based on the range of perturbation application, these methods can be categorized into patch-based adversarial examples and full-coverage adversarial examples. Patch-based adversarial examples, by adding printable perturbations to localized areas of the target, offer advantages such as low computational and deployment costs. However, their effectiveness depends on the geometric visibility of the perturbation area, and they are prone to failure when the viewing angle or target pose changes, exhibiting poor stability in multi-view scenarios. Full-coverage adversarial examples, by modeling the imaging process and optimizing the overall target texture, improve the localization accuracy of target trackers across multiple views, becoming the mainstream direction for generating physical adversarial examples.

[0004] Current technologies still have significant shortcomings: First, generating high-quality, full-coverage adversarial examples requires accurate target and camera pose information. Such data is costly to collect in real-world scenarios and difficult to acquire on a large scale. Existing technologies use virtual simulation platforms to generate synthetic data that differs from the real environment in terms of materials, lighting, and imaging noise, limiting the ability of adversarial examples to reduce the accuracy of target trackers in real-world scenarios. Second, traditional loss functions used for adversarial example optimization (such as feature loss based on Gram matrices) only count the correlation of feature channels, ignoring feature spatial distribution information. They are also susceptible to background noise and cannot accurately measure the semantic similarity of target regions. This results in insufficient adaptability to single-target tracking tasks, further reducing the actual effectiveness of physical adversarial examples in real-world scenarios.

[0005] Therefore, how to reduce the cost of data acquisition in various scenarios while ensuring data authenticity, and how to design adversarial example optimization methods adapted to single-target tracking tasks to improve the effectiveness of adversarial examples in reducing the localization accuracy of target trackers in multi-view scenarios, have become urgent technical problems to be solved in this field. Summary of the Invention

[0006] Therefore, it is necessary to provide an adversarial sample generation method, apparatus, device, and medium that can effectively improve the effectiveness and robustness of physical adversarial samples in multi-view real-world scenarios and is adapted to target tracking tasks, in order to address the above-mentioned technical problems.

[0007] An adversarial example generation method adapted for target tracking tasks, the method comprising:

[0008] Acquire the 3D model of the target to be tracked, a set of rendering parameters for the target and camera operation, and an original digital adversarial sample image; The original digital adversarial sample image is color-corrected using a physical color mapping network to obtain a physical adversarial sample image; The adversarial sample image is fused with the target 3D model using a neural renderer, and combined with the 3D coordinate point information and Euler angle information of the target to be tracked and the camera in the rendering parameters to generate a 2D adversarial sample video. The two-dimensional adversarial sample video is binarized to obtain a target mask video that distinguishes between foreground targets and background regions. A mask-driven prompt word generation model is used to generate structured prompt words containing environmental semantics and motion trajectory descriptions from the target mask video. The structured prompts and the target mask video are input into a large visual language model to generate a reconstructed video that conforms to the preset semantic description, motion trajectory and matches the distribution of the real scene. The two-dimensional adversarial sample video, the reconstructed video and the target mask video are fused to obtain a real video containing adversarial samples. Based on the real video, a joint loss function adapted to the target tracking task is constructed. The joint loss function integrates the feature loss function constructed for the target tracking feature matching dimension and the regression loss function constructed for the target tracking localization regression dimension. Based on the joint loss function, the adversarial samples are iteratively optimized, and then the original digital adversarial sample image is updated to generate physical adversarial samples for the target tracking task.

[0009] In one embodiment, the physical color mapping network is a three-layer fully connected neural network; The physical color mapping network is trained using the color set achievable by the printing device and the real observed color set under different shooting conditions, in order to learn the mapping relationship between the color set achievable by the printing device and the real observed color set.

[0010] In one embodiment, the mask-driven prompt word generation model includes a scene hypothesis branch, a target classification branch, and a trajectory prediction branch. The three branches jointly output the structured prompt words. The scene hypothesis branch samples environmental description information from a preset environmental language library. The target classification branch identifies target category labels based on the target shape in the target mask video and performs supervised learning through cross-entropy loss. The trajectory prediction branch generates a target motion pattern description based on the time-series displacement information in the target mask video analyzed by a convolutional neural network.

[0011] In one embodiment, the two-dimensional adversarial sample video, the reconstructed video, and the target mask video are fused together using the following calculation formula:

[0012] In the above formula, This refers to the actual video. This refers to the reconstructed video. This refers to the target masked video. This refers to the two-dimensional adversarial sample video. This indicates element-wise multiplication.

[0013] In one embodiment, when calculating the feature loss function: The real video is input into the target tracking model to be adapted, and the template frame feature matrix and search frame feature matrix output by a specific layer of the target tracking model feature extraction network are extracted. The template frame feature matrix and the search frame feature matrix are processed using the integral gradient method to calculate the corresponding feature response matrix. High response features in the feature response matrix are then selected to obtain the template frame feature importance matrix and the search frame feature importance matrix. Based on a preset threshold, a corresponding feature importance mask matrix is ​​generated for each feature importance matrix. The feature importance mask matrix is ​​then used to mask the original template frame feature matrix and the original search frame feature matrix, preserving key feature regions. The feature matrix of the masked template frame is used as the convolution kernel and convolved with the feature matrix of the masked search frame. The feature similarity between the two in the spatial dimension is calculated to obtain the feature importance loss function.

[0014] In one embodiment, when calculating the regression loss function: The real video is input into the target tracking model to be adapted, and several candidate prediction boxes and preset target expected bounding boxes are obtained from the regression branch of the target tracking model in each frame. Calculate the size difference between the candidate predicted bounding box and the target expected bounding box, and obtain the sum of size differences for a single candidate predicted bounding box; The regression loss function is obtained by squaring the size differences of all individual candidate prediction boxes and summing all the squared results.

[0015] In one embodiment, the joint loss function is expressed as:

[0016] In the above formula, Denotes the joint loss function, Represents the feature loss function, This represents the regression loss function. , This represents the weighting coefficient.

[0017] This application also provides an adversarial example generation apparatus adapted for target tracking tasks, the apparatus comprising: The raw data acquisition module is used to acquire the target's 3D model, a set of rendering parameters for the target and camera operation, and a raw digital adversarial sample image. The color correction module is used to perform color correction on the original digital adversarial sample image using a physical color mapping network to obtain a physical adversarial sample image. The 2D adversarial sample video generation module is used to fuse the physical adversarial sample image with the target 3D model through a neural renderer, and combine the 3D coordinate point information and Euler angle information of the target to be tracked and the camera in the rendering parameters to generate a 2D adversarial sample video. The structured cue word generation module is used to binarize the two-dimensional adversarial sample video to obtain a target mask video that distinguishes between foreground targets and background regions. The module then generates structured cue words containing environmental semantics and motion trajectory descriptions from the target mask video using a mask-driven cue word generation model. The real video generation module is used to input the structured prompt words and target mask video into the visual language big model, generate a reconstructed video that conforms to the preset semantic description, motion trajectory and matches the distribution of real scene, and fuse the two-dimensional adversarial sample video, the reconstructed video and the target mask video to obtain a real video containing adversarial samples. The loss function construction module is used to construct a joint loss function adapted to the target tracking task based on the real video. The joint loss function integrates the feature loss function constructed for the target tracking feature matching dimension and the regression loss function constructed for the target tracking localization regression dimension. The physical adversarial sample generation module is used to iteratively optimize the adversarial samples based on the joint loss function, and then update the original digital adversarial sample image to generate physical adversarial samples for the target tracking task.

[0018] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps in the above-described adversarial example generation method for adapting to target tracking tasks.

[0019] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above-described adversarial example generation method for adapting to target tracking tasks.

[0020] The aforementioned adversarial example generation method, apparatus, device, and medium adapted for target tracking tasks utilize a physical color mapping network to perform color correction on the original digital adversarial example image to obtain a physical adversarial example image. A neural renderer then fuses the physical adversarial example image with a target 3D model, and combines this with the 3D coordinates and Euler angles of the target and camera in the rendering parameters to generate a 2D adversarial example video. The 2D adversarial example video is binarized to obtain a target mask video that distinguishes between foreground and background regions. A mask-driven cue word generation model generates structured cue words containing environmental semantics and motion trajectory descriptions from the target mask video. The structured cue words and target mask video are input into a visual language big data model to generate a reconstructed video that conforms to the preset semantic description, motion trajectory and matches the distribution of real scene. The two-dimensional adversarial sample video, the reconstructed video and the target mask video are fused to obtain a real video containing adversarial samples. Based on the real video, a joint loss function adapted to the target tracking task is constructed. This joint loss function integrates a feature loss function constructed for the target tracking feature matching dimension and a regression loss function constructed for the target tracking localization regression dimension. The adversarial samples are iteratively optimized based on the joint loss function, and then the original digital adversarial sample image is updated to generate physical adversarial samples for the target tracking task.

[0021] Beneficial effects: This method combines physical color correction, 3D neural rendering, and mask-driven semantic cue generation to effectively improve the physical realism and scene integration of adversarial examples. At the same time, iterative optimization of adversarial examples based on the joint loss function of fusion feature matching and localization regression can accurately affect the core links of target tracking, significantly enhance the scene adaptability of physical adversarial examples to target tracking tasks, and reduce the localization accuracy of target trackers. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating an adversarial example generation method adapted for target tracking tasks in one embodiment. Figure 2 This is a schematic diagram illustrating the specific process of generating physical adversarial samples for a target tracking task using this method in one embodiment; Figure 3 This is a schematic diagram illustrating the feature importance loss calculation process in one embodiment; Figure 4 This is a structural block diagram of an adversarial example generation device adapted for target tracking tasks in one embodiment; Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] In existing technologies, adversarial examples adapted for target tracking tasks are mostly in the form of pure digital textures, lacking physical realism and exhibiting poor adaptability to real 3D scenes, target poses, and camera parameters. Furthermore, they do not incorporate the core requirements of feature matching and localization regression for target tracking, resulting in adversarial examples failing to effectively reduce the localization accuracy of target trackers in real-world scenarios. In this application, such as... Figure 1 As shown, an adversarial example generation method adapted for target tracking tasks is provided, which specifically includes the following steps: Step S100: Obtain the target 3D model of the target to be tracked, a set of rendering parameters of the target to be tracked and the camera, and an original digital adversarial sample image.

[0025] Step S110: Use a physical color mapping network to perform color correction on the original digital adversarial sample image to obtain a physical adversarial sample image.

[0026] Step S120: The physical adversarial sample image is fused with the target 3D model using a neural renderer, and combined with the 3D coordinate point information and Euler angle information of the target to be tracked and the camera in the rendering parameters to generate a 2D adversarial sample video.

[0027] Step S130: The two-dimensional adversarial sample video is binarized to obtain a target mask video that distinguishes between the foreground target and the background region. The mask-driven prompt word generation model generates structured prompt words containing environmental semantics and motion trajectory descriptions from the target mask video.

[0028] Step S140: Input the structured prompts and the target mask video into the visual language big model to generate a reconstructed video that conforms to the preset semantic description, motion trajectory and matches the distribution of the real scene. Then, fuse the two-dimensional adversarial sample video, the reconstructed video and the target mask video to obtain a real video containing adversarial samples.

[0029] Step S150: Construct a joint loss function adapted to the target tracking task based on real video. The joint loss function integrates the feature loss function constructed for the target tracking feature matching dimension and the regression loss function constructed for the target tracking localization regression dimension.

[0030] Step S160: Iteratively optimize the adversarial examples based on the joint loss function, and then update the original digital adversarial example image to generate physical adversarial examples for the target tracking task.

[0031] The adversarial example generation method for target tracking tasks proposed in this application can be widely applied to various real-world scenarios that require real-time target tracking and verification of the robustness of the tracking system. The target tracking tasks may include, but are not limited to, target tracking, autonomous driving target tracking, drone inspection target tracking, satellite remote sensing target tracking, and industrial scene target tracking.

[0032] Specifically, in security scenarios, the target to be tracked can be a face, a suspicious person, a key controlled vehicle, etc. The physical adversarial sample generated by this application can be attached to the surface of an object in the application scenario to test the tracking stability of the tracking system in complex environments and verify whether it can effectively resist the influence of the physical adversarial sample.

[0033] Specifically, in autonomous driving scenarios, the target to be tracked can be a vehicle, pedestrian, non-motorized vehicle, traffic sign, etc. The physical adversarial sample can be designed as a physical texture that adapts to the vehicle's appearance and road signs, and is used to test the target tracking module of the autonomous driving system to improve the system's safety and reliability in adversarial sample scenarios.

[0034] Specifically, in the scenario of drone inspection, the target to be tracked can be power lines, transmission towers, bridge components, farmland crops, etc. The physical adversarial samples generated by this application can simulate the influencing factors in the real environment and verify the accuracy of the drone tracking system in identifying and tracking the inspection target.

[0035] Specifically, in satellite remote sensing target tracking scenarios, the target to be tracked can be ground buildings, ships, vegetation areas, etc. The physical adversarial examples can be adapted to the satellite observation perspective and the ground environment to test the robustness of the remote sensing tracking system and ensure that it can still accurately track targets in complex ground environments.

[0036] Specifically, in industrial scenarios, the targets to be tracked can be production equipment, processed workpieces, assembly line materials, etc. Physical adversarial examples can be adapted to the industrial production environment to verify the stability and robustness of the industrial tracking system in the production process and ensure the orderly progress of the production process.

[0037] like Figure 2 The diagram shows a schematic of the specific process for generating physical adversarial samples for target tracking tasks using this method.

[0038] In step S100, based on the actual target tracking task to be performed, a 3D model of the target to be tracked is obtained. This 3D model must accurately match the shape of the target in the specific scenario. For example, in a security scenario, the 3D model of a vehicle to be tracked must reproduce the vehicle's outline, dimensions, and other key features; in a drone inspection scenario, the 3D model of a power transmission tower must accurately replicate the tower structure and component distribution; and in an autonomous driving scenario, the 3D model of a pedestrian must conform to human body proportions and common postures. Simultaneously, a set of rendering parameters for the target and camera operation is obtained. These parameters include the 3D coordinates and Euler angles of the target and camera, as well as an original digital adversarial sample image. This original digital adversarial sample image must be adapted to the environmental characteristics of different application scenarios. For example, in a security scenario, it must adapt to the lighting and color tone of the environment; in a satellite remote sensing scenario, it must adapt to the natural texture and color tone of the ground; and in an industrial scenario, it must adapt to the lighting and background texture of the production workshop. This serves as the basic texture material for subsequent physical color correction, scene fusion, and iterative optimization. Based on the aforementioned target 3D model and rendering parameters, the dataset required for training adversarial samples can be automatically generated.

[0039] In step S110, during the process of printing digital textures onto physical materials, discrepancies arise between digital domain colors and physical domain colors due to differences in printing equipment, material properties, and shooting conditions. This affects the effectiveness of adversarial examples in real-world scenarios. To address this issue, this method designs a learnable physical color mapping network. It is used to simulate the color changes that occur when digital textures are printed onto physical materials.

[0040] In this embodiment, the physical color mapping network It is a three-layer fully connected neural network. This physical color mapping network is trained using the color set achievable by the printing device and the real observed color set under different shooting conditions, so as to learn the mapping relationship between the color set achievable by the printing device and the real observed color set.

[0041] Specifically, the first step is to collect the set of colors that the printing device can produce. And the set of real observed colors obtained by sampling at different times and under different shooting conditions. Then, a three-layer fully connected neural network is constructed. Used to learn color sets arrive The mapping relationship is as follows:

[0042] The digital texture is color-corrected using a trained mapping function, resulting in a physical texture that more closely resembles the actual printing effect. (i.e., physical adversarial examples), that is:

[0043] In step S120, the rendering capabilities of the neural renderer are used to precisely attach the physically color-corrected adversarial sample image to the surface of the target 3D model, achieving deep fusion of texture and target shape. This avoids issues such as texture misalignment and distortion, ensuring that the adversarial sample can naturally adhere to the target being tracked. Simultaneously, by combining target pose information sampled in the 3D scene, such as the target's motion angle and posture changes (e.g., the tilt angle of a power transmission tower in a drone inspection scenario, or the limb movement posture of a pedestrian in an autonomous driving scenario), and camera pose information (e.g., the shooting angle, distance, and focal length of cameras, drone cameras, and vehicle cameras), the dynamic scenes of target movement and camera perspective changes during real target tracking can be simulated, generating continuous 2D adversarial sample videos. These 2D adversarial sample videos can recreate the dynamic process of target tracking in different application scenarios, ensuring that the subsequently generated adversarial samples not only possess physical realism but also adapt to dynamic tracking scenarios. This effectively solves the problem that static adversarial samples cannot accurately target dynamic target tracking tasks, providing dynamic materials that fit real-world scenarios for subsequent masking, semantic prompt generation, and iterative optimization of adversarial samples, further improving the effectiveness and scene adaptability of the final physical adversarial samples.

[0044] Specifically, through the neural renderer R The adversarial examples are fused with the 3D target model, and combined with the target pose information sampled in the 3D scene. V and camera pose information CGenerate corresponding two-dimensional adversarial sample videos. The process can be represented as follows:

[0045] In step S130, considering that the scene materials, lighting conditions, and background structures in simulation environments (such as CARLA, Car Learning to Act, an open-source autonomous driving simulation platform) are usually idealized and differ significantly from real environments, adversarial examples trained solely on simulation data have limited effect on reducing the accuracy of target tracker localization in real-world scenarios. Therefore, this method incorporates a physical simulation component to generate video data that more closely approximates real-world distributions through simulation of real-world scenarios.

[0046] In this embodiment, the two-dimensional adversarial sample video is first analyzed. Binarization is performed to distinguish the foreground target from the background region, resulting in the target mask video. Subsequently, a mask-driven prompt generation model, "Scene Description," was constructed to generate prompts from the target masked video. Structured prompts that automatically generate environmental semantics and motion trajectory descriptions .

[0047] Furthermore, the mask-driven prompt generation model includes a scene hypothesis branch, a target classification branch, and a trajectory prediction branch. These three branches jointly output the structured prompt. Specifically, the scene hypothesis branch samples environmental description information from a pre-defined environmental language library; the target classification branch identifies target category labels based on the target shape in the target mask video and performs supervised learning using cross-entropy loss; and the trajectory prediction branch analyzes the time-series displacement information in the target mask video using a convolutional neural network to generate a description of the target motion pattern. Finally, the structured prompt is generated through the joint operation of these three branches. .

[0048] Specifically, the scenario assumption branch samples environmental descriptions from a pre-defined environmental language library, such as "city streets" or "rural roads." The target classification branch identifies target category labels, such as vehicles or pedestrians, based on the shape of the targets in the masked video, and performs supervised learning using cross-entropy loss during the training phase. The trajectory prediction branch analyzes the time-series displacement information in the masked video using a convolutional neural network to generate descriptions of target motion patterns, such as going straight or turning. Finally, through the synergistic effect of these three branches, structured prompts that combine environmental semantics and motion trajectory information are generated. .

[0049] In step S140, the differences between the simulation environment and the real environment are further compensated by deeply integrating the previously generated adversarial examples with the real scene to obtain video materials with high realism. This provides a training basis that fits the actual application scenario for the iterative optimization of subsequent adversarial examples, thereby improving the effect and stability of the final physical adversarial examples in real target tracking tasks.

[0050] Specifically, the structured prompt words generated in step S130 With target mask video The common input is fed into a large visual language model, including structured prompt words. Provides precise semantic guidance and motion trajectory constraints for large-scale visual language models, clearly defining the environmental semantics (e.g., "city street, well-lit") and target motion patterns (e.g., "vehicle moving straight") that the video must conform to, and masking the target video. This provides the model with clear foreground target outline information, avoiding problems such as target misalignment and background mismatch in the generated scene. Based on the above input, the visual language big model, combined with its powerful real-world scene generation and semantic understanding capabilities, generates a reconstructed video that conforms to the preset semantic description, fits the target's motion trajectory, and matches the distribution of the real scene. The reconstructed video It can reproduce the lighting changes, background details, material textures and other features in real scenes, effectively solving the problem of idealizing simulation data.

[0051] Furthermore, regarding two-dimensional adversarial sample videos Reconstructing the video and target mask video The fusion process is performed using the target mask video. Using image fusion algorithms as the core constraint, two-dimensional adversarial example videos are transformed. Adversarial examples are precisely overlaid into the reconstructed video. The foreground target area, while preserving the reconstructed video. By incorporating realistic background environments and scene details, adversarial examples are seamlessly integrated with real-world scenes, avoiding jarring or incongruous situations. Ultimately, through this fusion process, a realistic video containing adversarial examples and closely resembling the distribution of real-world scene elements is obtained. This real video not only retains the effect of adversarial examples in two-dimensional adversarial example videos, but also has environmental adaptability to real scenes. It provides accurate and realistic training data for subsequent iterative optimization of adversarial examples based on joint loss functions, and further ensures that the final generated physical adversarial examples can play an effective role in real target tracking scenarios.

[0052] Specifically, the fusion process of the 2D adversarial sample video, the reconstructed video, and the target mask video is expressed by the following formula:

[0053] In the above formula, This refers to the actual video. This refers to the reconstructed video. This refers to the target masked video. This refers to the two-dimensional adversarial sample video. This indicates element-wise multiplication.

[0054] In step S150, after generating the real-scene video, the adversarial examples are further optimized by constructing an optimized loss function for the target tracking model. Traditional Gram matrix-based loss functions only characterize the statistical relationships between channels, ignoring spatial structure information and are easily affected by background information. Therefore, this method proposes a feature importance loss function, i.e., a feature loss function, such as... Figure 3 The diagram shows the calculation process of the feature importance loss function.

[0055] In this embodiment, when calculating the feature loss function: Real video is input into the target tracking model to be adapted; the template frame feature matrix and search frame feature matrix output from a specific layer of the target tracking model's feature extraction network are extracted; the integral gradient method is used to process the template frame feature matrix and search frame feature matrix respectively, calculating the corresponding feature response matrix; and high-response features in the feature response matrix are selected to obtain the template frame feature importance matrix. With the search frame feature importance matrix Then, based on a preset threshold, corresponding feature importance mask matrices are generated for each feature importance matrix. These mask matrices are then used to mask the original template frame feature matrix and the original search frame feature matrix, preserving key feature regions. Subsequently, the masked template frame feature matrix is ​​used as a convolution kernel and convolved with the masked search frame feature matrix. The spatial similarity between the two is calculated to obtain the feature importance loss function.

[0056] Specifically, the feature response matrices of specific layers in the template frame and search frame are calculated using the integral gradient method, and high-response features are selected to obtain the feature importance matrix. and The process is represented as follows:

[0057]

[0058] In the above formula, and These represent the feature matrices of the template frame and the search frame, respectively. This represents the regression box prediction branch of the tracker. This indicates dot product.

[0059] Specifically, based on the set threshold The threshold is typically selected using the top-k values ​​with high response. A feature importance mask matrix is ​​generated using the set threshold, while retaining key feature regions. This process is represented as follows:

[0060]

[0061] In the above formula, and These represent the template frame feature matrix and the search frame feature matrix after masking, respectively. Then, The template feature matrix is ​​used as the convolution kernel, and... Search the feature matrix and perform convolution operations. To calculate the feature similarity between the two in the spatial dimension, and to construct the feature importance loss function, expressed as:

[0062] In this embodiment, in the regression branch of the target tracking model, several candidate predicted boxes are generated for each frame. The regression box loss function is constructed by calculating the difference between the predicted box size and the target expected bounding box.

[0063] Specifically, when calculating the regression loss function: input the real video into the target tracking model to be adapted, obtain several candidate prediction boxes output by the regression branch of the target tracking model in each frame, as well as the preset target expected bounding box, calculate the size difference between the candidate prediction box and the target expected bounding box, and obtain the size difference sum of individual candidate prediction boxes. Square the size difference sum of all individual candidate prediction boxes respectively, and then sum all the squared results to obtain the regression loss function.

[0064] Furthermore, the regression loss function is expressed as:

[0065] In the above formula, Represents the set of prediction boxes. This represents the height and width of the candidate prediction box. This indicates the height and width of the target bounding box.

[0066] In this embodiment, the feature importance loss and the regression box loss are weighted and combined as a joint optimization objective, namely:

[0067] In the above formula, Denotes the joint loss function, Represents the feature loss function, This represents the regression loss function. , This represents the weighting coefficient.

[0068] In this embodiment, the real video containing adversarial examples obtained in step S140 is used. By combining a constructed joint loss function adapted to target tracking tasks, adversarial examples in videos are iteratively optimized. During the iteration process, the effectiveness of adversarial examples in real-world scenarios is calculated in real time using the joint loss function. This accurately captures the shortcomings of adversarial examples in the feature matching and localization regression stages of the target tracking system, and then dynamically adjusts the pixel distribution, texture details, and physical adaptation parameters of the adversarial examples, while simultaneously updating the original digital adversarial example image. After multiple rounds of iterative optimization until the adversarial examples meet the preset performance threshold, a physical adversarial example with strong capabilities is finally generated. This example accurately adapts to real-world target tracking scenarios and can effectively influence the core workings of the target tracking system, thus ensuring that it can stably achieve the expected results in various practical application scenarios.

[0069] In the aforementioned adversarial example generation method adapted for target tracking tasks, a physical color mapping network is first constructed to model the color deviation generated during the printing of digital textures onto physical materials. This ensures the consistency of the generated adversarial examples between the digital and physical domains, thereby improving the transferability of adversarial examples in real-world environments. Furthermore, neural rendering technology is used to generate target motion videos by combining target and camera pose information. Structured cue words are constructed using scene semantic descriptions and target motion trajectories to guide the visual language generation model in generating simulated video data that conforms to the distribution of real-world scenes. This effectively alleviates the domain offset problem between simulated and real data while reducing the cost of real data acquisition. Simultaneously, this method proposes a feature importance loss mechanism. By filtering key features and establishing spatial feature correspondences between template frames and search frames, the influence of background is effectively suppressed. Combined with a regression box loss, the localization branch of the tracker is perturbed, significantly reducing the localization accuracy of the target tracker. Through the synergistic effect of the above-mentioned technical features, this method can generate adversarial examples with higher realism and stronger physical scene adaptability, improve the effectiveness and stability of single-target tracking adversarial examples, and thus solve the problems of insufficient data realism and limited effect when relying on simulation platforms to generate adversarial examples in existing technologies.

[0070] It should be understood that, although Figure 1The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0071] In one embodiment, such as Figure 4 As shown, an adversarial example generation device adapted for target tracking tasks is provided, including: a raw data acquisition module 200, a color correction module 210, a two-dimensional adversarial example video generation module 220, a structured cue word generation module 230, a real video generation module 240, a loss function construction module 250, and a physical adversarial example generation module 260, wherein: The raw data acquisition module 200 is used to acquire the target's 3D model, a set of rendering parameters for the target and camera operation, and a raw digital adversarial sample image. Color correction module 210 is used to perform color correction on the original digital adversarial sample image using a physical color mapping network to obtain a physical adversarial sample image; The 2D adversarial sample video generation module 220 is used to fuse the physical adversarial sample image with the target 3D model through a neural renderer, and combine the 3D coordinate point information and Euler angle information of the target to be tracked and the camera in the rendering parameters to generate a 2D adversarial sample video. The structured cue word generation module 230 is used to binarize the two-dimensional adversarial sample video to obtain a target mask video that distinguishes between foreground targets and background regions, and to generate structured cue words containing environmental semantics and motion trajectory descriptions from the target mask video through a mask-driven cue word generation model. The real video generation module 240 is used to input the structured prompt words and target mask video into the visual language big model, generate a reconstructed video that conforms to the preset semantic description, motion trajectory and matches the distribution of real scene, and fuse the two-dimensional adversarial sample video, the reconstructed video and the target mask video to obtain a real video containing adversarial samples. The loss function construction module 250 is used to construct a joint loss function adapted to the target tracking task based on the real video. The joint loss function integrates the feature loss function constructed for the target tracking feature matching dimension and the regression loss function constructed for the target tracking localization regression dimension. The physical adversarial sample generation module 260 is used to iteratively optimize the adversarial samples based on the joint loss function, and then update the original digital adversarial sample image to generate physical adversarial samples for the target tracking task.

[0072] Specific limitations regarding the adversarial example generation device adapted for target tracking tasks can be found in the limitations of the adversarial example generation method adapted for target tracking tasks described above, and will not be repeated here. Each module in the aforementioned adversarial example generation device adapted for target tracking tasks can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0073] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements an adversarial example generation method adapted for target tracking tasks. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0074] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0075] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps: Acquire the 3D model of the target to be tracked, a set of rendering parameters for the target and camera operation, and an original digital adversarial sample image; The original digital adversarial sample image is color-corrected using a physical color mapping network to obtain a physical adversarial sample image; The adversarial sample image is fused with the target 3D model using a neural renderer, and combined with the 3D coordinate point information and Euler angle information of the target to be tracked and the camera in the rendering parameters to generate a 2D adversarial sample video. The two-dimensional adversarial sample video is binarized to obtain a target mask video that distinguishes between foreground targets and background regions. A mask-driven prompt word generation model is used to generate structured prompt words containing environmental semantics and motion trajectory descriptions from the target mask video. The structured prompts and the target mask video are input into a large visual language model to generate a reconstructed video that conforms to the preset semantic description, motion trajectory and matches the distribution of the real scene. The two-dimensional adversarial sample video, the reconstructed video and the target mask video are fused to obtain a real video containing adversarial samples. Based on the real video, a joint loss function adapted to the target tracking task is constructed. The joint loss function integrates the feature loss function constructed for the target tracking feature matching dimension and the regression loss function constructed for the target tracking localization regression dimension. Based on the joint loss function, the adversarial samples are iteratively optimized, and then the original digital adversarial sample image is updated to generate physical adversarial samples for the target tracking task.

[0076] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Acquire the 3D model of the target to be tracked, a set of rendering parameters for the target and camera operation, and an original digital adversarial sample image; The original digital adversarial sample image is color-corrected using a physical color mapping network to obtain a physical adversarial sample image; The adversarial sample image is fused with the target 3D model using a neural renderer, and combined with the 3D coordinate point information and Euler angle information of the target to be tracked and the camera in the rendering parameters to generate a 2D adversarial sample video. The two-dimensional adversarial sample video is binarized to obtain a target mask video that distinguishes between foreground targets and background regions. A mask-driven prompt word generation model is used to generate structured prompt words containing environmental semantics and motion trajectory descriptions from the target mask video. The structured prompts and the target mask video are input into a large visual language model to generate a reconstructed video that conforms to the preset semantic description, motion trajectory and matches the distribution of the real scene. The two-dimensional adversarial sample video, the reconstructed video and the target mask video are fused to obtain a real video containing adversarial samples. Based on the real video, a joint loss function adapted to the target tracking task is constructed. The joint loss function integrates the feature loss function constructed for the target tracking feature matching dimension and the regression loss function constructed for the target tracking localization regression dimension. Based on the joint loss function, the adversarial samples are iteratively optimized, and then the original digital adversarial sample image is updated to generate physical adversarial samples for the target tracking task.

[0077] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0078] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for generating adversarial examples adapted to target tracking tasks, characterized in that, The method includes: Acquire the 3D model of the target to be tracked, a set of rendering parameters for the target and camera operation, and an original digital adversarial sample image; The original digital adversarial sample image is color-corrected using a physical color mapping network to obtain a physical adversarial sample image; The adversarial sample image is fused with the target 3D model using a neural renderer, and combined with the 3D coordinate point information and Euler angle information of the target to be tracked and the camera in the rendering parameters to generate a 2D adversarial sample video. The two-dimensional adversarial sample video is binarized to obtain a target mask video that distinguishes between foreground targets and background regions. A mask-driven prompt word generation model is used to generate structured prompt words containing environmental semantics and motion trajectory descriptions from the target mask video. The structured prompts and the target mask video are input into a large visual language model to generate a reconstructed video that conforms to the preset semantic description, motion trajectory and matches the distribution of the real scene. The two-dimensional adversarial sample video, the reconstructed video and the target mask video are fused to obtain a real video containing adversarial samples. Based on the real video, a joint loss function adapted to the target tracking task is constructed. The joint loss function integrates the feature loss function constructed for the target tracking feature matching dimension and the regression loss function constructed for the target tracking localization regression dimension. Based on the joint loss function, the adversarial samples are iteratively optimized, and then the original digital adversarial sample image is updated to generate physical adversarial samples for the target tracking task.

2. The adversarial example generation method for target tracking tasks according to claim 1, characterized in that, The physical color mapping network is a three-layer fully connected neural network; The physical color mapping network is trained using the color set achievable by the printing device and the real observed color set under different shooting conditions, in order to learn the mapping relationship between the color set achievable by the printing device and the real observed color set.

3. The adversarial example generation method adapted for target tracking tasks according to claim 2, characterized in that, The mask-driven prompt generation model includes a scene hypothesis branch, a target classification branch, and a trajectory prediction branch. The three branches jointly output the structured prompt. The scene hypothesis branch samples environmental description information from a preset environmental language library. The target classification branch identifies target category labels based on the target shape in the target mask video and performs supervised learning through cross-entropy loss. The trajectory prediction branch analyzes the time-series displacement information in the target mask video based on a convolutional neural network to generate a target motion pattern description.

4. The adversarial example generation method for target tracking tasks according to claim 3, characterized in that, The fusion process of the two-dimensional adversarial sample video, the reconstructed video, and the target mask video is expressed by the following formula: In the above formula, This refers to the actual video. This refers to the reconstructed video. This refers to the target masked video. This refers to the two-dimensional adversarial sample video. This indicates element-wise multiplication.

5. The adversarial example generation method for target tracking tasks according to any one of claims 1-4, characterized in that, When calculating the feature loss function: The real video is input into the target tracking model to be adapted, and the template frame feature matrix and search frame feature matrix output by a specific layer of the target tracking model feature extraction network are extracted. The template frame feature matrix and the search frame feature matrix are processed using the integral gradient method to calculate the corresponding feature response matrix. High response features in the feature response matrix are then selected to obtain the template frame feature importance matrix and the search frame feature importance matrix. Based on a preset threshold, a corresponding feature importance mask matrix is ​​generated for each feature importance matrix. The feature importance mask matrix is ​​then used to mask the original template frame feature matrix and the original search frame feature matrix, preserving key feature regions. The feature matrix of the masked template frame is used as the convolution kernel and convolved with the feature matrix of the masked search frame. The feature similarity between the two in the spatial dimension is calculated to obtain the feature importance loss function.

6. The adversarial example generation method for target tracking tasks according to claim 5, characterized in that, When calculating the regression loss function: The real video is input into the target tracking model to be adapted, and several candidate prediction boxes and preset target expected bounding boxes are obtained from the regression branch of the target tracking model in each frame. Calculate the size difference between the candidate predicted bounding box and the target expected bounding box, and obtain the sum of size differences for a single candidate predicted bounding box; The regression loss function is obtained by squaring the size differences of all individual candidate prediction boxes and summing all the squared results.

7. The adversarial example generation method for target tracking tasks according to claim 6, characterized in that, The joint loss function is expressed as: In the above formula, Denotes the joint loss function, Represents the feature loss function, This represents the regression loss function. , This represents the weighting coefficient.

8. An adversarial example generation device adapted for target tracking tasks, characterized in that, The device includes: The raw data acquisition module is used to acquire the target's 3D model, a set of rendering parameters for the target and camera operation, and a raw digital adversarial sample image. The color correction module is used to perform color correction on the original digital adversarial sample image using a physical color mapping network to obtain a physical adversarial sample image. The 2D adversarial sample video generation module is used to fuse the physical adversarial sample image with the target 3D model through a neural renderer, and combine the 3D coordinate point information and Euler angle information of the target to be tracked and the camera in the rendering parameters to generate a 2D adversarial sample video. The structured cue word generation module is used to binarize the two-dimensional adversarial sample video to obtain a target mask video that distinguishes between foreground targets and background regions. The module then generates structured cue words containing environmental semantics and motion trajectory descriptions from the target mask video using a mask-driven cue word generation model. The real video generation module is used to input the structured prompt words and target mask video into the visual language big model, generate a reconstructed video that conforms to the preset semantic description, motion trajectory and matches the distribution of real scene, and fuse the two-dimensional adversarial sample video, the reconstructed video and the target mask video to obtain a real video containing adversarial samples. The loss function construction module is used to construct a joint loss function adapted to the target tracking task based on the real video. The joint loss function integrates the feature loss function constructed for the target tracking feature matching dimension and the regression loss function constructed for the target tracking localization regression dimension. The physical adversarial sample generation module is used to iteratively optimize the adversarial samples based on the joint loss function, and then update the original digital adversarial sample image to generate physical adversarial samples for the target tracking task.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images