Adversarial sample background semantic robustness enhancement method, system, device and medium

Abstract

The application belongs to the field of computer vision, and discloses an adversarial sample background semantic robustness enhancement method, system, device and medium, comprising: obtaining a target data set, and constructing a knowledge graph by defining object nodes and background semantic relationships; based on the knowledge graph, scene semantic information in the target data set is analyzed to obtain a background image data set; background images meeting index conditions are screened in the background image data set to generate a diverse background data set; based on the diverse background data set, adversarial samples are trained to obtain an attack success rate expectation value of the optimized adversarial samples. The application uses the knowledge graph to more deeply mine and effectively utilize the background semantic information, solves the problem that existing methods lack systematic guidance in selecting and generating representative backgrounds, greatly reduces the labor cost consumed in screening images; and can be applied to most adversarial attack algorithms to improve the robustness of adversarial samples generated by the adversarial attack algorithm training in diverse backgrounds.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to methods, systems, devices, and media for enhancing the robustness of adversarial example background semantics. Background Technology

[0002] The rapid development of artificial intelligence, while bringing convenience to our lives, has also raised numerous security issues. Adversarial attacks are an important research area for revealing security problems in real-world deep learning models and advancing adversarial defense technologies. An adversarial attack refers to an attack method targeting machine learning models, aiming to deceive the model into producing incorrect outputs. It primarily involves making small but adversarial modifications to the input data to generate adversarial examples, causing the model to misclassify the data.

[0003] However, in the complex and diverse context of the physical world, adversarial examples are often affected by background semantic information, resulting in a significant decrease in adversarial effectiveness and a loss of the ability to attack the target model. This is because the background of the physical world is complex, diverse, unpredictable, and cannot be manipulated. Not only can attacks not be launched by modifying image background information, but the adversarial effectiveness of adversarial examples is further reduced because object detection models utilize background context information to assist in reasoning and classification.

[0004] To address the impact of contextual semantic information on adversarial attacks, existing methods optimize the robustness of adversarial examples by incorporating a reasonably constructed context into the training process of the adversarial examples. While increasing contextual diversity during training does improve the robustness of adversarial examples, clear guidance on how to systematically select and generate representative contexts remains lacking.

[0005] Existing methods that rely on search engines to obtain diverse backgrounds are too crude. The results returned by search engines are mixed, and there may be derivative images. Whether the backgrounds obtained by searching directly by the target name are reasonable still needs to be further filtered by humans, which will undoubtedly consume a certain amount of labor costs and reduce the efficiency of generating adversarial examples. Secondly, obtaining keywords that are highly relevant to the target semantics is a difficult problem, and using expert knowledge will further increase the labor costs.

[0006] Therefore, how to provide a method, system, device, and medium for enhancing the robustness of adversarial sample background semantics is an urgent problem to be solved. Summary of the Invention

[0007] The embodiments of the present invention provide a method, system, device and medium for enhancing the robustness of adversarial sample background semantics, so as to solve the above-mentioned problems existing in the prior art.

[0008] To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or to describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0009] According to a first aspect of the present invention, a method for enhancing the robustness of adversarial sample background semantics is provided.

[0010] In one embodiment, the adversarial example context semantic robustness enhancement method includes:

[0011] Obtain the target dataset and construct a knowledge graph by defining the semantic relationship between object nodes and background.

[0012] Based on knowledge graphs, the scene semantic information in the target dataset is analyzed to obtain the background image dataset;

[0013] Filter background images that meet the index criteria from the background image dataset to generate diverse background datasets;

[0014] Based on diverse background datasets, adversarial examples are trained to obtain the expected attack success rate of the optimized adversarial examples.

[0015] In one embodiment, acquiring the target dataset and constructing a knowledge graph by defining the semantic relationships between object nodes and background includes:

[0016] Parse the annotation files of the images in the target dataset to extract object information for each image; based on the object information, construct the scene graph corresponding to the image.

[0017] Based on the scene graph, construct the adjacency matrix of each image and integrate them to form a set of adjacency matrices;

[0018] Select all objects associated with the labels in the label set of the target dataset and define them as nodes of the knowledge graph; then obtain the background semantic relationships between the objects and define them as edges of the knowledge graph.

[0019] By iteratively traversing the label set in the target dataset, all objects associated with the labels and their background semantic relationships are added to the knowledge graph, and the weights of the background semantic relationships between the labels are calculated using the adjacency matrix set.

[0020] In one embodiment, the object information includes object instance, relationship, and location; in the adjacency matrix, 1 indicates that two objects are semantically related in their backgrounds, and 0 indicates that two objects are semantically unrelated in their backgrounds.

[0021] In one embodiment, the step of parsing the scene semantic information in the target dataset based on the knowledge graph to obtain the background image dataset includes:

[0022] Select tags for the target objects of the adversarial attack, define them as keywords, and use the knowledge graph to search for object nodes corresponding to the keywords;

[0023] Based on the object nodes obtained from the search, a keyword index set is established; and images containing the keywords are searched in the target dataset to generate a background image dataset.

[0024] In one embodiment, the step of filtering background images that meet the index criteria from the background image dataset to generate diverse background datasets includes:

[0025] Based on the keyword index set, background images are filtered from the background image dataset;

[0026] Extract the target object from each image in the target dataset and fuse the target object with the background image to establish an adversarial attack target object image set;

[0027] Based on deep learning image fusion algorithms, the realism of the fused images is enhanced, resulting in a diverse background dataset with improved robustness.

[0028] In one embodiment, during the fusion of the target object and the background image, the position of the target object in the background image is optimized using a scene semantic relationship reasoning model.

[0029] In one embodiment, the step of training adversarial examples based on diverse background datasets to obtain the expected attack success rate of the optimized adversarial examples includes:

[0030] The diverse background dataset is input into the expectation transformation framework to train the expected attack success rate of adversarial examples under different background conditions, ultimately obtaining the optimized expected attack success rate of the adversarial examples; wherein, the diverse background dataset includes several background conditions.

[0031] According to a second aspect of the present invention, a system for enhancing the semantic robustness of adversarial sample context is provided.

[0032] In one embodiment, the adversarial example context semantic robustness enhancement system includes:

[0033] The knowledge graph module is used to acquire the target dataset and construct a knowledge graph by defining the semantic relationship between object nodes and background.

[0034] The background generation module is used to parse the scene semantic information in the target dataset based on the knowledge graph to obtain the background image dataset;

[0035] The background fusion module is used to filter background images that meet the index conditions in the background image dataset and generate diverse background datasets.

[0036] The training optimization module is used to train adversarial examples based on diverse background datasets to obtain the expected attack success rate of the optimized adversarial examples.

[0037] In one embodiment, the knowledge graph module includes: a parsing and transformation module, a matrix construction module, a graph definition module, and an iterative calculation module, wherein,

[0038] The parsing and conversion module is used to parse the annotation files of the images in the target dataset, extract the object information of each image, and construct the scene graph corresponding to the image based on the object information.

[0039] The matrix construction module constructs an adjacency matrix for each image based on the scene graph and integrates them to form an adjacency matrix set.

[0040] The graph definition module is used to select all objects associated with labels in the label set of the target dataset and define them as nodes of the knowledge graph; then, it obtains the background semantic relationships between the objects and defines them as edges of the knowledge graph.

[0041] The iterative calculation module is used to iteratively traverse the label set in the target dataset, add all the objects associated with the labels and the background semantic relationships to the knowledge graph, and calculate the weights of the background semantic relationships between the labels through the adjacency matrix set.

[0042] In one embodiment, the object information includes object instance, relationship, and location; in the adjacency matrix, 1 indicates that two objects are semantically related in their backgrounds, and 0 indicates that two objects are semantically unrelated in their backgrounds.

[0043] In one embodiment, the background generation module includes: a node search module and a background search module, wherein,

[0044] The node search module is used to select tags for the target objects of the adversarial attack, define them as keywords, and use the knowledge graph to search for object nodes corresponding to the keywords;

[0045] The background search module is used to establish a keyword index set based on the object nodes obtained from the search; and to search for images containing the keywords in the target dataset to generate a background image dataset.

[0046] In one embodiment, the background fusion module includes: an index filtering module, a fusion establishment module, and an algorithm enhancement module, wherein,

[0047] The index filtering module is used to filter background images in the background image dataset based on the keyword index set;

[0048] The fusion establishment module is used to extract target objects from each image in the target dataset and fuse the target objects with the background image to establish an adversarial attack target object image set;

[0049] The algorithm enhancement module is used to enhance the realism of the fused image based on the deep learning image fusion algorithm, thereby obtaining a diverse background dataset with improved robustness.

[0050] In one embodiment, during the fusion of the target object and the background image, the position of the target object in the background image is optimized using a scene semantic relationship reasoning model.

[0051] In one embodiment, the training optimization module includes: an input training module and an enhancement optimization module, wherein,

[0052] The input training module is used to input the diverse background dataset into the desired transformation framework;

[0053] The enhancement and optimization module is used to train the expected attack success rate of adversarial samples under different background conditions, and finally obtain the expected attack success rate of the optimized adversarial samples; wherein, the diverse background dataset includes several background conditions.

[0054] According to a third aspect of the present invention, a computer device is provided.

[0055] In some embodiments, the computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described above.

[0056] According to a fourth aspect of the present invention, a computer-readable storage medium is provided.

[0057] In one embodiment, a computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the above method.

[0058] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0059] 1. By leveraging knowledge graphs to more deeply mine and effectively utilize background semantic information, this invention addresses the lack of systematic guidance in selecting and generating representative backgrounds in current methods, significantly reducing the manual costs associated with image selection. Furthermore, the invention possesses a universal framework applicable to most adversarial attack algorithms, improving the robustness of adversarial examples trained by various algorithms in diverse backgrounds. This addresses the challenges posed by background variations to adversarial attacks, providing a new approach to enhancing the adversarial nature of adversarial examples in complex real-world contexts.

[0060] 2. This invention addresses the problem that unpredictable and complex backgrounds in the physical world greatly reduce the adversarial effectiveness of adversarial examples. It utilizes knowledge graphs to deeply mine and effectively utilize the background semantic information of images, providing clear guidance for systematically selecting and generating representative backgrounds for robust training of adversarial examples. This generates diverse background datasets for robust training. While effectively improving the success rate of adversarial example attacks in complex backgrounds, it reduces the manual cost of background dataset construction and significantly improves the efficiency of adversarial attacks.

[0061] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0062] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0063] Figure 1 This is a flowchart illustrating an adversarial example background semantic robustness enhancement method according to an exemplary embodiment;

[0064] Figure 2 This is a logical flowchart illustrating an adversarial example background semantic robustness enhancement method according to an exemplary embodiment;

[0065] Figure 3 This is a block diagram illustrating the principle of an adversarial sample background semantic robustness enhancement system according to an exemplary embodiment;

[0066] Figure 4 This is a schematic diagram of the structure of a computer device according to an exemplary embodiment. Detailed Implementation

[0067] The following description and accompanying drawings fully illustrate specific embodiments described herein to enable those skilled in the art to practice them. Some embodiments may include or substitute parts and features of other embodiments. The scope of the embodiments herein encompasses the entire scope of the claims and all available equivalents thereof. Throughout this document, the terms “first,” “second,” etc., are used only to distinguish one element from another without requiring or implying any actual relationship or order between the elements. Indeed, a first element can also be referred to as a second element, and vice versa. Furthermore, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a structure, apparatus, or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a structure, apparatus, or device. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the structure, apparatus, or device that includes said element. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0068] The terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer" used in this document to indicate orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings. They are used solely for the convenience of describing the document and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In the description herein, unless otherwise specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two elements; they can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0069] In this document, unless otherwise stated, the term "multiple" means two or more.

[0070] In this article, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0071] In this article, the term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0072] It should be understood that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order constraint on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the diagram may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0073] The modules in the apparatus or system of this application can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0074] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0075] Figure 1 An embodiment of the adversarial sample background semantic robustness enhancement method of the present invention is shown.

[0076] In this optional embodiment, the adversarial example background semantic robustness enhancement method includes:

[0077] Step S101: Obtain the target dataset and construct a knowledge graph by defining the semantic relationship between object nodes and background.

[0078] Step S103: Based on the knowledge graph, parse the scene semantic information in the target dataset to obtain the background image dataset;

[0079] Step S105: Filter background images that meet the index conditions in the background image dataset to generate diverse background datasets;

[0080] Step S107: Based on diverse background datasets, train adversarial examples to obtain the expected attack success rate of the optimized adversarial examples.

[0081] In this optional embodiment, when acquiring the target dataset and constructing a knowledge graph by defining object nodes and background semantic relationships, the annotation files of the images in the target dataset can be parsed to extract object information for each image; based on the object information, a scene graph corresponding to the image is constructed (the scene graph is used for the structured representation of the image); based on the scene graph, an adjacency matrix for each image is constructed (used to represent the relationship between objects in the image), and integrated to form an adjacency matrix set; all objects associated with labels in the label set of the target dataset are selected and defined as nodes of the knowledge graph; then, the background semantic relationships between objects are obtained (background semantic relationships are represented by positive integers v, the larger v is, the deeper the background semantic relationship between the two objects connected by this edge), and defined as edges of the knowledge graph; by iteratively traversing the label set in the target dataset, all objects associated with labels and background semantic relationships are added to the knowledge graph, and the weights of the background semantic relationships between labels are calculated through the adjacency matrix set.

[0082] Specifically, a scene graph is first constructed by extracting information describing object instances, relationships, and locations in each image of the target dataset using a script. Using the scene graph as an effective source of visual information for constructing a knowledge graph, the knowledge graph integrates rich semantic information from the images by transforming visual elements in the images and the relationships between them into a graph structure. This includes background semantic information needed to solve the problem.

[0083] In this optional embodiment, the object information includes object instance, relationship, and location; in the adjacency matrix, 1 indicates that two objects are semantically related in the background, and 0 indicates that two objects are semantically unrelated in the background.

[0084] In this optional embodiment, when parsing the scene semantic information in the target dataset based on the knowledge graph to obtain the background image dataset, the label of the adversarial attack target object can be selected and defined as a keyword, and the knowledge graph can be used to search for the object node corresponding to the keyword; based on the searched object node, a keyword index set is established; and images containing the keyword are searched in the target dataset to generate the background image dataset.

[0085] In this optional embodiment, when filtering background images that meet the index conditions in the background image dataset to generate diverse background datasets, background images can be filtered in the background image dataset based on the keyword index set; target objects are extracted from each image in the target dataset, and the target objects are fused with the background images to establish an adversarial attack target object image set; the realism of the fused image is enhanced based on a deep learning image fusion algorithm (a deep learning image fusion algorithm based on Poisson mixture loss), resulting in a diverse background dataset with improved robustness; this ensures that the target objects naturally blend into the new background, approximating the physical images of the real world, thereby improving the robustness of generating adversarial examples.

[0086] In this optional embodiment, during the fusion of the target object and the background image, the position of the target object in the background image is optimized using a scene semantic relationship reasoning model. This is because the improved scene semantic relationship reasoning model focuses on spatial semantic relationships to ensure that the position of the target object in the new background is reasonable.

[0087] In this optional embodiment, when training adversarial examples based on diverse background datasets to obtain the expected attack success rate of optimized adversarial examples, the diverse background datasets can be input into the expectation transformation framework to train the expected attack success rate of adversarial examples under different background conditions, and finally obtain the expected attack success rate of optimized adversarial examples; wherein, the diverse background datasets include several background conditions; by improving the robustness of adversarial examples under diverse background conditions, the adaptability of adversarial examples under diverse background conditions is enhanced.

[0088] Figure 3 An embodiment of the adversarial sample background semantic robustness enhancement system of the present invention is shown.

[0089] In this optional embodiment, the adversarial example background semantic robustness enhancement system includes:

[0090] The knowledge graph module 201 is used to acquire the target dataset and construct a knowledge graph by defining the semantic relationship between object nodes and background.

[0091] Background generation module 203 is used to parse scene semantic information in the target dataset based on knowledge graph to obtain background image dataset;

[0092] Background fusion module 205 is used to filter background images that meet the index conditions in the background image dataset to generate diverse background datasets;

[0093] The training optimization module 207 is used to train adversarial examples based on diverse background datasets to obtain the expected attack success rate of the optimized adversarial examples.

[0094] In this optional embodiment, the knowledge graph module 201 includes: a parsing and transformation module (not shown in the figure), a matrix construction module (not shown in the figure), a graph definition module (not shown in the figure), and an iterative calculation module (not shown in the figure). The parsing and transformation module is used to parse the annotation files of the images in the target dataset, extract object information for each image, and construct a scene graph corresponding to the image based on the object information. The matrix construction module constructs an adjacency matrix for each image based on the scene graph and integrates them to form an adjacency matrix set. The graph definition module is used to select all objects associated with labels within the label set in the target dataset and define them as nodes of the knowledge graph; then, it obtains the background semantic relationships between objects and defines them as edges of the knowledge graph. The iterative calculation module is used to iteratively traverse the label set in the target dataset, add all objects associated with labels and their background semantic relationships to the knowledge graph, and calculate the weights of the background semantic relationships between labels using the adjacency matrix set.

[0095] In this optional embodiment, the object information includes object instance, relationship, and location; in the adjacency matrix, 1 indicates that two objects are semantically related in the background, and 0 indicates that two objects are semantically unrelated in the background.

[0096] In this optional embodiment, the background generation module 203 includes: a node search module (not shown in the figure) and a background search module (not shown in the figure), wherein the node search module is used to select the tags of the adversarial attack target objects, define them as keywords, and use a knowledge graph to search for object nodes corresponding to the keywords; the background search module is used to establish a keyword index set based on the searched object nodes; and search for images containing keywords in the target dataset to generate a background image dataset.

[0097] In this optional embodiment, the background fusion module 205 includes: an index filtering module (not shown in the figure), a fusion establishment module (not shown in the figure), and an algorithm enhancement module (not shown in the figure). The index filtering module is used to filter background images in the background image dataset based on a keyword index set. The fusion establishment module is used to extract target objects from each image in the target dataset and fuse the target objects with the background images to establish an adversarial attack target object image set. The algorithm enhancement module is used to enhance the realism of the fused images based on a deep learning image fusion algorithm to obtain a diverse background dataset with improved robustness.

[0098] In this optional embodiment, during the process of fusing the target object with the background image, the position of the target object in the background image is optimized using a scene semantic relationship reasoning model.

[0099] In this optional embodiment, the training optimization module 207 includes: an input training module (not shown in the figure) and an enhancement optimization module (not shown in the figure), wherein the input training module is used to input the diverse background dataset into the desired transformation framework; the enhancement optimization module is used to train the expected attack success rate of the adversarial sample under different background conditions, and finally obtain the optimized expected attack success rate of the adversarial sample; wherein the diverse background dataset includes several background conditions.

[0100] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0101] like Figure 1 , Figure 2 As shown in this example, the adversarial example background semantic robustness enhancement method includes the following steps:

[0102] The first step is to construct a knowledge graph from the perspective of scene semantics.

[0103] First, a Python script parses the XML annotation files of images in the VrR-VG dataset to extract information describing object instances, relationships, and locations in each image. This information is then used to construct a scene graph for structured image representation. Using the scene graphs extracted from the VrR-VG dataset, an n*n adjacency matrix is ​​constructed for each image to represent the relationships between objects. Here, n is the number of objects in the image, rows and columns represent individual objects, and element values ​​represent relationships between objects: 1 indicates a close relationship between two objects (i.e., semantically related to the background), while 0 indicates a weak or no close relationship between two objects (i.e., semantically unrelated to the background).

[0104] In this way, the scene graphs extracted from the VrR-VG dataset are transformed into a set of adjacency matrices, which store the background semantic relationships between objects in the image in the form of matrices, laying the groundwork for the calculation and construction of the knowledge graph below.

[0105] The nodes of the knowledge graph G are defined as objects associated with all labels in the label set P of the VrR-VG dataset, and represented by labels. The edges of G are defined as the background semantic relationships between objects, represented by positive integers v; the larger v is, the deeper the background semantic relationship between the two objects connected by the edge. The algorithm for constructing the knowledge graph G using the adjacency matrix set M transformed from the scene graph (Algorithm 1: Knowledge Graph Construction Process) is as follows:

[0106] Input: Label set P = {p_i, p_(i+1)..., p_n}, adjacency matrix set M = {m_j, m_(j+1)..., m_n}

[0107] Output: Knowledge Graph G

[0108] step:

[0109] Initialize the knowledge graph G = φ, i = 1, j = 1, k = i + 1

[0110] While p_i∈P

[0111] k = i + 1

[0112] While m_j∈M

[0113] While p_k∈P

[0114] If m_j(p_i, p_k)==1

[0115] If node p_i or p_k does not exist in the knowledge graph G, add the missing node and initialize the edge v_ik = 1; if both nodes already exist, then v_ik = v_ik + 1.

[0116] End if

[0117] End while

[0118] End while

[0119] End while

[0120] Return G

[0121] In Algorithm 1, the label set P in the VrR-VG dataset is traversed in a bubble-like manner. All labels in the set are added to the knowledge graph G iteratively. During this process, the background semantic relationship weights v between the labels are calculated through the adjacency matrix set M. Finally, a knowledge graph containing the object nodes corresponding to all labels and the background semantic relationships between the objects is obtained.

[0122] The second step involves obtaining the background image set guided by the knowledge graph.

[0123] The first step is to extract background semantic relationships. Using the label of the adversarial attack target object as keywords, a knowledge graph is used to search for object nodes corresponding to that label. The purpose of this step is to find other object nodes that are semantically related to the target object, and these relationships reflect the possible background semantic environment in which the target object may appear. By calculating the edge weights in the knowledge graph, the top ten object nodes with the highest weights are selected. These nodes represent the background semantic information most relevant to the target object.

[0124] Next, the background image dataset is constructed. Using the previously obtained object node labels as a keyword index set, images containing these labeled objects are searched from the VrR-VG dataset. These images constitute the background image dataset, used to generate diverse backgrounds for adversarial examples.

[0125] The third step is to generate diverse background datasets based on the target.

[0126] The target object is separated from the original image through image matting and then fused with images from the background image dataset to create an adversarial attack target object image set with diverse backgrounds. During the fusion process, an improved scene semantic relationship inference model is adopted, focusing on spatial semantic relationships to ensure the target object's reasonable placement in the new background. A deep learning image fusion algorithm based on Poisson mixture loss is used to further enhance the realism of the fused image, ensuring the target object naturally integrates into the new background, approximating a real-world physical image, thereby improving the robustness of the generated adversarial examples. The core of this method lies in using highly realistic fused images to represent the distribution of objects in the real world, thus strengthening the adaptability of adversarial examples to complex backgrounds.

[0127] The processes in steps two and three are defined by formulas. Assume the target object of the adversarial attack is labeled p, G is a knowledge graph constructed based on the VrR-VG dataset, S(G, p) is the set of object nodes related to its background semantics obtained from the knowledge graph G using p as the keyword index, L(S) is the set of labels corresponding to S(G, p), D(L) is the background image dataset retrieved from the VrR-VG dataset based on L(S), and B(D) is the adversarial attack target object dataset under diverse backgrounds. Thus, the algorithm for generating diverse backgrounds (Algorithm 2: Generation Process of Diverse Background Image Datasets) can be defined as follows:

[0128] Input: adversarial attack target object label p, knowledge graph G, VrR-VG dataset X, label set P

[0129] Output: Dataset B of adversarial attack targets in diverse contexts

[0130] step:

[0131] Initialize a collection of object nodes related to the background semantics of the target object.

[0132] While g∈G

[0133] If G(p, g) exists

[0134] Add g node in S

[0135] While p^'∈P

[0136] If p^'==g

[0137] Add p^'in L

[0138] End if

[0139] End while

[0140] End if

[0141] End while

[0142] Let S(G, p) = S, L(S) = L

[0143] For x in X

[0144] If x contains any l in L

[0145] Add x in D

[0146] End if

[0147] End for

[0148] Let D(L) = D

[0149] The adversarial attack target object τ will be extracted from the target image.

[0150] For d in D

[0151] Add Merge(d, τ)in B

[0152] End for

[0153] Let B(D) = B

[0154] Return B(D)

[0155] Algorithm 2, guided by a knowledge graph, effectively guides the selection of background image datasets by utilizing the background semantic information contained within them, making these datasets more consistent with rich and realistic semantic environments. This process first uses the tags of the adversarial attack target objects as keywords to search the knowledge graph for object nodes corresponding to those tags and other semantically related object nodes. By calculating and selecting the top ten object nodes with the highest edge weights, a background image dataset is obtained that reflects both the rich relationships between the target object and the background and the diversity of the real world.

[0156] The fourth step is to train the framework to improve its robustness against adversarial examples.

[0157] The diverse background image datasets generated in step three are fed into the EOT (Expectation Over Transformation) framework for training and optimization of the expected attack success rate of adversarial examples under various backgrounds. This improves the robustness of adversarial examples in changing backgrounds and enhances their adaptability to complex and diverse real-world backgrounds, while maintaining good adversarial performance.

[0158] After completing the above operations, adversarial examples enhanced with enhanced background semantic robustness can be effectively used for adversarial attacks in physical worlds with complex backgrounds, directly revealing the hidden security issues of deep learning models in reality. Compared to existing methods, this method utilizes knowledge graphs to provide clear guidance for systematically selecting and generating representative backgrounds for adversarial example robustness training, improving the success rate of adversarial example attacks in complex backgrounds while significantly reducing the manual cost of image selection.

[0159] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores static and dynamic information data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the above method embodiments.

[0160] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0161] In addition, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0162] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0163] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0164] This invention is not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this invention is limited only by the appended claims.

Claims

1. A method for enhancing the robustness of adversarial example background semantics, characterized in that, include: Obtain the target dataset and construct a knowledge graph by defining the semantic relationship between object nodes and background. Based on knowledge graphs, the scene semantic information in the target dataset is analyzed to obtain the background image dataset; Based on the keyword index set, background images are filtered from the background image dataset; Extract the target object from each image in the target dataset and fuse the target object with the background image to establish an adversarial attack target object image set; Based on deep learning image fusion algorithms, the realism of the fused images is enhanced, resulting in a diverse background dataset with improved robustness. The diverse background dataset is input into the expectation transformation framework to train the expected attack success rate of the adversarial sample under different background conditions, and finally obtains the optimized expected attack success rate of the adversarial sample; wherein, the diverse background dataset includes several background conditions.

2. The adversarial example background semantic robustness enhancement method according to claim 1, characterized in that, The process of acquiring the target dataset and constructing a knowledge graph by defining the semantic relationships between object nodes and background includes: Parse the annotation files of the images in the target dataset to extract object information for each image; based on the object information, construct the scene graph corresponding to the image. Based on the scene graph, construct the adjacency matrix of each image and integrate them to form a set of adjacency matrices; Select all objects associated with the labels in the label set of the target dataset and define them as nodes of the knowledge graph; then obtain the background semantic relationships between the objects and define them as edges of the knowledge graph. By iteratively traversing the label set in the target dataset, all objects associated with the labels and their background semantic relationships are added to the knowledge graph, and the weights of the background semantic relationships between the labels are calculated using the adjacency matrix set.

3. The adversarial example background semantic robustness enhancement method according to claim 2, characterized in that, The object information includes object instance, relationship and location; in the adjacency matrix, 1 indicates that two objects are semantically related in the background, and 0 indicates that two objects are semantically unrelated in the background.

4. The adversarial example background semantic robustness enhancement method according to claim 1, characterized in that, The background image dataset obtained by parsing the scene semantic information in the target dataset based on the knowledge graph includes: Select tags for the target objects of the adversarial attack, define them as keywords, and use the knowledge graph to search for object nodes corresponding to the keywords; Based on the object nodes obtained from the search, a keyword index set is established; and images containing the keywords are searched in the target dataset to generate a background image dataset.

5. The adversarial example background semantic robustness enhancement method according to claim 1, characterized in that, During the process of fusing the target object with the background image, the position of the target object in the background image is optimized using a scene semantic relationship reasoning model.

6. A system for enhancing the robustness of adversarial example background semantics, characterized in that, include: The knowledge graph module is used to acquire the target dataset and construct a knowledge graph by defining the semantic relationship between object nodes and background. The background generation module is used to parse the scene semantic information in the target dataset based on the knowledge graph to obtain the background image dataset; The background fusion module is used to filter background images that meet the index conditions in the background image dataset and generate diverse background datasets. The training optimization module is used to train adversarial examples based on diverse background datasets to obtain the expected attack success rate of the optimized adversarial examples. The background blending module includes: The index filtering module is used to filter background images in the background image dataset based on a set of keyword indexes. The fusion module is used to extract target objects from each image in the target dataset and fuse the target objects with the background images to create an adversarial attack target object image set. The algorithm enhancement module is used to enhance the realism of the fused image based on the deep learning image fusion algorithm, and obtain a diverse background dataset with improved robustness. The training optimization module includes: The input training module is used to input the diverse background datasets into the desired transformation framework; An enhancement and optimization module is used to train the expected attack success rate of adversarial examples under different background conditions, and finally obtain the expected attack success rate of the optimized adversarial examples; wherein, the diverse background dataset includes several background conditions.

7. The adversarial example background semantic robustness enhancement system according to claim 6, characterized in that, The knowledge graph module includes: a parsing and transformation module, a matrix construction module, a graph definition module, and an iterative calculation module, wherein... The parsing and conversion module is used to parse the annotation files of the images in the target dataset, extract the object information of each image, and construct the scene graph corresponding to the image based on the object information. The matrix construction module constructs an adjacency matrix for each image based on the scene graph and integrates them to form an adjacency matrix set. The graph definition module is used to select all objects associated with labels in the label set of the target dataset and define them as nodes of the knowledge graph; then, it obtains the background semantic relationships between the objects and defines them as edges of the knowledge graph. The iterative calculation module is used to iteratively traverse the label set in the target dataset, add all the objects associated with the labels and the background semantic relationships to the knowledge graph, and calculate the background semantic relationship weights between the labels through the adjacency matrix set.

8. The adversarial example background semantic robustness enhancement system according to claim 7, characterized in that, The object information includes object instance, relationship and location; in the adjacency matrix, 1 indicates that two objects are semantically related in the background, and 0 indicates that two objects are semantically unrelated in the background.

9. The adversarial example background semantic robustness enhancement system according to claim 6, characterized in that, The background generation module includes: a node search module and a background search module, wherein... The node search module is used to select tags for the target objects of the adversarial attack, define them as keywords, and use the knowledge graph to search for object nodes corresponding to the keywords; The background search module is used to establish a keyword index set based on the object nodes obtained from the search; and to search for images containing the keywords in the target dataset to generate a background image dataset.

10. The adversarial example background semantic robustness enhancement system according to claim 9, characterized in that, During the process of fusing the target object with the background image, the position of the target object in the background image is optimized using a scene semantic relationship reasoning model.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images