Image recognition method and device based on cross-attention semantic and fingerprint bridging

Abstract

The application discloses a kind of based on cross attention's semantic and fingerprint bridging image recognition method and device, belong to image recognition technical field.Its method includes: the image to be measured is input image recognition model, and output image recognition result;In local structure branch, obtain the space mark sequence of reservation space position relationship;In global semantic branch, obtain global semantic vector;In high-frequency fingerprint branch, obtain high-frequency fingerprint;Space mark sequence, global semantic vector and high-frequency fingerprint are mapped to uniform feature space, and after mapping, global semantic mark and high-frequency fingerprint are spliced into semantic and fingerprint bridging mark set;Based on cross attention, obtain the space mark and bridging mark after two-way interaction and sequentially carry out fusion and pooling to it, obtain image-level representation;Image-level representation is classified, and output image recognition result.The application can effectively align and two-way interaction to local structure, global semantic and high-frequency fingerprint information.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to an image recognition method and apparatus based on semantic and fingerprint bridging of cross-attention. Background Technology

[0002] With the development of generative adversarial networks and diffusion models, generated images are becoming increasingly similar to real images in terms of texture, lighting, object shape, and overall semantics, making them increasingly difficult to recognize with the naked eye.

[0003] Existing generative image recognition methods typically utilize clues such as spatial domain artifacts, frequency statistics, local pixel relationships, pre-trained semantic representations, or diffusion reconstruction responses to output generation probabilities or binary classification results for input images.

[0004] Spatial domain, frequency, and local relational cues can reflect subtle traces left by the generation process, but they are easily affected by post-processing such as compression, scaling, blurring, and noise. Furthermore, differences in network structure, upsampling methods, and sampling processes among different generators can weaken the recognition model's adaptability to unknown generators. Pre-trained visual language models have strong semantic transfer capabilities, but they are insufficient in characterizing subtle noise, local textures, and imaging residuals. Diffusion reconstruction methods usually require additional inversion or denoising processes, resulting in significant computational overhead.

[0005] Existing multi-source fusion methods often use direct splicing to process semantic, spatial, and high-frequency fingerprint features, making it difficult to explicitly establish the correspondence between global semantics and local responses, which can easily lead to unbalanced information utilization. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide an image recognition method and device based on cross-attention semantic and fingerprint bridging, which can effectively align and bidirectionally interact local structure, global semantics and high-frequency fingerprint information, and improve the recognition stability of unknown generators and degraded images.

[0007] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0008] On one hand, the present invention provides an image recognition method based on semantic and fingerprint bridging of cross-attention, comprising:

[0009] Acquire the image to be tested;

[0010] The image to be tested is input into the image recognition model, and the image recognition result is output.

[0011] The processing steps of the image recognition model include:

[0012] In the local structure branch, the image to be tested is subjected to residual construction to obtain local structure residuals. The local structure residuals are then subjected to hierarchical convolutional encoding through a spatial feature coding network to obtain a spatial label sequence that preserves spatial positional relationships.

[0013] In the global semantic branch, the image to be tested is preprocessed to obtain a preprocessed image that meets the input specifications. The semantic image coding network is used to extract semantics from the preprocessed image to obtain a global semantic vector.

[0014] In the high-frequency fingerprint branch, the high-frequency residual map of the image under test is extracted by at least one high-pass filter kernel, and the high-frequency residual map is encoded by a high-frequency fingerprint encoding network to obtain a high-frequency fingerprint;

[0015] The spatial label sequence, global semantic vector, and high-frequency fingerprint are mapped to a unified feature space, and the mapped global semantic label and high-frequency fingerprint are concatenated into a semantic and fingerprint bridging label set.

[0016] Based on cross-attention, the forward interaction uses the mapped spatial label sequence as the query and the semantic and fingerprint bridging label set as the key and value to update the spatial label sequence. The reverse interaction uses the semantic and fingerprint bridging label set as the query and the updated spatial label as the key and value to update the semantic and fingerprint bridging label set, thus obtaining the spatial label and bridging label after bidirectional interaction.

[0017] The spatial markers and bridging markers after the bidirectional interaction are sequentially fused and pooled to obtain an image-level representation;

[0018] The image-level representation is classified, and the image recognition result is output.

[0019] Optionally, in the local structure branch, residual construction is performed on the image to be tested to obtain local structure residuals. These local structure residuals are then subjected to hierarchical convolutional encoding via a spatial feature encoding network to obtain a spatially labeled sequence that preserves spatial relationships, including:

[0020] The image to be tested is subjected to nearest neighbor downsampling and nearest neighbor upsampling in sequence to obtain the restored image;

[0021] Based on the difference between the image to be tested and the restored image, the local structural residual is obtained;

[0022] The local structural residuals are input into a spatial feature encoding network. In each stage, the local structural residuals are sequentially subjected to depthwise convolution, layer normalization, pointwise convolution, GELU activation, and global response normalization to obtain a spatial feature map. The spatial feature map is then flattened into a spatial label sequence that preserves spatial positional relationships. The feature channels in each stage are sequentially expanded.

[0023] Optionally, the local structural residual is expressed as:

[0024] ;

[0025] in, Indicates local structural residuals; Indicates the residual scaling factor; Represents the image to be tested; Indicates nearest neighbor upsampling; This indicates nearest neighbor downsampling;

[0026] The spatial label sequence that preserves spatial location relationships is represented as follows:

[0027] ;

[0028] in, This represents a sequence of spatial markers that preserves spatial relationships. Represents a spatial feature encoding network; The learnable parameters of a spatial feature coding network; Indicates the number of spatial locations; The feature dimension representing the spatial label; It represents the set of real numbers.

[0029] Optionally, the preprocessed image that satisfies the input specification is represented as:

[0030] ;

[0031] in, This represents a preprocessed image that meets the input specifications. Represents the image to be tested; These represent the channel mean and channel standard deviation used by CLIP, respectively. This means adjusting the input image to 224×224;

[0032] The global semantic vector is represented as follows:

[0033] ;

[0034] in, Represents a global semantic vector; Represents a semantic image coding network; These represent the parameters of a semantic image coding network; Represents the set of real numbers; Represents the loss function Parameters relative to semantic image coding networks The gradient.

[0035] Optionally, the high-frequency residual map is represented as follows:

[0036] ;

[0037] in, Indicates the first High-frequency residual map extracted by a high-pass filter kernel; Indicates the first One high-pass filter core; Represents the image to be tested; Represents two-dimensional convolution;

[0038] The high-frequency fingerprint is represented as follows:

[0039] ;

[0040] in, Indicates high-frequency fingerprints; Represents a high-frequency fingerprint coding network; These represent the high-frequency residual images extracted by the 1st, 2nd, and 3rd high-pass filter kernels, respectively. It represents the set of real numbers.

[0041] Optionally, the mapped spatial label sequence is represented as:

[0042] ;

[0043] in, This represents the mapped spatial label sequence; This represents a sequence of spatial markers that preserves spatial relationships. The learnable projection matrix representing the local structural branches; Representation layer normalization;

[0044] The mapped global semantic tag is represented as follows:

[0045] ;

[0046] in, This represents the mapped global semantic tagging; Represents a global semantic vector; The learnable projection matrix representing the global semantic branch;

[0047] The mapped high-frequency fingerprint is represented as follows:

[0048] ;

[0049] in, This represents the mapped high-frequency fingerprint; Indicates high-frequency fingerprints; The learnable projection matrix representing the high-frequency fingerprint branch;

[0050] The semantic and fingerprint bridging tag set is represented as follows:

[0051] ;

[0052] in, This represents a set of semantic and fingerprint bridging tags; This indicates splicing along the direction of the marker sequence.

[0053] Optionally, the spatial markers after the bidirectional interaction are represented as follows:

[0054] ;

[0055] in, Indicates spatial markers after two-way interaction; This represents the mapped spatial label sequence; This represents a set of semantic and fingerprint bridging tags; Representation layer normalization; This indicates cross attention; This represents the projection matrix used to generate the spatial query; This represents the projection matrix used to generate the bridging key; The projection matrix that generates the bridging values;

[0056] The bridging marker after the bidirectional interaction is represented as follows:

[0057] ;

[0058] in, This indicates a bridging marker after two-way interaction; This represents the projection matrix used to generate the bridging query; This represents the projection matrix for generating the updated space keys; This represents the projection matrix used to generate the updated spatial values.

[0059] Optionally, the image-level representation is expressed as:

[0060] ;

[0061] in, Represents an image-level representation; This indicates cross attention; This indicates a learnable pooled query; This indicates a bridging marker after two-way interaction; Indicates spatial markers after two-way interaction; This indicates splicing along the direction of the marker sequence.

[0062] Optionally, the image recognition result is obtained through the image generation probability, which is obtained by classifying the image-level representation using a preset classification function;

[0063] The image generation probability is expressed as:

[0064] ;

[0065] In the formula, ;

[0066] in, Indicates the probability of image generation; Represents the sigmoid function; This indicates the unnormalized classification score; These represent the weights and biases of the first linear layer, respectively. Represents an image-level representation; Representation layer normalization; These represent the weights and biases of the second linear layer, respectively. Indicates nonlinear activation; This indicates random deactivation during the training phase.

[0067] On the other hand, the present invention provides an image recognition device based on semantic and fingerprint bridging of cross-attention, comprising:

[0068] The image acquisition module is used to: acquire the image to be tested;

[0069] The image recognition module is used to: input the image to be tested into the image recognition model and output the image recognition result;

[0070] The model processing module is used for: the processing steps of the image recognition model include:

[0071] In the local structure branch, the image to be tested is subjected to residual construction to obtain local structure residuals. The local structure residuals are then subjected to hierarchical convolutional encoding through a spatial feature coding network to obtain a spatial label sequence that preserves spatial positional relationships.

[0072] In the global semantic branch, the image to be tested is preprocessed to obtain a preprocessed image that meets the input specifications. The semantic image coding network is used to extract semantics from the preprocessed image to obtain a global semantic vector.

[0073] In the high-frequency fingerprint branch, the high-frequency residual map of the image under test is extracted by at least one high-pass filter kernel, and the high-frequency residual map is encoded by a high-frequency fingerprint encoding network to obtain a high-frequency fingerprint;

[0074] The spatial label sequence, global semantic vector, and high-frequency fingerprint are mapped to a unified feature space, and the mapped global semantic label and high-frequency fingerprint are concatenated into a semantic and fingerprint bridging label set.

[0075] Based on cross-attention, the forward interaction uses the mapped spatial label sequence as the query and the semantic and fingerprint bridging label set as the key and value to update the spatial label sequence. The reverse interaction uses the semantic and fingerprint bridging label set as the query and the updated spatial label as the key and value to update the semantic and fingerprint bridging label set, thus obtaining the spatial label and bridging label after bidirectional interaction.

[0076] The spatial markers and bridging markers after the bidirectional interaction are sequentially fused and pooled to obtain an image-level representation;

[0077] The image-level representation is classified, and the image recognition result is output.

[0078] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0079] This invention's local structural residual and spatial feature encoding network retains upsampling-related local pixel dependencies, structural distortions, and edge continuity information, avoiding premature global averaging that dilutes local artifacts. Using the local structural residual and spatial feature encoding network further highlights neighborhood structural differences. The semantic image encoding network forms a relatively stable global semantic representation, while the high-pass filter kernel and high-frequency fingerprint encoding network form a high-frequency fingerprint with weak content relation. Two auxiliary features compensate for the spatial feature encoding network's shortcomings in cross-domain semantics and fine-grained statistics. The semantic branch uses a frozen image encoder, while the high-frequency branch uses a fixed high-pass kernel and its lightweight encoder. Three types of heterogeneous features are first mapped to a unified latent space, and then the global semantic label and high-frequency fingerprint label constitute semantic and fingerprint bridging information. Bidirectional cross-attention completes the mutual reading between spatial labels and bridging labels, overcoming the problems of direct splicing making it difficult to explicitly establish correspondences and uneven information utilization. Attention pooling uses learnable queries to adaptively aggregate discriminative evidence from fused labels, enabling image-level representation to focus on unevenly distributed local artifacts without performing diffusion model inversion or diffusion reconstruction during the inference stage. Attached Figure Description

[0080] Figure 1 A flowchart illustrating the image recognition method based on semantic and fingerprint bridging based on cross-attention provided in an embodiment of the present invention;

[0081] Figure 2 A schematic diagram of the processing flow of local structural branches provided in an embodiment of the present invention;

[0082] Figure 3A schematic diagram illustrating the processing flow of global semantic branches and high-frequency fingerprint branches provided in an embodiment of the present invention;

[0083] Figure 4 This is a flowchart illustrating semantic and fingerprint bridging, bidirectional cross-attention, attention pooling, and classification processing provided in an embodiment of the present invention.

[0084] Figure 5 This is a schematic diagram illustrating the process of model training and image recognition in an embodiment of the present invention. Detailed Implementation

[0085] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0086] The term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0087] Example 1

[0088] like Figure 1 As shown, this embodiment introduces an image recognition method based on semantic and fingerprint bridging using cross-attention. The network structure, processing procedure, parameter states, training configuration, and technical effects are given. Label y=0 represents a real image, and label y=1 represents a generated image. In the testing phase, only the input image is needed; diffusion model inversion or diffusion reconstruction is not performed. The image to be tested is fed in parallel into the local structure branch, global semantic branch, and high-frequency fingerprint branch, and then unified projection, bidirectional cross-attention based on semantic and fingerprint bridging, attention pooling, and binary classification are performed sequentially. The three branches represent the local spatial structure, global semantic layout, and high-frequency fingerprint, respectively. During the fusion process, the global semantics provides a common overall reference for the local spatial response and high-frequency fingerprint, enabling the three types of evidence to establish a correspondence that can be adjusted by attention weights. The method includes the following steps:

[0089] Step 1: Obtain the image to be tested, specifically:

[0090] Obtain the image to be tested.

[0091] The image to be tested is input into the image recognition model, and the image recognition result is output.

[0092] The image to be tested is further resized and normalized before being input into the image recognition model.

[0093] The processing steps of the image recognition model are as follows.

[0094] Step 2: Image processing of local structural branches, specifically:

[0095] In the local structure branch, residual construction is performed on the image to be tested to obtain local structure residuals. The local structure residuals are then subjected to hierarchical convolutional encoding through a spatial feature encoding network to obtain a spatial label sequence that preserves spatial positional relationships.

[0096] The process deals with a red, green, and blue (RGB) image to be tested. When generative networks progressively enlarge low-resolution features into high-resolution images, they may create local pixel dependencies that differ from natural imaging; when RGB values ​​are used directly, these differences are easily masked by scene content and color information.

[0097] To highlight neighborhood relationships, such as Figure 2 As shown, the image to be tested By sequentially performing nearest neighbor downsampling and nearest neighbor upsampling, the image is restored to its original size, resulting in the restored image.

[0098] Based on the difference between the image to be tested and the reconstructed image, the local structural residual is obtained. Represented as:

[0099] ;

[0100] in, Indicates the residual scaling factor; This indicates nearest neighbor upsampling, responsible for restoring the low-resolution representation to the level of the image under test. Same space dimensions; This indicates nearest neighbor downsampling, responsible for forming a low-resolution representation using discrete neighborhood sampling.

[0101] The difference within square brackets represents local changes in the original image that cannot be preserved by nearest-neighbor downsampling and upsampling. This process does not introduce learnable parameters; its purpose is to weaken the intensity of the original content and highlight the local pixel relationships related to upsampling, providing a more direct structural artifact input to the backbone encoder. The local structural residuals in this embodiment... This represents the residuals of Neighboring Pixel Relationships (NPR).

[0102] Local structural residuals Input spatial feature encoding network, and process local structural residuals at each stage. The spatial feature map is obtained by sequentially performing depthwise convolution, layer normalization, pointwise convolution, Gaussian Error Linear Unit (GELU) activation, and global response normalization. This spatial feature map is then flattened into a spatially labeled sequence that preserves spatial relationships; the feature channels at each stage are progressively enlarged. In this embodiment, the spatial feature encoding network is a second-generation convolutional neural network (ConvNeXt V2-Base) backbone encoder. .

[0103] Local structural residuals Input ConvNeXt V2-Base trunk encoder The main trunk affects the local structural residuals. Hierarchical convolutional encoding is performed to output spatial feature maps, which are then flattened into a spatial label sequence, without performing global averaging at this point. This preserves the spatial relationships within the spatial label sequence. Represented as:

[0104] ;

[0105] in, This represents the spatial feature coding network, namely the ConvNeXt V2 backbone encoder; The learnable parameters of a spatial feature coding network; Indicates the number of spatial locations; The feature dimension representing the spatial label; It represents the set of real numbers.

[0106] In this embodiment, the final spatial feature map is 7×7×1024, therefore , The purpose of retaining 49 spatial locations is to prevent local anomalies that only appear in background textures, object edges, or repeating structures from being prematurely averaged, and to provide a basis for subsequent bridging marker reading or verification of forensic evidence by spatial location.

[0107] The ConvNeXt V2-Base backbone encoder consists of four stages, each comprising depthwise convolution, layer normalization, pointwise mapping, GELU activation, and global response normalization. The depth and channel variations of each stage are shown in Table 1.

[0108] Table 1. Depth and Channel Variations at Different Stages of the ConvNeXt V2-Base Backbone Encoder

[0109]

[0110] The four stages in Table 1 progressively expand the feature channels and form a hierarchical representation, with the final stage's 1024-channel features preserving local responses in a 7×7 spatial location. This structure allows the NPR residuals to be gradually transformed from pixel-level differences into spatial markers usable by the attention module.

[0111] Step 3: Image processing of the global semantic branch, specifically:

[0112] In the global semantic branch, the image to be tested is preprocessed to obtain a preprocessed image that meets the input specifications. The semantic image coding network is then used to extract semantics from the preprocessed image to obtain the global semantic vector.

[0113] Local structural branches focus on low-level and meso-level structures, but their responses are easily altered by compression, scaling, or generator structure. Therefore, the semantic image encoding network is a Contrastive Language-Image Pre-Training with ResNet-50 (CLIP RN50) image encoder based on a 50-layer residual network. This step utilizes the CLIP RN50 image encoder to extract global semantics that can characterize global content distribution, object relationships, and semantic layout.

[0114] like Figure 3 As shown, the image to be tested By directly performing size adjustments and channel normalization according to CLIP's pre-training specifications without going through local structural branches, a preprocessed image that meets the input specifications can be obtained. Represented as:

[0115] ;

[0116] in, These represent the channel mean and channel standard deviation used by CLIP, respectively. This means adjusting the input image to 224×224.

[0117] Size and normalization rules preserve the input distribution of the pre-trained image encoder, preventing changes in input norms from destroying its original semantic representation, and global semantic vectors. 1024-dimensional global semantic vector Represented as:

[0118] ;

[0119] in, This refers to the semantic image coding network, namely the CLIP RN50 image encoder; These represent the parameters of a semantic image coding network; Represents the loss function Parameters relative to semantic image coding networks gradient, express Compared to The gradient is zero, meaning this branch remains frozen during training. The freezing operation addresses the problem of over-fine-tuning on data with only a single trained generator, which weakens the open-domain semantic space. Instead of directly providing a true or false conclusion, it serves as semantic bridging information for subsequent fusion, providing a unified global context for local structures and high-frequency fingerprints.

[0120] Step 4: Image processing of high-frequency fingerprint branches, specifically:

[0121] In the high-frequency fingerprint branch, the high-frequency residual map of the image under test is extracted by at least one high-pass filter kernel, and the high-frequency residual map is encoded by a high-frequency fingerprint coding network to obtain the high-frequency fingerprint.

[0122] Global semantic features cannot directly describe subtle noise, local texture, or imaging residuals. They are used to provide supplementary evidence that is less relevant to image content, such as... Figure 3 As shown, this process uses three fixed 5×5 Spatial Rich Model (SRM) high-pass filter kernels to observe local prediction errors. High-frequency residual plot extracted by a high-pass filter kernel Represented as:

[0123] ;

[0124] in, Indicates the first One high-pass filter core; This represents a two-dimensional convolution.

[0125] In this embodiment, We take 1, 2, and 3 in sequence. The three filter kernels respond differently to local prediction errors, thus preserving noise, edges, and fine-grained statistical changes from three directions; fixing the filter kernel does not participate in parameter updates, which reduces the risk of this branch memorizing specific textures on a single generator training set.

[0126] High-frequency fingerprint 128-dimensional, high-frequency fingerprint Represented as:

[0127] ;

[0128] in, Represents a high-frequency fingerprint coding network; These represent the high-frequency residual images extracted by the 1st, 2nd, and 3rd high-pass filter kernels, respectively. This indicates that the three types of residual maps are spliced ​​along the channel dimension.

[0129] High-frequency fingerprint coding network according to Figure 3 The diagram shows the sequential application of convolution, group normalization, Sigmoid Linear Unit (SiLU) activation, and global pooling to compress the three-channel residual map into an image-level high-frequency fingerprint. This branch differs from the local structure branch in that the local structure branch emphasizes local pixel dependencies caused by upsampling, while this branch emphasizes high-frequency prediction errors; both provide evidence at different scales.

[0130] Step 5: Alignment of the three types of heterogeneous features, specifically:

[0131] spatial label sequence Global semantic vector and high-frequency fingerprint Mapping to a unified feature space yields the mapped spatial label sequence. Global semantic vector and high-frequency fingerprint Mapped global semantic tags and high-frequency fingerprint Concatenate into a set of semantic and fingerprint bridging tags .

[0132] Spatial marker sequence Global semantic vector and high-frequency fingerprint Sequences with different lengths, feature dimensions, and statistical distributions cannot be directly compared within the same scaled dot product attention. Therefore, layer normalization and linear projection are performed separately to map the spatially labeled sequences. Represented as:

[0133] ;

[0134] in, The learnable projection matrix representing the local structural branches; Representation layer normalization.

[0135] Mapped global semantic tags Represented as:

[0136] ;

[0137] in, The learnable projection matrix represents the global semantic branch.

[0138] Mapped high-frequency fingerprint Represented as:

[0139] ;

[0140] in, The learnable projection matrix represents the high-frequency fingerprint branch.

[0141] Semantic and fingerprint bridging tag set Represented as:

[0142] ;

[0143] in, This indicates splicing along the direction of the marker sequence.

[0144] The dimension of the target latent space, i.e., the dimension of the attention feature. Using 512, normalization reduces the difference in branch scale, and linear projection makes the three types of heterogeneous features have a consistent feature dimension, thus establishing a common coordinate for the subsequent dot product comparison of bidirectional attention.

[0145] Table 2 shows the input, output, and latent space dimensions of each branch. The spatial branch retains 49 positions. The global semantic branch and the high-frequency fingerprint branch form global semantic tags and high-frequency fingerprint tags, respectively. The two form a semantic and fingerprint bridging tag set, which is bidirectionally interacted with the spatial tags to obtain 51 512-dimensional fusion tags.

[0146] Table 2. Three-branch processing and feature size changes

[0147]

[0148] Step Six: Bridging semantics and fingerprints based on cross-attention, specifically:

[0149] Based on cross-attention, the forward interaction uses the mapped spatial label sequence as the query and the semantic and fingerprint bridging label set as the key and value to update the spatial label sequence. The reverse interaction uses the semantic and fingerprint bridging label set as the query and the updated spatial label as the key and value to update the semantic and fingerprint bridging label set, thus obtaining the spatial label and bridging label after bidirectional interaction.

[0150] Direct splicing cannot clearly define which type of auxiliary information should be read at a specific spatial location, nor can it enable global semantics and high-frequency fingerprints to reverse verify which local locations the discrimination criteria originate from. For example... Figure 4 As shown, this embodiment combines global semantic tags and high-frequency fingerprint tags to form a semantic and fingerprint bridging tag set, connecting the local spatial structure, global semantics, and high-frequency fingerprints. It uses standard scaled dot product attention as the basic operator, expressed as:

[0151] ;

[0152] in, This indicates cross attention; These represent the query matrix, key matrix, and value matrix, respectively. Indicates the dimension of the attention feature; This represents the softmax function, which converts the score of each query relative to each key into weights, and the weights are proportional to... Multiplying them yields the aggregated information retrieved from the query; This indicates the matrix transpose.

[0153] The forward interaction uses the 49 mapped spatial markers as queries and the mapped global semantic markers and high-frequency fingerprint markers as keys and values. This allows each local location to read high-frequency fingerprint evidence under the overall reference provided by the global semantics, resulting in the spatial markers after the bidirectional interaction. Represented as:

[0154] ;

[0155] in, This represents the projection matrix used to generate the spatial query; This represents the projection matrix used to generate the bridging key; The projection matrix that generates the bridging values; Cross-attention refers to the information read from global semantics and high-frequency fingerprints by spatial labels; the addition of residuals preserves the original spatial response, and this process enables local structural anomalies to be judged for their evidentiary significance by combining global semantics and high-frequency fingerprint evidence.

[0156] The reverse interaction then uses the mapped global semantic tags and high-frequency fingerprint tags as queries, and the updated 49 spatial tags as keys and values, to allow the global semantic tags and high-frequency fingerprint tags to return to the specific local context to find supporting evidence, resulting in the bridging tags after the bidirectional interaction. Represented as:

[0157] ;

[0158] in, This represents the projection matrix used to generate the bridging query; This represents the projection matrix for generating the updated space keys; This represents the projection matrix that generates the updated spatial values; Cross attention refers to the information read from spatial labels by global semantics and high-frequency fingerprints; reverse reading subjectes global semantic labels and high-frequency fingerprint labels to spatial response constraints, preventing bridging information from being separated from local image evidence and independently determining the classification result.

[0159] Forward and reverse interactions together form a closed loop for reading spatial evidence semantic-fingerprint bridging information and bridging marker reverse lookup of local support.

[0160] Step 7: Attention pooling and classification, specifically:

[0161] The spatial tags and bridging tags after bidirectional interaction are successively fused and pooled to obtain an image-level representation.

[0162] After completing the two-way interaction, mark the space after the two-way interaction. Bridge markers after bidirectional interaction Concatenate along the direction of the marker sequence.

[0163] Since generated artifacts may only exist in a few spatial locations, directly taking the arithmetic mean of the 51 markers could still dilute weak local responses. This embodiment introduces a learnable pooling query. Adaptive aggregation of image-level representations from fused tags; image-level representations 512-dimensional, image-level representation Represented as:

[0164] ;

[0165] in, ; This indicates splicing along the direction of the marker sequence.

[0166] Pooled queries learn the importance of each fused label for true / false classification during training, enabling the model to focus on reading more discriminative spatial labels, global semantic vectors, or high-frequency fingerprint responses.

[0167] Image-level representation The system performs classification and outputs image recognition results.

[0168] Image recognition results are obtained through image generation probabilities, which are then obtained by classifying the image-level representation using a preset classification function. This represents the probability that the input image belongs to the generated class, i.e., the image generation probability. Represented as:

[0169] ;

[0170] In the formula, ;

[0171] in, Represents the sigmoid function; This indicates the unnormalized classification score; These represent the weights and biases of the first linear layer, which maps the 512-dimensional representation to 256 dimensions. These represent the weights and biases of the second linear layer, which maps the 256-dimensional representation to a logit. Indicates nonlinear activation; This indicates random deactivation during the training phase.

[0172] A threshold of 0.5 is used for identification. When When the value reaches 0.5, the image to be tested is determined to be a generated image; when... When the probability is below 0.5, the image to be tested is classified as a real image. This process converts the continuous generation probability into an image-level binary classification result; at the same time, it can directly retain... The predicted score is used to calculate the average precision (AP) independent of a fixed threshold.

[0173] Step 8: Training the image recognition model, specifically:

[0174] like Figure 5 As shown, the training set is denoted as ,in, This represents the number of training samples; For the first Image; for The corresponding tags Represents a real image. This represents the generated image. Each training sample undergoes the above steps for three-way feature extraction, semantic and fingerprint bridging based on cross-attention, attention pooling, and classification. Probabilities are generated using binary cross-entropy constraints, and the binary cross-entropy loss for a single sample is calculated. Represented as:

[0175]

[0176] in, Indicates the true label; This represents the probability of image generation predicted by the model; This represents the logarithmic function. When... At that time, the first thing to promote Increase; when At that time, the second promotion This reduces the predicted score of the generated image to 1 and the predicted score of the real image to 0.

[0177] The model parameters, including the ConvNeXt V2 backbone encoder, SRM high-pass filter kernel, three-way projection, bidirectional cross-attention, attention pooling, and classification head parameters, are expressed as follows:

[0178]

[0179] in,, This represents the learnable portion excluding the frozen CLIP encoder and the fixed SRM filter core; This indicates the search for the parameter that minimizes the average loss. .

[0180] During training, the CLIP RN50 parameters are not updated, the three SRM high-pass filter kernels remain fixed, but the SRM lightweight encoder participates in the update.

[0181] This embodiment uses a subset of the Stable Diffusion Version 1.4 (Stable Diffusion v1.4) model from the GenImage: AMillion-Scale Benchmark for Detecting AI-Generated Image (GenImage) dataset for training. During testing, unseen generators are evaluated to verify cross-generator generalization. The training inputs, optimizer, and numerical computation settings are shown in Table 3.

[0182] Table 3 Image Processing and Model Training Configuration

[0183]

[0184] Training images are first scaled to 256×256 and then randomly cropped to 224×224. This introduces positional variations while satisfying the three-branch input scale. Random horizontal flipping increases the orientation variation of the training images. The backbone learning rate is lower than the master learning rate of the newly built fusion module, allowing the loaded master intervention training representation to be updated with smaller steps. 16-bit Brain Floating Point Format (BF16), 32-bit TensorFloat-32 (TF32), and Channels Last Memory Format (channels-last) are used for numerical and memory acceleration during the training phase without altering the recognition process.

[0185] Example 2

[0186] Based on Example 1, this example presents an experimental case of an image recognition method based on semantic and fingerprint bridging of cross-attention:

[0187] To verify the effectiveness of the technical solution of this invention, Stable Diffusion v1.4 was used as the training generator, and various models were tested on real images, including Big Generative Adversarial Network (BigGAN), Guided Language to Image Diffusion for Generation and Editing (GLIDE), Ablated Diffusion Model (ADM), Wukong Large-Scale Text-to-Image Diffusion Model (Wukong), Vector Quantized Diffusion Model (VQDM), Stable Diffusion Version 1.5 (Stable Diffusion v1.5), Midjourney Text-to-Image Model (Midjourney), and Dali-E 2 Text-to-Image Model (DALL-E). 2) Validate the generated images and calculate the average accuracy and average precision of each method on the same set of generators. The comparison results are shown in Table 4.

[0188] Table 4 compares the method of this invention with the following ten representative generative image recognition methods: Convolutional Neural Network Generated Images Are Surprisingly Easy to Spot... forNow (CNNSpot), Fusing Global and Local Features for Generalized AI-Synthesized Image Detection (Fusing), Leveraging Frequency Analysis for Deep Fake Image Recognition (FreDect), Diffusion Reconstruction Error (DIRE), Towards Universal Fake Image Detectors ThatGeneralize Across Generative Models (UnivFD), Neighboring Pixel Relationships (NPR), Forgery-Aware Adaptive Transformer (FatFormer), AI-Generated Image Detector with Hybrid Features (AIDE), and Background-Noise-Driven Diffusion Generated Image Detection. Detection of Diffusion-Generated Images (BND) and Multi-Scale Decoupled Contrastive Learning (MS-DCL).

[0189] Table 4. Average performance comparison of representative generative image recognition methods

[0190]

[0191] As shown in Table 4, the method of this invention achieves an average accuracy of 92.31% and an average precision of 96.62%. The average accuracy is higher than MS-DCL's 86.96% and NPR's 90.58%, and the average precision is higher than AIDE's 94.21% and BND's 94.30%, demonstrating the best performance among ten popular methods. These results correspond to a setting where multiple unseen generators were tested after training on only a subset of generators, indicating that collaborative modeling of local structure, global semantics, and high-frequency fingerprints can improve the stability and ranking ability of cross-generator recognition.

[0192] Under degradation conditions such as compression, Gaussian blur, and Gaussian noise, the method of this invention still maintains relatively stable recognition ability when faced with mild to moderate perturbations. This result shows that the collaborative modeling of multidimensional evidence can alleviate the problem of single low-level features being destroyed by post-processing. Therefore, this invention has good stability and generalization performance and is more suitable for complex situations in real-world environments.

[0193] Example 3

[0194] Based on Embodiment 1 or 2, this embodiment introduces an image recognition device based on semantic and fingerprint bridging of cross-attention, comprising:

[0195] The image acquisition module is used to: acquire the image to be tested;

[0196] The image recognition module is used to: input the image to be tested into the image recognition model and output the image recognition result;

[0197] The model processing module is used for: the processing steps of the image recognition model include:

[0198] In the local structure branch, the image to be tested is subjected to residual construction to obtain local structure residuals. The local structure residuals are then subjected to hierarchical convolutional encoding through a spatial feature coding network to obtain a spatial label sequence that preserves spatial positional relationships.

[0199] In the global semantic branch, the image to be tested is preprocessed to obtain a preprocessed image that meets the input specifications. The semantic image coding network is used to extract semantics from the preprocessed image to obtain a global semantic vector.

[0200] In the high-frequency fingerprint branch, the high-frequency residual map of the image under test is extracted by at least one high-pass filter kernel, and the high-frequency residual map is encoded by a high-frequency fingerprint encoding network to obtain a high-frequency fingerprint;

[0201] The spatial label sequence, global semantic vector, and high-frequency fingerprint are mapped to a unified feature space, and the mapped global semantic label and high-frequency fingerprint are concatenated into a semantic and fingerprint bridging label set.

[0202] Based on cross-attention, the forward interaction uses the mapped spatial label sequence as the query and the semantic and fingerprint bridging label set as the key and value to update the spatial label sequence. The reverse interaction uses the semantic and fingerprint bridging label set as the query and the updated spatial label as the key and value to update the semantic and fingerprint bridging label set, thus obtaining the spatial label and bridging label after bidirectional interaction.

[0203] The spatial markers and bridging markers after the bidirectional interaction are sequentially fused and pooled to obtain an image-level representation;

[0204] The image-level representation is classified, and the image recognition result is output.

[0205] The specific functions of each module described above are explained in the relevant content of the method in Embodiment 1, and will not be repeated here.

[0206] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0207] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0208] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0209] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0210] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. An image recognition method based on semantic and fingerprint bridging using cross-attention, characterized in that, include: Acquire the image to be tested; The image to be tested is input into the image recognition model, and the image recognition result is output. The processing steps of the image recognition model include: In the local structure branch, the image to be tested is subjected to residual construction to obtain local structure residuals. The local structure residuals are then subjected to hierarchical convolutional encoding through a spatial feature coding network to obtain a spatial label sequence that preserves spatial positional relationships. In the global semantic branch, the image to be tested is preprocessed to obtain a preprocessed image that meets the input specifications. The semantic image coding network is used to extract semantics from the preprocessed image to obtain a global semantic vector. In the high-frequency fingerprint branch, the high-frequency residual map of the image under test is extracted by at least one high-pass filter kernel, and the high-frequency residual map is encoded by a high-frequency fingerprint encoding network to obtain a high-frequency fingerprint; The spatial label sequence, global semantic vector, and high-frequency fingerprint are mapped to a unified feature space, and the mapped global semantic label and high-frequency fingerprint are concatenated into a semantic and fingerprint bridging label set. Based on cross-attention, the forward interaction uses the mapped spatial label sequence as the query and the semantic and fingerprint bridging label set as the key and value to update the spatial label sequence. The reverse interaction uses the semantic and fingerprint bridging label set as the query and the updated spatial label as the key and value to update the semantic and fingerprint bridging label set, thus obtaining the spatial label and bridging label after bidirectional interaction. The spatial markers and bridging markers after the bidirectional interaction are sequentially fused and pooled to obtain an image-level representation; The image-level representation is classified, and the image recognition result is output.

2. The image recognition method based on semantic and fingerprint bridging according to claim 1, characterized in that, In the local structure branch, residual construction is performed on the image under test to obtain local structure residuals. These residuals are then subjected to hierarchical convolutional encoding via a spatial feature coding network to obtain a spatially labeled sequence that preserves spatial relationships, including: The image to be tested is subjected to nearest neighbor downsampling and nearest neighbor upsampling in sequence to obtain the restored image; Based on the difference between the image to be tested and the restored image, the local structural residual is obtained; The local structural residuals are input into a spatial feature encoding network. In each stage, the local structural residuals are sequentially subjected to depthwise convolution, layer normalization, pointwise convolution, GELU activation, and global response normalization to obtain a spatial feature map. The spatial feature map is then flattened into a spatial label sequence that preserves spatial positional relationships. The feature channels in each stage are sequentially expanded.

3. The image recognition method based on semantic and fingerprint bridging according to claim 2, characterized in that, The local structural residual is expressed as: ； in, Indicates local structural residuals; Indicates the residual scaling factor; Represents the image to be tested; Indicates nearest neighbor upsampling; This indicates nearest neighbor downsampling; The spatial label sequence that preserves spatial location relationships is represented as follows: ； in, This represents a sequence of spatial markers that preserves spatial relationships. Represents a spatial feature encoding network; The learnable parameters of a spatial feature coding network; Indicates the number of spatial locations; The feature dimension representing the spatial label; It represents the set of real numbers.

4. The image recognition method based on semantic and fingerprint bridging according to claim 1, characterized in that, The preprocessed image that meets the input specification is represented as: ； in, This represents a preprocessed image that meets the input specifications. Represents the image to be tested; These represent the channel mean and channel standard deviation used by CLIP, respectively. This means adjusting the input image to 224×224; The global semantic vector is represented as follows: ； in, Represents the global semantic vector; Represents a semantic image coding network; These represent the parameters of a semantic image coding network. Represents the set of real numbers; Represents the loss function Parameters relative to semantic image coding networks The gradient.

5. The image recognition method based on semantic and fingerprint bridging according to claim 1, characterized in that, The high-frequency residual map is represented as follows: ； in, Indicates the first High-frequency residual plot extracted by a high-pass filter kernel; Indicates the first One high-pass filter core; Represents the image to be tested; Represents two-dimensional convolution; The high-frequency fingerprint is represented as follows: ； in, Indicates high-frequency fingerprints; Represents a high-frequency fingerprint coding network; These represent the high-frequency residual images extracted by the 1st, 2nd, and 3rd high-pass filter kernels, respectively. It represents the set of real numbers.

6. The image recognition method based on semantic and fingerprint bridging according to claim 1, characterized in that, The mapped spatial label sequence is represented as follows: ； in, This represents the mapped spatial label sequence; This represents a sequence of spatial markers that preserves spatial relationships. The learnable projection matrix representing the local structural branches; Representation layer normalization; The mapped global semantic tag is represented as follows: ； in, This represents the mapped global semantic tagging; Represents the global semantic vector; The learnable projection matrix representing the global semantic branch; The mapped high-frequency fingerprint is represented as follows: ； in, This represents the mapped high-frequency fingerprint; Indicates high-frequency fingerprints; The learnable projection matrix representing the high-frequency fingerprint branch; The semantic and fingerprint bridging tag set is represented as follows: ； in, This represents a set of semantic and fingerprint bridging tags; This indicates splicing along the direction of the marker sequence.

7. The image recognition method based on semantic and fingerprint bridging according to claim 1, characterized in that, The spatial markers after the bidirectional interaction are represented as follows: ； in, Indicates spatial markers after two-way interaction; This represents the mapped spatial label sequence; This represents a set of semantic and fingerprint bridging tags; Representation layer normalization; This indicates cross attention; This represents the projection matrix used to generate the spatial query; This represents the projection matrix used to generate the bridging key; The projection matrix that generates the bridging values; The bridging marker after the bidirectional interaction is represented as: ； in, This indicates a bridging marker after two-way interaction; This represents the projection matrix used to generate the bridging query; This represents the projection matrix for generating the updated space keys; This represents the projection matrix used to generate the updated spatial values.

8. The image recognition method based on semantic and fingerprint bridging according to claim 1, characterized in that, The image-level representation is expressed as follows: ； in, Represents an image-level representation; This indicates cross attention; This indicates a learnable pooled query; This indicates a bridging marker after two-way interaction; Indicates spatial markers after two-way interaction; This indicates splicing along the direction of the marker sequence.

9. The image recognition method based on semantic and fingerprint bridging according to claim 1, characterized in that, The image recognition result is obtained through the image generation probability, which is obtained by classifying the image-level representation using a preset classification function. The image generation probability is expressed as: ； In the formula, ; in, Indicates the probability of image generation; Represents the sigmoid function; This indicates the unnormalized classification score; These represent the weights and biases of the first linear layer, respectively. Represents an image-level representation; Representation layer normalization; These represent the weights and biases of the second linear layer, respectively. Indicates nonlinear activation; This indicates random deactivation during the training phase.

10. An image recognition device based on semantic and fingerprint bridging of cross-attention, characterized in that, include: The image acquisition module is used to: acquire the image to be tested; The image recognition module is used to: input the image to be tested into the image recognition model and output the image recognition result; The model processing module is used for: the processing steps of the image recognition model include: In the local structure branch, the image to be tested is subjected to residual construction to obtain local structure residuals. The local structure residuals are then subjected to hierarchical convolutional encoding through a spatial feature coding network to obtain a spatial label sequence that preserves spatial positional relationships. In the global semantic branch, the image to be tested is preprocessed to obtain a preprocessed image that meets the input specifications. The semantic image coding network is used to extract semantics from the preprocessed image to obtain a global semantic vector. In the high-frequency fingerprint branch, the high-frequency residual map of the image under test is extracted by at least one high-pass filter kernel, and the high-frequency residual map is encoded by a high-frequency fingerprint encoding network to obtain a high-frequency fingerprint; The spatial label sequence, global semantic vector, and high-frequency fingerprint are mapped to a unified feature space, and the mapped global semantic label and high-frequency fingerprint are concatenated into a semantic and fingerprint bridging label set. Based on cross-attention, the forward interaction uses the mapped spatial label sequence as the query and the semantic and fingerprint bridging label set as the key and value to update the spatial label sequence. The reverse interaction uses the semantic and fingerprint bridging label set as the query and the updated spatial label as the key and value to update the semantic and fingerprint bridging label set, thus obtaining the spatial label and bridging label after bidirectional interaction. The spatial markers and bridging markers after the bidirectional interaction are sequentially fused and pooled to obtain an image-level representation; The image-level representation is classified, and the image recognition result is output.

Images