Image authentic identification method based on joint model, electronic equipment and storage medium

Abstract

The invention provides an image authentic identification method based on a joint model, electronic equipment and a storage medium, and belongs to the technical field of image processing. In the method, a frequency analysis module generates feature maps corresponding to different attack types according to information of different frequency bands of an image to be detected and the attack types; the knowledge fine adjustment module performs fine adjustment on the distribution rate between the central difference convolution and the original convolution in each adapter sub-module according to the feature maps corresponding to different attack types, and the knowledge recombination module can determine the output weight corresponding to each adapter sub-module according to the feature maps corresponding to different attack types. Therefore, the encoder module can perform feature extraction on the to-be-detected image based on the adapter sub-modules after fine tuning to obtain the image feature corresponding to each adapter sub-module, and performs weighted fusion on each image feature according to the determined output weight corresponding to each adapter sub-module to obtain the to-be-detected image. And accurate and effective authentic identification detection can be carried out on a plurality of counterfeiting methods.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image authentication method, electronic device and storage medium based on a joint model. Background Art

[0002] Image recognition systems are widely used in security, surveillance, identity verification, finance and other fields. Therefore, ensuring the security of image recognition systems is crucial.

[0003] Existing joint authentication methods simply train samples of different attacks together, which causes the model to lose detailed forgery clues of specific attack types and has poor detection performance when facing realistic attacks. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose an image authentication method, electronic device and storage medium based on a joint model, aiming to accurately and effectively detect a variety of counterfeiting methods and improve adaptability and robustness to unknown attack types.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes an image authentication method based on a joint model, wherein the joint model includes a frequency analysis module, a knowledge fine-tuning module, a knowledge reorganization module, an encoder module and a classifier module, wherein each layer of the encoder module is deployed with an adapter submodule, and the method includes:

[0006] Acquire the image to be detected;

[0007] The frequency analysis module generates feature graphs corresponding to different attack types according to the information of different frequency bands of the image to be detected and the attack type;

[0008] The knowledge fine-tuning module fine-tunes the distribution ratio between the central difference convolution and the original convolution in each of the adapter submodules according to the feature graphs corresponding to the different attack types;

[0009] The knowledge reorganization module determines the output weights corresponding to each of the adapter submodules according to the feature graphs corresponding to the different attack types;

[0010] The encoder module extracts features of the image to be detected based on the fine-tuned adapter submodule to obtain image features corresponding to each adapter submodule, and performs weighted fusion on each image feature according to the determined output weights corresponding to each adapter submodule to obtain a fused feature image;

[0011] The classifier module obtains the fused feature image and predicts the true or false classification of the image to be detected.

[0012] In one embodiment of the present application, the frequency analysis module generates feature graphs corresponding to different attack types according to the information of different frequency bands of the image to be detected and the attack type, including:

[0013] The frequency analysis module converts the image to be detected from the spatial domain to the frequency domain through Fourier transform to obtain a spectrum diagram;

[0014] According to the distribution of different frequency bands on the spectrum graph, inverse Fourier transform is used to convert the information of different frequency bands back to the spatial domain to generate feature graphs of different frequency bands;

[0015] According to the attack type, different weights are assigned to the feature maps of each frequency band, and the feature maps of each frequency band are weighted fused according to the assigned weights to obtain the feature maps corresponding to different attack types.

[0016] In one embodiment of the present application, the knowledge fine-tuning module fine-tunes the distribution ratio between the central difference convolution and the original convolution in each of the adapter submodules according to the feature graphs corresponding to the different attack types, including:

[0017] The knowledge fine-tuning module fine-tunes the distribution ratio between the central differential convolution and the original convolution in each of the adapter sub-modules according to the importance of different attack types in different frequency bands, so as to adjust the weights corresponding to the feature maps corresponding to each different attack type.

[0018] In one embodiment of the present application, the knowledge reorganization module determines the output weights corresponding to each of the adapter submodules according to the feature graphs corresponding to the different attack types, including:

[0019] The knowledge reorganization module performs weighted averaging on the feature graphs corresponding to the different attack types to obtain an average feature graph;

[0020] According to the average feature map, the output weights corresponding to each of the adapter sub-modules are predicted.

[0021] In one embodiment of the present application, the method also includes adopting a multi-stage training strategy and an adaptive weight adjustment mechanism to train the joint model, the multi-stage training strategy includes a pre-training stage, a fine-tuning stage and an inference stage, and the adaptive weight adjustment mechanism is to automatically adjust the output weights of each adapter sub-module in the knowledge reorganization module according to different attack types.

[0022] In one embodiment of the present application, the pre-training stage is used to train each adapter sub-module in the encoder module, and the pre-training stage includes:

[0023] Constructing a first training sample, wherein the first training sample includes a real image, forged images of different attack types, and an attack type label;

[0024] The first training sample is input into the encoder module, and the corresponding adapter submodule is trained according to the attack type label.

[0025] In one embodiment of the present application, the fine-tuning stage is a stage of continuing training after the pre-training stage is completed. The fine-tuning stage is used to train the knowledge fine-tuning module to adjust the distribution ratio between the central difference convolution and the original convolution of each of the adapter sub-modules. The fine-tuning stage includes:

[0026] Constructing a second training sample, wherein the second training sample includes a real image and forged images of different attack types;

[0027] Inputting the second training sample into the frequency analysis module, so that the frequency analysis module generates feature maps corresponding to different attack types according to the information of different frequency segments of the image to be detected and the attack type;

[0028] Inputting the feature graphs corresponding to the different attack types into the knowledge fine-tuning module, so that the knowledge fine-tuning module fine-tunes the distribution ratio between the central difference convolution and the original convolution corresponding to the trained adapter submodule according to the feature graphs corresponding to the different attack types;

[0029] Inputting the characteristic graphs corresponding to the different attack types into the knowledge reorganization module, so that the knowledge reorganization module determines the output weights corresponding to each of the adapter submodules according to the characteristic graphs corresponding to the different attack types;

[0030] The encoder module extracts features of the image to be detected based on the fine-tuned adapter submodule to obtain image features corresponding to each adapter submodule, and performs weighted fusion on each image feature according to the determined output weights corresponding to each adapter submodule to obtain a fused feature image;

[0031] The fused feature image is input into the classifier module so that the classifier module predicts the true or false classification of the image to be detected.

[0032] In one embodiment of the present application, the inference stage is a testing stage after the fine-tuning stage is completed. The process of the inference stage is the same as the process of the fine-tuning stage. The inference stage is used to evaluate the performance of the model after the fine-tuning stage is completed.

[0033] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method described in the first aspect of the embodiment of the present application when executing the computer program.

[0034] To achieve the above-mentioned purpose, the third aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method described in the first aspect of the embodiments of the present application is implemented.

[0035] In the technical solution provided in the embodiment of the present application, the joint model includes a frequency analysis module, a knowledge fine-tuning module, a knowledge reorganization module, an encoder module and a classifier module. Among them, each layer of the encoder module is deployed with an adapter submodule. In this method, the frequency analysis module generates feature maps corresponding to different attack types according to the information of different frequency bands of the image to be detected and the attack type, and can extract various subtle and precise counterfeit identification knowledge for different attack types to ensure efficient identification of specific attack types. Then the feature maps corresponding to different attack types are transmitted to the knowledge fine-tuning module, so that the knowledge fine-tuning module can fine-tune the distribution rate between the central difference convolution and the original convolution in each adapter submodule according to the feature maps corresponding to different attack types, so as to weight each feature map according to the importance of different attack types in different frequency bands, so that different input samples can highlight their own characteristics, so as to make more effective use of existing knowledge. Then the feature maps corresponding to different attack types are transmitted to the knowledge reorganization module, so that the knowledge reorganization module can determine the output weights corresponding to each adapter submodule according to the feature maps corresponding to different attack types. Thus, the encoder module can extract features of the image to be detected based on the fine-tuned adapter submodule, obtain the image features corresponding to each adapter submodule, and perform weighted fusion on each image feature according to the determined output weights corresponding to each adapter submodule to obtain a fused feature image. Accurate and effective discriminative features can be obtained. The fused feature image is then input into the classifier module, which predicts the authenticity classification of the image to be detected. In this way, accurate and effective counterfeiting detection can be performed on a variety of counterfeiting methods, improving the adaptability and robustness to unknown attack types. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a schematic diagram of the architecture of the joint model provided in one embodiment of the present application.

[0037] Figure 2 It is a flowchart of the steps of the pre-training stage provided in one embodiment of the present application.

[0038] Figure 3This is the step flow of the fine-tuning stage provided in an embodiment of the present application.

[0039] Figure 4 This is a flowchart of an image authentication method based on a joint model provided in one embodiment of the present application.

[0040] Figure 5 It is a flowchart of the steps of generating feature maps corresponding to different attack types according to the information of different frequency bands of the image to be detected and the attack type provided by the frequency analysis module of one embodiment of the present application.

[0041] Figure 6 It is a flowchart of the steps of determining the output weights corresponding to each adapter sub-module according to the feature graphs corresponding to different attack types provided by the knowledge reorganization module in one embodiment of the present application.

[0042] Figure 7 It is a schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0046] Image recognition systems are widely used in security, surveillance, identity verification, and finance. Therefore, it is crucial to ensure the security of image recognition systems. However, they are vulnerable to counterfeit attacks, which include physical attacks (PAs) and digital attacks (DAs). Among them, physical attacks (PAs) are used to forge images through physical displays that are captured by imaging devices and presented in the form of digital photos, such as printed paper, video playback, and 3D prints. Digital attacks (DAs) aim to edit images through digital forgery techniques and then pass them to the system for verification. Specific means include editing the attributes of the image or adding adversarial noise.

[0047] Since the forgery clues of these two attacks are distributed in different areas, most image forgery detection methods aim to detect one of the two categories, physical attacks (PAs) or digital attacks (DAs). For physical attacks (PAs), existing methods capture physical forgery traces in images by extracting various LBP (Local Binary Patterns) descriptors in different color spaces (such as grayscale, RGB, HSV, and YCbCr) to distinguish between real and fake images. Image distortion analysis focuses on detecting surface distortions of facial skin caused by low image quality compared to reality. Other studies have introduced additional supervisory signals such as depth maps, reflectance maps, and r-PPG signals to improve features by focusing on disentanglement, thereby achieving better regularization. For digital attacks (DAs), existing methods extract imperceptible anomalies caused by digital techniques in images by analyzing frequency domain features, color channel inconsistencies, deep learning feature anomalies, local texture inconsistencies, and noise pattern analysis.

[0048] Existing joint methods mix various samples together to solve the joint image authentication problem through a unified classification or multi-task learning framework. Although these methods have made some progress, their generalization ability is still limited. Coarsely mixing the data of the two attacks during training may mislead the model to only focus on the general attack knowledge and ignore the learning of fine-grained knowledge of each attack type, resulting in limited generalization ability. In addition, when encountering unknown attacks that are difficult to identify through coarse general knowledge, the performance of the model may deteriorate.

[0049] It can be seen that the existing joint authentication method simply trains samples of different attacks together, which will cause the model to lose the refined forgery clues of specific attack types and have poor detection performance when facing realistic attacks.

[0050] Based on this, the embodiments of the present application propose an image authentication method, electronic device and storage medium based on a joint model, aiming to accurately and effectively detect a variety of counterfeiting methods and improve adaptability and robustness to unknown attack types.

[0051] Reference Figure 1 , Figure 1 FIG. 1 is a schematic diagram of the architecture of a joint model provided by an embodiment of the present application. Figure 1 As shown, the joint model includes a frequency analysis module 11, a knowledge fine-tuning module 12, a knowledge reorganization module 13, an encoder module 14 and a classifier module 15. Among them, each layer of the encoder module 14 is deployed with an adapter submodule. Among them, the real image and the images of various attack types correspond to an adapter submodule respectively, which aims to use the central differential convolution of a specific task to mine small and detailed forgery clues specific to a certain attack type, so that the model is more robust to realistic attacks, and at the same time, the pre-trained large model parameters can be better adapted to downstream tasks and the common features between various attacks can be mined.

[0052] Among them, the frequency analysis module 11 is used to generate feature maps corresponding to different attack types according to the information of different frequency bands of the image to be detected and the attack type. In the field of image processing and computer vision, different frequency bands of the image contain different information. By analyzing this information, the feature map of the image can be extracted. The feature map can help us better understand and utilize the information in the image, especially in tasks such as target detection, image segmentation and image recognition. In image processing, frequency usually refers to the speed of grayscale value change in the image. The high-frequency part usually represents the edge and details of the image, while the low-frequency part represents the general shape and color of the image. Through Fourier transform, the image can be converted from the spatial domain to the frequency domain, thereby separating different frequency components. The image can be converted from the spatial domain to the frequency domain through Fourier transform to obtain a spectrum diagram. The spectrum diagram describes the frequency structure of the image and the relationship between the frequency and the amplitude of the frequency signal. The horizontal axis is the frequency and the vertical axis is the amplitude of the frequency signal. On the spectrum diagram, different frequency bands (low frequency, medium frequency, high frequency) have different distribution characteristics. The frequency analysis module 11 can convert the information of the selected frequency band back to the spatial domain by inverse Fourier transform according to the distribution of different frequency bands on the spectrum, thereby generating a feature map of the corresponding frequency band. For example, for a low-frequency band, the coefficients corresponding to the low-frequency part in the center of the spectrum can be selected, and the low-frequency feature map can be obtained by inverse Fourier transform; for a high-frequency band, the coefficients corresponding to the high-frequency part around the spectrum can be selected and inverse Fourier transform can be performed to obtain a high-frequency feature map.

[0053] In some embodiments, different filters can be used to extract specific frequency bands of an image. For example, a high-pass filter can enhance the edges and details of an image, while a low-pass filter can highlight the overall shape and color of the image. By designing different types of filters, feature maps of different frequency bands can be generated. In addition, wavelet transform (an effective multi-resolution analysis tool) can be used to decompose the image into frequency components at different scales. Through wavelet transform, a series of wavelet coefficients can be obtained, which reflect the characteristics of the image in different frequency bands. These wavelet coefficients can be regarded as feature maps for subsequent image analysis tasks.

[0054] The frequency analysis module 11 can generate feature maps corresponding to different attack types based on the information of different frequency bands of the image. Specifically, it can separate the low-frequency, medium-frequency and high-frequency information of the image, thereby obtaining a low-frequency feature map reflecting the general appearance and outline of the image, and a high-frequency feature map reflecting the edge, noise and details of the image.

[0055] In the process of generating feature maps, the attack type is also an important consideration. Different attack types may have different effects on certain frequency bands of the image. For example, some attack types may increase the high-frequency components of the image, resulting in more noise and details in the image; while some attack types may affect the low-frequency components of the image, changing the overall structure and background of the image. The frequency analysis module 11 can assign different weights to the feature maps of each frequency band according to the importance of the characteristics of different frequency bands in each attack type. Thereby, the feature maps of each frequency band can be further weighted and fused according to the assigned weights to obtain feature maps corresponding to different attack types.

[0056] Exemplarily, if there are three types of attacks, for the first type of attack, different weights can be assigned to the feature graphs corresponding to different attack types based on the different importance of the attack type in different frequency bands, and then weighted fusion can be performed to obtain the feature graph corresponding to the first type of attack. For the second type of attack, different weights can be assigned to the feature graphs corresponding to different attack types based on the different importance of the attack type in different frequency bands, and then weighted fusion can be performed to obtain the feature graph corresponding to the second type of attack. For the third type of attack, different weights can be assigned to the feature graphs corresponding to different attack types based on the different importance of the attack type in different frequency bands, and then weighted fusion can be performed to obtain the feature graph corresponding to the third type of attack. The weighted fused feature graphs corresponding to the three attack types can be obtained.

[0057] It should be noted that the frequency analysis module may include multiple frequency analysis submodules, each of which processes a sample of an attack type to generate a corresponding feature map. For example, if there are 5 attack types, there are 5 frequency analysis submodules to generate 5 feature maps.

[0058] The frequency analysis module 11 generates feature maps corresponding to different attack types according to the information of different frequency bands of the image to be detected and the attack type, and can extract various subtle and precise authentication knowledge for different attack types to ensure efficient identification of specific attack types.

[0059] The knowledge fine-tuning module 12 is used to fine-tune the distribution rate (distribution ratio) between the center difference convolution and the original convolution in each adapter submodule according to the feature graphs corresponding to different attack types. Among them, the original convolution is a basic operation in image processing. It multiplies the convolution kernel with the corresponding point of the image and then adds them, and replaces the value of the template center point with the final result. This process not only involves the local receptive field of the image, but also involves the global information integration of the image. The center difference convolution introduces the concept of center difference on the basis of the original convolution to improve its representation and generalization capabilities. Specifically, the center difference convolution includes two steps: sampling and aggregation. The sampling step is similar to the original convolution, but the aggregation step tends to aggregate the central gradient of the sampling value. The original convolution is suitable for application scenarios that need to retain the basic structure and contour information of the image, while the center difference convolution is suitable for application scenarios that need to capture high-frequency details of the image. Therefore, the knowledge fine-tuning module 12 fine-tunes the corresponding adapter submodule according to different attack types and weighted feature graphs corresponding to different attack types, which can more effectively utilize existing knowledge to ensure that the model can be flexibly adjusted to a wider range of attack types and scenarios. According to the feature maps of each frequency band after weighted balance, the differential degree of the adapter submodule (i.e., the allocation ratio between the central differential convolution and the original convolution) is adjusted, and the existing attack-specific refined knowledge is fine-tuned to make it more in line with the needs of the joint authentication task.

[0060] It should be noted that the knowledge fine-tuning module 12 may include multiple knowledge fine-tuning submodules, each of which is used to adjust the differential degree of the adapter submodule. The number of knowledge fine-tuning submodules is the same as the number of adapter submodules, and the differential degree of each adapter submodule is fine-tuned by the corresponding knowledge fine-tuning submodule.

[0061] The knowledge reorganization module 13 is used to determine the output weights corresponding to each adapter submodule according to the feature graphs corresponding to different attack types. Among them, the role of the adapter submodule in image processing is to match different image processing tasks with corresponding processing methods. Specifically, the knowledge reorganization module 13 will predict the selection probability of different attack-specific knowledge at each layer in the encoder module through a linear function, and then reorganize them together to output more essential and refined features. The knowledge reorganization module 13 can distribute and reorganize existing knowledge according to the average image (obtained by weighted averaging the feature graphs corresponding to each different attack type), and explore the complementary relationship between different attack types, so that the existing knowledge can be adaptively selected according to the sample, making it more robust to unseen attack types and scenarios.

[0062] It should be noted that the knowledge reorganization module 13 may include multiple knowledge selection submodules (which may be constructed using linear layers), each of which corresponds to the output weight of the corresponding adapter submodule in the encoder module. The number of knowledge selection submodules is the same as the number of adapter submodules, and the output weight of each adapter submodule is predicted by the corresponding knowledge selection submodule.

[0063] The encoder module 14 is used to extract features of the image to be detected based on the fine-tuned adapter submodules, obtain the image features corresponding to each adapter submodule, and perform weighted fusion of each image feature according to the output weights corresponding to each adapter submodule, to obtain a fused feature image. That is, in the embodiment of the present application, after the distribution rate (distribution ratio) between the central difference convolution and the original convolution in each adapter submodule is fine-tuned by the knowledge fine-tuning module 12, and the output weights corresponding to each adapter submodule are determined by the knowledge reorganization module 13, the encoder module 14 can extract features of the image to be detected based on the adapter submodules after the distribution rate is fine-tuned, obtain the image features corresponding to each adapter submodule, and perform weighted fusion of each image feature according to the output weights corresponding to each adapter submodule, to obtain a fused feature image. In this way, a more robust feature representation can be extracted.

[0064] The classifier module 15 is used to obtain the fused feature image and predict the true and false classification of the image to be detected. That is, the classifier module 15 can analyze the specific features in the fused feature image to classify the true and false images. Among them, the classifier module may include k-nearest neighbor (kNN), support vector machine (SVM), naive Bayes, linear regression, logistic regression, random forest, etc.

[0065] In the embodiment of the present application, in order to use the above-mentioned joint model for image authentication, the joint model needs to be further trained, so that the trained joint model can be used for image authentication. Specifically, the embodiment of the present application adopts a multi-stage training strategy and an adaptive weight adjustment mechanism to train the joint model. Among them, the multi-stage training strategy includes a pre-training stage, a fine-tuning stage and an inference stage, and the adaptive weight adjustment mechanism is to automatically adjust the output weights of each adapter sub-module in the knowledge reorganization module according to the different attack types. Among them, the pre-training stage is used to train each adapter sub-module in the encoder module 14, the fine-tuning stage is a stage for continuing training after the pre-training stage is completed, and the fine-tuning stage is used to train the knowledge fine-tuning module 12 to adjust the distribution rate between the central differential convolution and the original convolution of each adapter sub-module. The inference stage is a testing stage after the fine-tuning stage is completed. The process of the inference stage is the same as the process of the pre-fine-tuning stage. The inference stage is used to evaluate the performance of the model after the fine-tuning stage is completed.

[0066] Specifically, the embodiment of the present application introduces a multi-stage training strategy and an adaptive weight adjustment mechanism to cope with the unified physical and digital image authentication challenge. Specifically, the multi-stage training strategy includes three stages: pre-training stage, fine-tuning stage, and recommendation stage. Each stage is optimized for different attack types and scenarios to improve the generalization ability of the model and its adaptability to unknown attack types. The adaptive weight adjustment mechanism is to introduce an adaptive weight adjustment mechanism in the knowledge reorganization module 13, which automatically adjusts the output weights of each adapter submodule according to the different attack types to extract more robust feature representations.

[0067] Among them, refer to Figure 2 , Figure 2 It is a flowchart of the steps of the pre-training stage provided in an embodiment of the present application, including but not limited to steps S210 to S220.

[0068] Step S210, constructing a first training sample, where the first training sample includes a real image, forged images of different attack types, and an attack type label;

[0069] Step S220: input the first training sample into the encoder module, and train the corresponding adapter submodule according to the attack type label.

[0070] In the embodiment of the present application, a first training sample is first constructed, wherein the first training sample includes a real image, a forged image of different attack types, and an attack type label. The first training sample is then input into the encoder module 14, and the corresponding adapter submodule is trained according to the attack type label. Specifically, the process of the pre-training stage is as follows:

[0071] S1: Construct a first training sample, where the first training sample includes an image to be detected and its corresponding attack type label, wherein the image to be detected may include a real image and forged images of different attack types.

[0072] S2: Input the image to be detected and its corresponding attack label into the encoder module 14, wherein the encoder module 14 has corresponding adapter submodules deployed in each layer, such as adapter submodules corresponding to real images, adapter submodules corresponding to physical attack type images, and adapter submodules corresponding to digital attack type images deployed in the encoder module 14. The encoder module 14 may use a pre-trained model Vi T, etc.

[0073] S3: According to the attack type label corresponding to the image to be detected, the corresponding adapter submodule is selected for training. The adapter submodule will extract the corresponding image features for subsequent counterfeit detection tasks. If the attack type label corresponding to the image to be detected is physical attack, the adapter submodule corresponding to the physical attack type image is selected for training, and if the attack type label corresponding to the image to be detected is digital attack, the adapter submodule corresponding to the digital attack type image is selected for training.

[0074] S4: The image features output from step S3 are transmitted to the classifier module 15, and the classifier module 15 is used to identify the authenticity of the image

[0075] S5: Store the model trained in the pre-training phase to facilitate subsequent retrieval for training.

[0076] In the embodiment of the present application, the pre-training stage is used to train each adapter submodule in the encoder module 14. By training each adapter submodule in the encoder module 14, the ability to extract the forgery detection features of the image can be effectively improved. That is, the trained adapter submodule has a strong ability to extract the forgery detection features of a specific attack sample. By training the corresponding adapter submodule separately with different attack samples, it is intended to mine the attack type-specific, fine and detailed discriminative features that can serve as forgery detection features of realistic attack samples, and prevent the model from overfitting to broad common knowledge.

[0077] Among them, refer to Figure 3 , Figure 3 It is a flowchart of the steps of the fine-tuning stage provided in an embodiment of the present application, including but not limited to steps S310 to S360.

[0078] Step S310, constructing a second training sample, where the second training sample includes real images and forged images of different attack types;

[0079] Step S320, inputting the second training sample into the frequency analysis module, so that the frequency analysis module generates feature maps corresponding to different attack types according to the information of different frequency segments of the image to be detected and the attack type;

[0080] Step S330, inputting the feature graphs corresponding to different attack types into the knowledge fine-tuning module, so that the knowledge fine-tuning module fine-tunes the distribution ratio between the central difference convolution and the original convolution corresponding to the trained adapter submodule according to the feature graphs corresponding to different attack types;

[0081] Step S340, inputting the characteristic graphs corresponding to different attack types into the knowledge reorganization module, so that the knowledge reorganization module determines the output weights corresponding to each adapter submodule according to the characteristic graphs corresponding to different attack types;

[0082] Step S350: the encoder module extracts features of the image to be detected based on the fine-tuned adapter submodules to obtain image features corresponding to each adapter submodule, and performs weighted fusion on each image feature according to the determined output weights corresponding to each adapter submodule to obtain a fused feature image.

[0083] Step S360: input the fused feature image into the classifier module so that the classifier module predicts the authenticity classification of the image to be detected.

[0084] In the embodiment of the present application, unlike the first training sample in the pre-training stage, the second training sample in the fine-tuning stage includes real images and forged images of different attack types, but does not include attack type labels. It should be noted that the second training sample can be obtained by deleting the attack type label on the basis of the original first training sample. The fine-tuning stage is a stage for continuing training after the pre-training stage is completed, that is, after the pre-training stage is completed, the pre-trained model can be loaded and continued to train on this basis. Specifically, the second training sample is first constructed, and then the second training sample is input into the frequency analysis module 11, so that the frequency analysis module 11 can generate feature maps corresponding to different attack types according to the information of different frequency segments of the image to be detected and the attack type. Then the feature maps corresponding to different attack types are output to the knowledge fine-tuning module 12, so that the knowledge fine-tuning module 13 can fine-tune the distribution rate between the central difference convolution and the original convolution corresponding to the trained adapter submodule according to the feature maps corresponding to different attack types. At the same time, the feature maps corresponding to different attack types are input into the knowledge reorganization module 13, so that the knowledge reorganization module 13 can determine the output weights corresponding to each adapter submodule according to the feature maps corresponding to different attack types. The feature maps corresponding to different attack types can be weighted averaged to obtain an average feature map, so that the output weights corresponding to each adapter submodule in the encoder module 14 can be predicted based on the average feature map. As a result, the encoder module 15 can extract features of the image to be detected based on the fine-tuned adapter submodules, obtain the image features corresponding to each adapter submodule, and perform weighted fusion of each image feature according to the determined output weights corresponding to each adapter submodule to obtain a fused feature image. Accurate and effective discriminative features can be obtained. The fused feature image is then input into the classifier module, and the classifier module predicts the authenticity classification of the image to be detected.

[0085] In an embodiment of the present application, the fine-tuning stage can be used to train the model on how to adjust the allocation ratio between the center difference convolution and the original convolution in each adapter sub-module, so that the model after the fine-tuning stage can adjust the allocation ratio between the center difference convolution and the original convolution in each adapter sub-module based on different input images (including different attack types), without over-reliance on existing knowledge, thereby increasing the robustness of the model.

[0086] After the fine-tuning phase is completed, the inference phase is entered, and the model after the fine-tuning phase can be loaded, and the model test is performed on this basis to evaluate the performance of the model. Since the process of the inference phase is the same as that of the fine-tuning phase, it will not be repeated here. It should be noted that the test sample in the inference phase is a test image different from the second training sample.

[0087] Reference Figure 4 , Figure 4It is a flowchart of an image authentication method based on a joint model provided in an embodiment of the present application, which is executed by the above-mentioned trained joint model, including but not limited to steps S410 to S460.

[0088] Step S410, obtaining an image to be detected.

[0089] In the embodiment of the present application, an image to be detected is obtained, and the image to be detected may be a real image or a forged image.

[0090] Step S420: The frequency analysis module generates feature graphs corresponding to different attack types according to the information of different frequency bands of the image to be detected and the attack type.

[0091] In the embodiment of the present application, the image to be detected is input into the frequency analysis module 11, so that the frequency analysis module can generate feature maps corresponding to different attack types according to the information of different frequency bands of the image to be detected and the attack type.

[0092] Reference Figure 5 , Figure 5 It is a flowchart of the steps of generating feature maps corresponding to different attack types according to the information of different frequency bands of the image to be detected and the attack type provided by the frequency analysis module of an embodiment of the present application, including but not limited to steps S510 to S530.

[0093] Step S510, the frequency analysis module converts the image to be detected from the spatial domain to the frequency domain through Fourier transform to obtain a spectrum diagram;

[0094] Step S520, according to the distribution of different frequency bands on the spectrum graph, using inverse Fourier transform to convert the information of different frequency bands back to the spatial domain to generate feature graphs of different frequency bands;

[0095] Step S530, assigning different weights to the feature graphs of each frequency band according to the attack type, and performing weighted fusion on the feature graphs of each frequency band according to the assigned weights to obtain feature graphs corresponding to different attack types.

[0096] In the embodiment of the present application, the frequency analysis module 11 first converts the image to be detected from the spatial domain to the frequency domain by Fourier transform to obtain a spectrum map. Then, according to the distribution of different frequency bands on the spectrum map, the information of different frequency bands is converted back to the spatial domain by inverse Fourier transform to generate feature maps of different frequency bands. Considering that the frequency bands used to identify different attack types are different, different weights can be assigned to the feature maps of each frequency band according to the attack type, and then the feature maps of each frequency band are weighted and fused according to the assigned weights to obtain feature maps corresponding to different attack types. For example, if the frequency band used to identify the role of attack type A is a high frequency band, the weight corresponding to the feature map of the high frequency band is increased, and the weight corresponding to the feature map of other frequency bands is reduced, and then the weight-adjusted feature maps are weighted and fused to obtain the feature map corresponding to attack type A. If the frequency band used to identify the role of attack type B is a low frequency band, the weight corresponding to the feature map of the low frequency band is increased, and the weight corresponding to the feature map of other frequency bands is reduced, and then the weight-adjusted feature maps are weighted and fused to obtain the feature map corresponding to attack type B. Specifically, for a picture, the frequency analysis module 11 can divide it into N frequency segments, predict the importance of the features of each frequency segment in the whole picture through the convolution layer, and assign weights to the feature maps of the corresponding frequency segments according to the importance, and then perform weighted fusion. In this way, the characteristics of different samples can be highlighted, avoiding semantic loss and overfitting caused by using RGB images.

[0097] Step S430: The knowledge fine-tuning module fine-tunes the distribution ratio between the central difference convolution and the original convolution in each adapter submodule according to the feature graphs corresponding to different attack types.

[0098] In the embodiment of the present application, after the frequency analysis module 11 generates the feature graphs corresponding to different attack types, the feature graphs corresponding to different attack types can be transmitted to the knowledge fine-tuning module 12, so that the knowledge fine-tuning module 12 can fine-tune the distribution ratio between the central differential convolution and the original convolution in each adapter submodule according to the importance of different attack types in different frequency bands, so as to adjust the weights corresponding to the feature graphs corresponding to each different attack type, and make more effective use of existing knowledge. That is, the knowledge fine-tuning module 12 can fine-tune the difference degree (the distribution ratio between the central differential convolution and the original convolution) of each adapter submodule according to the feature graphs corresponding to different attack types, that is, fine-tune the existing attack-specific refined knowledge to make it more in line with the needs of the joint counterfeit detection task, so that the fine-tuned knowledge can better adapt to the scene and counterfeit clues of the current sample.

[0099] Step S440: The knowledge reorganization module determines the output weights corresponding to the respective adapter submodules according to the feature graphs corresponding to the different attack types.

[0100] In an embodiment of the present application, after generating feature graphs corresponding to different attack types, the frequency analysis module 11 can transmit the feature graphs corresponding to different attack types to the knowledge reorganization module 13, so that the knowledge reorganization module 13 can determine the output weights corresponding to each adapter sub-module according to the feature graphs corresponding to different attack types.

[0101] Reference Figure 6 , Figure 6 It is a flowchart of the steps of determining the output weights corresponding to each adapter sub-module according to the feature graphs corresponding to different attack types provided by the knowledge reorganization module of an embodiment of the present application, including but not limited to steps S610 to S620.

[0102] Step S610, performing weighted averaging on the feature graphs corresponding to different attack types to obtain an average feature graph;

[0103] Step S620: predict the output weights corresponding to each adapter submodule according to the average feature map.

[0104] In the embodiment of the present application, after obtaining the feature graphs corresponding to different attack types, the knowledge reorganization module 13 can perform weighted averaging on the feature graphs corresponding to different attack types to obtain an average feature graph, and then predict the output weights corresponding to each adapter submodule based on the average feature graph. Specifically, a linear function is used to predict the selection probability of different attack-specific knowledge at each layer in the encoder module, and then they are reorganized together to output more essential and detailed features. The complementary relationship between different attack types can be mined, so that existing knowledge can be adaptively selected according to samples, making it more robust to unseen attack types and scenarios.

[0105] In step S450, the encoder module extracts features of the image to be detected based on the fine-tuned adapter sub-modules to obtain image features corresponding to each adapter sub-module, and performs weighted fusion on each image feature according to the determined output weights corresponding to each adapter sub-module to obtain a fused feature image.

[0106] In an embodiment of the present application, after the knowledge fine-tuning module 12 fine-tunes the distribution ratio between the central difference convolution and the original convolution in each adapter sub-module, and the knowledge reorganization module 13 determines the output weights corresponding to each adapter sub-module, the encoder module 14 performs feature extraction on the image to be detected based on the fine-tuned adapter sub-module to obtain the image features corresponding to each adapter sub-module, and performs weighted fusion on each image feature according to the determined output weights corresponding to each adapter sub-module to obtain a fused feature image, so as to obtain accurate and effective discriminative features.

[0107] Step S460: the classifier module obtains the fused feature image and predicts the authenticity classification of the image to be detected.

[0108] In the embodiment of the present application, after the encoder module 14 performs feature extraction to obtain a fused feature image, the fused feature image can be input into the classifier module 15, so that the classifier module 15 can predict the authenticity classification of the image to be detected.

[0109] The embodiment of the present application learns the small and detailed forgery clues of different attacks through the adapter submodule, which can reduce the loss of semantic information specific to a certain attack. In addition, the embodiment of the present application introduces a multi-stage training strategy and an adaptive weight adjustment mechanism, so that the model can be flexibly adjusted to a wider range of attack types and scenarios, and improves the adaptability and robustness to unknown attack types. At the same time, it can also provide a solid foundation for the subsequent mining of the complementary relationship between different attack types, reduce the vulnerability of the original single detection method to joint tasks, and avoid the decline in the detection performance of the existing joint detection method for realistic attack samples.

[0110] In addition, the embodiments of the present application can explore the complementary relationship between different attack types by adaptively reorganizing existing refined knowledge, focus on fundamental authentication features, and solve the problem of performance degradation of existing joint detection methods for unseen attack types and scenarios.

[0111] See also Figure 7 , Figure 7 : is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application, the electronic device includes:

[0112] The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0113] The memory 702 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 702 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 702, and the processor 701 calls and executes the image authentication method based on the joint model in the embodiment of this application;

[0114] Input / output interface 703, used to implement information input and output;

[0115] Communication interface 704, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0116] A bus 705 that transmits information between the various components of the device (e.g., the processor 701, the memory 702, the input / output interface 703, and the communication interface 704);

[0117] The processor 701 , the memory 702 , the input / output interface 703 and the communication interface 704 are connected to each other in communication within the device via a bus 705 .

[0118] An embodiment of the present application further provides a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, the above-mentioned image authentication method based on the joint model is implemented.

[0119] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0120] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0121] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0122] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0123] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0124] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0125] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0126] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0127] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0128] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0129] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0130] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and substance of the present invention should be within the scope of the rights of the present invention.

Claims

1. A method for detecting counterfeit images based on a joint model, characterized in that: The joint model includes a frequency analysis module, a knowledge fine-tuning module, a knowledge reorganization module, an encoder module and a classifier module, wherein each layer of the encoder module is deployed with an adapter submodule, and the method includes: Acquire the image to be detected; The frequency analysis module generates feature graphs corresponding to different attack types according to the information of different frequency bands of the image to be detected and the attack type; The knowledge fine-tuning module fine-tunes the distribution ratio between the central difference convolution and the original convolution in each of the adapter submodules according to the feature graphs corresponding to the different attack types; The knowledge reorganization module determines the output weights corresponding to each of the adapter submodules according to the feature graphs corresponding to the different attack types; The encoder module extracts features of the image to be detected based on the fine-tuned adapter submodule to obtain image features corresponding to each adapter submodule, and performs weighted fusion on each image feature according to the determined output weights corresponding to each adapter submodule to obtain a fused feature image; The classifier module obtains the fused feature image and predicts the true or false classification of the image to be detected.

2. The method according to claim 1, characterized in that The frequency analysis module generates feature graphs corresponding to different attack types according to the information of different frequency bands of the image to be detected and the attack type, including: The frequency analysis module converts the image to be detected from the spatial domain to the frequency domain through Fourier transform to obtain a spectrum diagram; According to the distribution of different frequency bands on the spectrum graph, inverse Fourier transform is used to convert the information of different frequency bands back to the spatial domain to generate feature graphs of different frequency bands; According to the attack type, different weights are assigned to the feature maps of each frequency band, and the feature maps of each frequency band are weighted fused according to the assigned weights to obtain the feature maps corresponding to different attack types.

3. The method according to claim 1, characterized in that The knowledge fine-tuning module fine-tunes the distribution ratio between the central difference convolution and the original convolution in each of the adapter submodules according to the feature graphs corresponding to the different attack types, including: The knowledge fine-tuning module fine-tunes the distribution ratio between the central differential convolution and the original convolution in each of the adapter sub-modules according to the importance of different attack types in different frequency bands, so as to adjust the weights corresponding to the feature maps corresponding to each different attack type.

4. The method according to claim 1, characterized in that: The knowledge reorganization module determines the output weights corresponding to the respective adapter submodules according to the feature graphs corresponding to the different attack types, including: The knowledge reorganization module performs weighted averaging on the feature graphs corresponding to the different attack types to obtain an average feature graph; According to the average feature map, the output weights corresponding to each of the adapter sub-modules are predicted.

5. The method according to claim 1, characterized in that The method also includes adopting a multi-stage training strategy and an adaptive weight adjustment mechanism to train the joint model, wherein the multi-stage training strategy includes a pre-training stage, a fine-tuning stage, and an inference stage, and the adaptive weight adjustment mechanism is to automatically adjust the output weights of each adapter sub-module in the knowledge reorganization module according to different attack types.

6. The method according to claim 5, characterized in that The pre-training stage is used to train each adapter sub-module in the encoder module, and the pre-training stage includes: Constructing a first training sample, wherein the first training sample includes a real image, forged images of different attack types, and an attack type label; The first training sample is input into the encoder module, and the corresponding adapter submodule is trained according to the attack type label.

7. The method according to claim 5, characterized in that The fine-tuning stage is a stage for continuing training after the pre-training stage is completed. The fine-tuning stage is used to train the knowledge fine-tuning module to adjust the distribution ratio between the central differential convolution and the original convolution of each adapter submodule. The fine-tuning stage includes: Constructing a second training sample, wherein the second training sample includes a real image and forged images of different attack types; Inputting the second training sample into the frequency analysis module, so that the frequency analysis module generates feature graphs corresponding to different attack types according to the information of different frequency segments of the image to be detected and the attack type; Inputting the feature graphs corresponding to the different attack types into the knowledge fine-tuning module, so that the knowledge fine-tuning module fine-tunes the distribution ratio between the central difference convolution and the original convolution corresponding to the trained adapter submodule according to the feature graphs corresponding to the different attack types; Inputting the characteristic graphs corresponding to the different attack types into the knowledge reorganization module, so that the knowledge reorganization module determines the output weights corresponding to the respective adapter submodules according to the characteristic graphs corresponding to the different attack types; The encoder module extracts features of the image to be detected based on the fine-tuned adapter submodule to obtain image features corresponding to each adapter submodule, and performs weighted fusion on each image feature according to the determined output weights corresponding to each adapter submodule to obtain a fused feature image; The fused feature image is input into the classifier module so that the classifier module predicts the true or false classification of the image to be detected.

8. The method according to claim 5, characterized in that The inference stage is a testing stage after the fine-tuning stage is completed. The process of the inference stage is the same as the process of the fine-tuning stage. The inference stage is used to evaluate the performance of the model after the fine-tuning stage is completed.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images