Deep learning-based image abnormal tampering automatic detection method, storage medium and equipment

Abstract

The application discloses a kind of based on deep learning's image abnormal tampering automatic detection method, storage medium and equipment, including digital watermark embedding, image similarity measurement, establish detection model and four steps of abnormal tampering detection, in digital watermark embedding, on the basis of least significant bit replacement (LSB) fusion semi-fragile watermark characteristics will be embedded in digital watermark key frame, the sensitivity of image to slight change is improved;In image similarity measurement, all-round judgment for abnormal tampering is realized by measuring index screening;In the establishment of detection model, the basis of abnormal tampering automatic detection deep learning model is constructed;Finally in abnormal tampering detection, multiple similarity measurement indexes and convolutional neural network (CNN) are combined for training, to realize the automatic detection of abnormal tampering.The method of the application realizes the fast and accurate abnormal tampering identification, and can be used in various new media video types such as short video, to provide a solution for the authenticity supervision of various video pictures.

Description

Technical Field

[0001] This invention belongs to the technical field of information security and relates to new applications of deep learning and digital watermarking technologies in real-time detection of the authenticity of various new media video images such as camera images and short videos. It mainly involves an automatic detection method, storage medium and device for abnormal image tampering based on deep learning. Background Technology

[0002] Cameras, ubiquitous in daily life, serve as crucial tools for acquiring digital image information, and the captured image data is widely disseminated and applied across numerous industries. Therefore, protecting image data from unauthorized tampering and ensuring its security during acquisition, storage, processing, and transmission is paramount.

[0003] Current research on image protection for various types of video streams and images is abundant, but most focuses on defending against passive attacks, neglecting the defense and alerting against active attacks. Some studies have successfully detected minor changes in text and images using trained deep learning models, but their application in real-life scenarios is limited. Other studies emphasize encryption of camera footage to prevent privacy leaks during transmission, but cannot guarantee the authenticity of the footage received by the user. In fields such as evidence identification, methods for identifying video streams and camera footage focus on post-extraction verification, leaving gaps in detecting whether evidence has been damaged or altered before extraction, and failing to prevent malicious human intervention.

[0004] A secure and efficient camera image transmission mechanism should meet three conditions: low-latency transmission, high-quality video stream, and accurate and rapid anomaly tampering detection. Low-latency transmission means a short transmission time from the camera device to the user terminal, meeting the requirements of the camera product to provide real-time detection of the target scene. High-quality video stream means that the video stream remains clear and smooth after all intermediate processing. Accurate and rapid anomaly tampering detection means that the detection function can quickly filter out abnormal frames from the real-time video stream acquired by the camera and continuously output detection results.

[0005] Camera video streams are susceptible to interference during transmission, resulting in minor, irrelevant changes in the image, such as color variations and rotation. This increases the difficulty of detecting these subtle changes and necessitates new solutions.

[0006] Original works on new media platforms may be maliciously altered and then reposted, harming the personal interests of the original authors. The diverse forms of reposting mean that reposted videos may undergo geometric transformations, but not content transformations. Distinguishing between geometric transformations and content transformations is challenging and requires the introduction of new technologies such as deep learning to assist in solving this problem. Summary of the Invention

[0007] This invention addresses the shortcomings of existing technologies, such as the lack of proactive attack defense and detection of subtle content changes. It proposes a deep learning-based automatic image tampering detection method, storage medium, and device, comprising four steps: digital watermark embedding, image similarity measurement, detection model building, and tampering detection. In digital watermark embedding, the semi-fragile watermark characteristic is integrated into the keyframe based on the least significant bit replacement (LSB) method, enhancing the image's sensitivity to subtle changes. In image similarity measurement, comprehensive judgment of tampering is achieved through the selection of measurement indicators. In the detection model building, an automatic tampering detection deep learning model is constructed. Finally, in tampering detection, multiple similarity metrics and a convolutional neural network (CNN) are combined for training, thereby achieving automatic detection of tampering. This invention achieves accurate and rapid tampering identification and can be applied to various new media video types, such as short videos, providing a solution for the supervision of the authenticity of various video images.

[0008] To achieve the above objectives, the technical solution adopted by this invention is: an automatic detection method for image anomaly tampering based on deep learning, comprising the following steps:

[0009] S1. Digital watermark embedding: The video stream keyframe extraction algorithm extracts keyframes from the video stream, records the keyframe sequence number in the video stream, and embeds the digital watermark into the keyframe based on the least significant bit substitution method in the spatial domain combined with the semi-fragile digital watermark characteristics. After watermark embedding, the image is stored in the database. For a single image, the digital watermark is directly embedded, the index is recorded, and it is stored in the database.

[0010] S2. Image Similarity Measurement: Set evaluation metrics. Using the sequence number or single image index recorded during keyframe extraction in step S1, find the stored initial image in the database and compare it with the image to be detected to determine the similarity between the images. The evaluation metrics include at least Euclidean distance, scale-invariant feature transformation, pixel matching algorithm, and normalized mutual information.

[0011] S3. Establish a detection model: The detection model is a deep learning model based on a convolutional neural network, including at least an adaptive feature fusion module, a fully connected layer, and a global average pooling layer. The adaptive feature fusion module adaptively fuses high-level visual features of the image and similarity metrics. The global average pooling layer performs global average pooling on the input feature map. The fully connected layer maps the fused features to the output. In model optimization, a binary cross-entropy loss function is used to calculate the loss value between the output feature map and the true label. An optimizer is introduced to obtain the gradient value of each weight through backpropagation, and an adaptive learning rate algorithm is used to dynamically adjust the weights. In each iteration, the model calculates the weight gradient, updates the first and second moment estimates, and completes the weight update based on the corrected estimates until all training rounds are completed.

[0012] S4. Abnormal Tampering Detection: Establish a dataset for training the detection model. The dataset contains multiple sets of images, each set containing at least one original image and an image after attack simulation. Input the dataset into the detection model established in step S3 for pre-training of the detection model. Apply the pre-trained detection model to the actual image tampering detection task to determine whether the image to be detected has been maliciously tampered with.

[0013] As an improvement of the present invention, in step S1, the video stream keyframe extraction algorithm converts each frame in the video stream from the original color space to the LUV color space. Then, for each frame, it is compared with the previous frame, and the sum of the absolute differences of all pixels in the current frame and the previous frame in the three components of the LUV color space is calculated. The average difference is then divided by the total number of pixels to obtain the average value of the difference. This average value is stored as the difference feature value of the current frame. As the video is read frame by frame, a sequence containing the difference feature values ​​of all frames is generated. The difference feature value sequence is processed by a weighted sliding window smoothing method. The local maximum value of the smoothed difference value sequence is found, and the frame corresponding to the value is selected as the keyframe.

[0014] As another improvement of the present invention, the index calculation in step S2 includes the following specific process:

[0015] When calculating the Euclidean distance index, firstly, two images are read and converted into grayscale images. Then, the image data is converted into NumPy arrays. According to a preset default threshold, the pixel values ​​in the two NumPy arrays are binarized: if the pixel value is higher than the threshold, it is set to 1; otherwise, it is set to 0. Then, the square root of the sum of the squares of the differences between each pixel in the binarized image is taken to obtain the Euclidean distance between the two images, thus quantifying the degree of difference between the two images.

[0016] During the scale-invariant feature transformation calculation, Gaussian blur is first applied to the two grayscale images to reduce noise and details, making edges clearer and enhancing the robustness and accuracy of feature detection. The SIFT detector is initialized to extract keypoints and descriptors from the two images, and the FLANN matcher is initialized and configured to use the KD-tree algorithm, constructing five KD-trees with a search parameter of 50 for fast matching of these descriptors. During the matching process, the FLANN matcher returns the two nearest neighbor matches for each keypoint and filters the matching results by applying a ratio test with a threshold of 0.7, finally returning the filtered matching results.

[0017] When calculating the pixel matching algorithm metrics, the Pixelmatch library is first used to compare the color values ​​of corresponding pixels in two images pixel by pixel, and a color difference threshold is set to determine the similarity between pixels. During this process, the includeAA parameter is set to True to strictly account for the impact of anti-aliasing on pixel comparison, ensuring the rigor of the comparison process. For pixels determined to be mismatched, Euclidean distance is used to quantify their color difference, and the total number of mismatched pixels is finally returned.

[0018] When calculating the normalized mutual information index, firstly, two images are read and converted from the BGR color space to the Lab color space. Then, the pixel values ​​of the two Lab images are flattened into a two-dimensional array, with each row representing the three channel values ​​of a pixel. The K-means++ clustering algorithm is used to cluster the pixels of the two images, converting the continuous color values ​​of the pixels into discrete category labels. The mutual information is calculated and normalized based on the category labels obtained from the clustering. Finally, the normalized mutual information value is returned.

[0019] As another improvement of the present invention, step S3, establishing the detection model, specifically includes the following steps:

[0020] S31. Feature Extraction: The high-level visual features of the image extracted by the convolutional neural network and the calculated similarity metric are fed into the detection model together.

[0021] S32. Feature Fusion: After feature extraction in step S31, the high-level visual features of the image and the calculated similarity metric are input into the adaptive feature fusion module. They are concatenated along the channel dimension to form a fused feature tensor, which is then fed into the convolutional layer of the adaptive feature fusion module. One half of the convolutional layer is used to extract intermediate features from the fused feature tensor, and the other half maps the intermediate features to a single-channel feature map. The single-channel feature map is normalized using the Softmax function to generate a weight tensor where each weight value is in the range [0,1]. By multiplying the single-channel feature map and the weight tensor element by element in the adaptive feature fusion stage, the fused feature tensor is obtained.

[0022] S33, Global Average Pooling: Input the feature tensor obtained in step S32 into the global average pooling layer of the detection model to perform a compression operation on the spatial dimension of the feature tensor; the global average pooling transforms the two-dimensional feature map corresponding to each feature channel into a single value and removes redundant information.

[0023] S34: Fully connected layer processing: The feature vector after global average pooling in step S33 is input into the fully connected layer of the detection model. The fully connected layer performs weighted summation and non-linear mapping on the input features through linear transformation and Sigmoid activation function.

[0024] As an improvement of the present invention, the dataset construction process in step S4 includes the following steps:

[0025] S41: During the dataset construction process, the original image is assigned the label 0, and the labels of the tampered images are assigned according to different detection requirements. The detection requirements specifically include: in different application scenarios and network conditions, the tampering attack types corresponding to the content judged to be maliciously tampered are different. When labeling the dataset images, images that are not judged to be maliciously tampered under the current situation are labeled as 0, and the remaining images that need to be judged to have been maliciously tampered are labeled as 1. Then, the images are loaded with the PIL library and subjected to standardized preprocessing. The preprocessing includes at least adjusting the image size and performing normalization operations. For each pair of original images and attacked images, the similarity metric mentioned in S2 is calculated. The preprocessed images, the calculated similarity metric, and the corresponding labels are integrated to construct a batch dataset, which is then packaged into a TensorDataset and a DataLoader is created.

[0026] S42: The detection model is pre-trained on the constructed dataset; after the feature extraction step in S31 and the feature fusion step in S32, the detection model uses the binary cross-entropy loss function to calculate the loss value between the mapping result obtained by the fully connected layer and the true label in 10 training rounds. The gradient value of each weight is obtained through backpropagation of the optimizer, and the weights are dynamically adjusted using the adaptive learning rate algorithm until all training rounds are completed, so that the value of the loss function reaches the minimum and the optimal weights are obtained.

[0027] S43: Apply the pre-trained detection model to the image tampering detection task; the detection model first obtains the index corresponding to the image to be detected, and then queries and retrieves the corresponding initial image from the database through the index; the detection model calculates the similarity evaluation index in step S2, and calculates the fusion feature tensor in step S3. The saved optimal weights are loaded and multiplied by the fusion feature tensor to obtain the tampering probability, which is compared with the threshold of 0.5. If the probability is greater than 0.5, it is judged as malicious tampering.

[0028] To achieve the above objectives, the present invention also adopts the following technical solution: a computer-readable storage medium storing a computer program thereon, which is executed by a processor to implement the image abnormal tampering automatic detection method based on deep learning as described in any one of claims 1-6.

[0029] To achieve the above objectives, the present invention also adopts the following technical solution: a computer device, comprising:

[0030] Memory, used to store instructions;

[0031] A processor is configured to execute the instructions, causing the computer device to perform the deep learning-based automatic image tampering detection method as described in any one of claims 1-6.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] (1) Least Significant Bit (LSB) digital watermarking technology can embed watermarks without significantly affecting the visual quality of the image. Semi-fragile digital watermarking combines robustness and fragility, allowing it to tolerate a certain degree of normal signal processing operations (such as compression, slight noise interference, etc.) while remaining highly sensitive to malicious tampering. In the digital watermark embedding step S1 of this invention, the above two digital watermarking technologies are combined to embed watermarks into the video frame. The sensitivity of the video frame or image to malicious tampering can be significantly improved, and the camera video stream can also meet the requirements of low latency and high quality transmission.

[0034] (2) When measuring image similarity, the method of the present invention combines a variety of evaluation indicators with different advantages, thus solving the problem of limitations of a single evaluation indicator.

[0035] (3) By simulating the structure of the biological visual system, the convolutional neural network extracts local features of the image using convolutional and pooling layers, which can achieve efficient feature learning and pattern recognition. The detection model of this invention can train an abnormal tampering detection model on the basis of deep learning and convolutional neural networks by constructing a targeted image dataset, and achieve accurate and fast abnormal tampering identification.

[0036] (4) When automatically identifying abnormal image tampering, the method of the present invention, through the selection of metric indicators and the training of a deep learning model, can eliminate the interference of normal geometric transformations, improve the sensitivity to malicious tampering such as content changes, and alert users to abnormal transformations that truly require detection. Furthermore, the method of the present invention can adjust the detection intensity according to the application scenario, effectively reducing the false detection rate and improving the user experience.

[0037] (5) The present invention has a complete method system, which constructs a comprehensive, innovative and highly integrated solution, and designs a seamless system for users from video stream or image acquisition to final judgment result output. Attached Figure Description

[0038] Figure 1 This is a flowchart simulating the secure transmission mechanism of camera images in this invention;

[0039] Figure 2 This is a schematic diagram illustrating the construction of the dataset for the automatic image tampering detection model in this invention;

[0040] Figure 3 This is a schematic diagram of the automatic image tampering detection model structure in this invention;

[0041] Figure 4 This is a schematic diagram of the image abnormal tampering automatic detection method based on deep learning in this invention. Detailed Implementation

[0042] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0043] Example 1

[0044] The main problem addressed by this method is the design of a secure transmission process for camera footage and the automatic detection of malicious tampering of video images. This ensures the authenticity of the images received by camera users and achieves a seamless system from video stream acquisition to final judgment output. Specifically, as follows... Figure 1 As shown, Figure 1 This paper simulates the mechanism and process of secure transmission of camera images, focusing on malicious tampering attacks such as minor content alterations, and automatically outputs detection results, providing users with real-time anomaly detection services. Therefore, a deep learning-based automatic image anomaly tampering detection method is proposed. Figure 4 As shown, the specific steps include the following:

[0045] Step S1: Digital watermark embedding stage.

[0046] The simulated secure transmission process of camera footage in this embodiment begins with capturing a video stream from the camera device. A video keyframe extraction algorithm is then used to precisely extract the most critical frames from the video stream. These most critical frames exhibit the greatest inter-frame difference among all video frames over a given period. Subsequently, watermarking is embedded into these keyframes to enhance their sensitivity to subtle changes. After watermarking, the watermarked keyframe images are uploaded to a storage system for archiving. Once the camera video stream is successfully transmitted to the client, it is again accurately identified and located from the continuous video stream. An image anomaly detection method based on deep learning is used to determine if the video stream is intact and unaltered. If the video stream is intact, the user is directly shown the real-time video feed from the camera. If tampering is detected, the method provides an anomaly alert.

[0047] In the digital watermark embedding stage, the least significant bit substitution method based on the spatial domain is used to integrate the semi-fragile digital watermark characteristics and embed the digital watermark into keyframes. These keyframes are extracted using a video stream keyframe extraction algorithm. The algorithm first converts each frame in the video stream from the original color space to the LUV color space. Then, for each frame, it compares it with the previous frame, calculates the sum of the absolute differences of all pixels in the current frame and the previous frame in the three components of the LUV color space, divides this sum by the total number of pixels to obtain the average difference, and stores this average as the difference feature value of the current frame. As the video is read frame by frame, a sequence containing the difference feature values ​​of all frames is generated. A weighted sliding window smoothing method is used to process the difference feature value sequence, finding the local maximum value in the smoothed difference value sequence, and selecting the frame corresponding to this value as the keyframe.

[0048] Subsequently, by combining spatial domain-based Least Significant Bit (LSB) digital watermarking embedding technology with semi-fragile watermarking technology, digital watermarking is embedded into keyframes. LSB digital watermarking is a spatial domain-based method that embeds watermark information by modifying the least significant bit of image pixels. Since the human eye is insensitive to changes in the LSB of an image, this technology can embed watermarks without significantly affecting the visual quality of the image. Semi-fragile digital watermarking falls between robust and fragile watermarking, possessing both robustness and fragility. Digital watermarks incorporating semi-fragile watermarking technology have minimal impact on the visual quality of the image after embedding, are invisible, and can tolerate a certain degree of normal signal processing operations (such as compression and slight noise interference) while remaining highly sensitive to malicious tampering. By combining these two digital watermarking technologies to embed watermarks into video frames, the sensitivity of video frames to malicious tampering can be significantly improved, and the camera video stream can meet the requirements for low-latency, high-quality transmission.

[0049] Step S2: Image similarity measurement.

[0050] The initial image is retrieved from the database using the sequence number recorded during keyframe extraction and compared with the image to be detected. Similarity metrics assess the similarity of image content. The selected metrics must be able to exclude interference from geometric transformations, filter additions, etc., possess geometrically invariant and scale-invariant features, and ensure sensitivity to local content changes. Therefore, multiple evaluation metrics, such as Euclidean Distance, Scale-Invariant Feature Transform (SIFT), PixelMatch algorithm, and Normalized Mutual Information (NMI), are used in a joint measurement to avoid limitations imposed by a single metric when facing complex tampering methods.

[0051] The Euclidean distance metric refers to the straight-line distance between any two points in a multidimensional space. Its principle is based on the Euclidean geometry formula for the distance between two points, calculated by taking the square root of the sum of the squares of the differences between the two points in each dimension. This metric is simple to use and highly intuitive, but it has drawbacks such as sensitivity to scale and outliers.

[0052] The core principle of the Scale Invariant Feature Transform (SIFT) metric is to simulate the human visual system in detecting and describing local features in an image, ensuring these features remain invariant under different scales, rotations, and lighting conditions. The specific method includes four main steps: First, scale-space extrema detection is performed, using the Difference of Gaussians (DoG) function to find extrema points as feature points at different scales; then, the feature points are precisely located, excluding points with low contrast and edge response; next, the principal orientation of the feature points is determined, assigning a direction value to each feature point using a Gaussian-weighted gradient orientation histogram; finally, a feature descriptor is generated, dividing the region surrounding the feature point into multiple sub-blocks, calculating the gradient orientation histogram within each sub-block, forming a unique vector descriptor with good scale invariance, rotation invariance, and robustness to changes in lighting and viewing angle.

[0053] The pixel matching algorithm (PixelMatch) is a direct comparison method based on the grayscale or color values ​​of image pixels. Its basic principle is to compare the pixels of two images one by one, calculate the degree of difference between corresponding pixels, and then combine the differences of all pixels to obtain an overall matching metric. While the pixel matching algorithm (PixelMatch) is simple and intuitive, it is computationally intensive, and its matching accuracy may be affected when dealing with complex situations such as noise, changes in lighting, and geometric deformations.

[0054] The Normalized Mutual Information (NMI) metric is a key indicator for measuring the correlation between two images. Optimizing the selection of initial cluster centers in cluster analysis using the K-Means++ algorithm enhances the reliability of NMI calculation. The K-Means++ algorithm reduces the similarity between initial cluster centers, improves clustering quality, and accelerates the convergence of clustering results. Mutual information is calculated and normalized based on the clustering results to obtain the normalized mutual information value, which has the advantage of being insensitive to changes in illumination.

[0055] Step S3: Establish the detection model.

[0056] The detection model needs to focus on malicious tampering attacks, requiring it to exclude normal transformations such as rotation and lighting changes, and accurately identify malicious tampering such as minor content alterations. Therefore, to meet the functional requirements of the detection model, the dataset used for pre-training the model needs to be carefully designed, and the image similarity measurement method needs to be carefully selected.

[0057] Define a custom deep learning model, such as Figure 3 As shown, the model includes the main model structure such as the Adaptive Feature Fusion (ASFF) module, the fully connected layer, and the global average pooling layer. The Adaptive Feature Fusion (ASFF) module is used to adaptively fuse high-level visual features and similarity measures of the image. The fully connected layer is used to map the fused features to the output. The global average pooling layer is used to perform global average pooling on the input feature map.

[0058] The adaptive feature fusion (ASFF) module of the detection model is used to adaptively fuse input features. The core idea is to dynamically adjust the importance of different feature channels by learning weights. The specific structure is divided into two parts: initialization and forward propagation.

[0059] In the initialization phase, the goal is to build a network structure that can efficiently process input features and reduce channel dimensionality. To this end, multiple convolutional layers are defined. By properly designing the parameters of these multiple convolutional layers, such as kernel size, number of channels, and activation functions, it can be ensured that the features retain sufficient information for subsequent processing while reducing dimensionality.

[0060] In the forward propagation phase, the main task is to achieve adaptive feature fusion. First, the input features undergo multiple convolutional layers to extract more compact feature representations, representing initial dimensionality reduction and semantic extraction of the input features. These features then enter other convolutional layers to further reduce the number of channels, generating a set of single-channel feature maps. These single-channel feature maps are normalized to obtain a weight tensor. Subsequently, the weight tensor and the single-channel feature maps are combined to complete the weighted fusion of features. The fused features, through adaptive weight adjustment, retain the core information of the input features.

[0061] The global average pooling layer in the detection model is used to perform global average pooling on the adaptively fused feature maps, ultimately compressing the feature map of each channel into a single value. The input feature maps are multi-dimensional data input into the convolutional neural network (CNN), including batch size, number of channels, feature map size, etc.

[0062] The fully connected layer of the detection model maps the pooled features to the output, representing the probability that the image has been tampered with. The fully connected layer uses linear transformations and non-linear mappings to map the dimension of the input features from the sum of the image features and the metric dimensions to 1, outputting a numerical value representing the probability that the image has been tampered with.

[0063] In this embodiment, during the feature extraction process, high-level visual features of the image are extracted using a convolutional neural network. These features, with a shape of [batch size, 3, 256, 256], are used as input to the detection model. This input, along with the calculated similarity metric, with a shape of [batch size, 4, 1, 1], is then fed into a custom detection model. The detection model expands the similarity metric to the same spatial dimension of 256×256 as the high-level visual features of the image to ensure that the two can be fused.

[0064] The high-level visual features of the image and the calculated similarity metric are input into a custom feature fusion module, where they are concatenated along the channel dimension to form a fused feature tensor with the shape [batch size, 7, 256, 256]. This fused feature tensor is then fed into the core of the adaptive feature fusion module, namely the convolutional layer. One half of the convolutional layer extracts more representative intermediate features from the fused feature tensor, mapping the 7 input channels to 4 channels and changing the tensor shape to [batch size, 4, 256, 256] to reduce feature redundancy. The other half of the convolutional layer further maps the intermediate features into single-channel feature maps with the shape [batch size, 1, 256, 256]. The single-channel feature maps are then normalized using the Softmax function to generate a weight tensor where each weight value is in the range [0, 1]. Finally, the fused feature tensor is obtained by element-wise multiplying the single-channel feature maps and the weight tensor during the adaptive feature fusion stage.

[0065] The feature tensors processed by the feature fusion module are input to the global average pooling layer. This layer primarily performs compression operations on the spatial dimensions of the feature tensors, namely their height and width. Global average pooling transforms the two-dimensional feature map corresponding to each feature channel into a single numerical value, thereby preserving the global information of the feature map while removing local details and redundant information contained in the spatial dimension. The global average pooling layer generates a 1×1 feature vector by calculating the arithmetic mean of all pixel values ​​in each feature channel, transforming the feature vector shape into [batch size, 1, 1, 1].

[0066] The feature vectors, after global average pooling, are input into the fully connected layer of the detection model. The fully connected layer performs a series of linear transformations and a sigmoid activation function to weighted summation and non-linear mapping of the input features. In this model, the output dimension of the fully connected layer is set to 1, meaning it will generate a single numerical output representing the probability that the image has been attacked.

[0067] During the model optimization phase, the binary cross-entropy loss function is first used to compare the model's output feature map with the true labels and calculate the loss value. After obtaining the loss value, the optimizer calculates the gradient value of each weight through backpropagation, which is used for weight updates.

[0068] The detection model employs an adaptive learning rate optimization algorithm to calculate weights, aiming for optimal convergence. The optimizer defines and initializes parameters including the learning rate and hyperparameters. These parameters dynamically adjust the learning rate during training to adapt to varying gradient changes. In each iteration, the detection model first calculates the gradient value of each weight, then updates the first and second moment estimates. Combining the iteration count, it calculates bias-corrected estimates of these two values, ultimately updating the gradient and adjusting the weights based on the corrected estimates. This iterative process continues until all training rounds are completed, and the final trained model weights are saved in a file format. This algorithm allows the detection model to progressively adjust its weights to minimize the loss function, thereby improving its learning performance.

[0069] Step S4: Abnormal tampering detection.

[0070] Pre-trained deep learning models are used for automatic discrimination. The detection model is trained by combining multiple similarity metrics and convolutional neural networks (CNNs). Through feature selection, data augmentation and other methods, it focuses on capturing changes in image content and eliminates interference from geometric transformations, filter additions and other situations, providing accurate and effective protection for camera users.

[0071] The process of constructing the dataset used for model training is as follows: Figure 2 As shown, the dataset used for pre-training the abnormal tampering detection model contains multiple image sets, each set containing an original image and an image after attack simulation. The attack simulation includes minor alterations, filters, rotations, cropping, and other attack techniques to simulate malicious tampering that might occur in real-world situations. The dataset undergoes labeling, training set partitioning, and image preprocessing to form training and test sets. During labeling, the judgment result of the images in the dataset is defined, distinguishing between abnormal tampering that needs to be detected and normal transformations that do not need to be detected, marked with 1 and 0 respectively. During training set partitioning, all image sets are divided into half as the training set and the other half as the test set.

[0072] The dataset construction process utilizes a wide range of original images, including natural landscapes, urban street scenes, portraits, and photographs of various objects, to accurately simulate real-world scenarios. Subsequently, simulated attacks are applied to the original images to ensure that the generated altered images not only conform to the characteristics of malicious attacks but also maintain a certain visual correlation with the original images, thus providing high-quality data samples for subsequent model training.

[0073] The simulated attacks include various specific methods, such as: meticulously adjusting parameters such as pixel values ​​and color channel intensity of the image to ensure that the changes are subtle and difficult to detect, yet sufficient to potentially affect the authenticity of the image; adding filters of different styles, such as Gaussian blur and color enhancement, to simulate situations where malicious tamperers use filters to cover up the original information of the image or create misleading visual effects.

[0074] The dataset assigns corresponding labels to the original and tampered images to clarify their category. Specifically, the original images are assigned label 0, while the labels of tampered images are assigned according to detection requirements. These requirements include that the types of tampering attacks corresponding to content judged as maliciously tampered differ in different application scenarios and network conditions. When labeling images in the dataset, images not judged as maliciously tampered under the given circumstances are labeled 0, while other images judged as maliciously tampered are labeled 1. Based on this, images are loaded using the PIL library and subjected to standardized preprocessing, including adjusting image size and performing normalization operations to ensure that the image data meets the input specifications of the detection model. Then, for each pair of images, including the original image and the attacked image, or the original image and itself, multiple similarity metrics are calculated. The preprocessed high-level visual features, calculated similarity metrics, and corresponding labels are integrated to construct a batch dataset, which is then packaged into a TensorDataset and a DataLoader is created. This series of operations provides standardized input data for model training, completing the conversion from original images to a format acceptable to the model. The dataset processed as described above will be input into the automatic image tampering detection model. After feature extraction (S31) and feature fusion (S32), in 10 training epochs, the detection model uses the binary cross-entropy loss function to calculate the loss value between the mapping result of step S34 and the true label. The gradient value of each weight is obtained through backpropagation using an optimizer, and the weights are dynamically adjusted using an adaptive learning rate algorithm until all training epochs are completed, minimizing the loss function and obtaining the optimal weights. Thus, the pre-trained automatic image tampering detection model is obtained.

[0075] The pre-trained automatic image tampering detection model was then applied to a real-world detection task. First, the initial keyframe images stored in the database and the actual images received by the user were acquired and jointly measured using multiple metrics in step S2. Then, the calculated results of Euclidean Distance, Scale Invariant Feature Transform (SIFT), PixelMatch algorithm, and Normalized Mutual Information (NMI) were combined to form a similarity metric, which, along with the image, was input into the automatic image tampering detection model. Step S3 was then used to calculate the fused feature tensor. The saved optimal weights were loaded and multiplied by the fused feature tensor to obtain the tampering probability. This probability was compared to a threshold of 0.5; if the probability was greater than 0.5, malicious tampering was considered.

[0076] Example 2

[0077] This embodiment is a computer device, including a processor and a memory, the memory storing code for executing the methods in the above embodiment.

[0078] The processor can be a multi-core processor or may contain multiple processors. In some embodiments, the processor may include a general-purpose main processor and one or more specialized coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, the processor may be implemented using custom circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0079] Memory can include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM can store static data or instructions required by the processor or other modules of the computer. Permanent storage devices can be read-write storage devices. Permanent storage devices can be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use high-capacity storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices can be removable storage devices (e.g., floppy disks, optical drives). System memory can be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory can store some or all of the instructions and data required by the processor during operation. Furthermore, memory can include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks can also be used. In some implementations, the memory may include removable storage devices that are readable and / or writable, such as laser discs (CDs), read-only digital versatile optical discs (e.g., DVD-ROMs, dual-layer DVD-ROMs), read-only Blu-ray discs, ultra-high density optical discs, flash memory cards (e.g., SD cards, mini SD cards, Micro-SD cards, etc.), magnetic floppy disks, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0080] The memory stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0081] Example 3

[0082] This embodiment provides a non-transitory machine-readable storage device that stores executable code. When the executable code is executed by a processor of an electronic device, the processor performs the method described in the above embodiment.

[0083] A non-transitory machine-readable storage device (or computer-readable storage device, or machine-readable memory) storing executable code (or computer program, or computer instruction code) that, when executed by a processor of an electronic device (or computing device, server, etc.), causes the processor to perform the steps of the method described above according to the present invention.

[0084] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both.

[0085] In summary, the present invention's method, in the digital watermark embedding part, selects the least significant bit replacement method as the basis for watermark embedding, based on the requirement of ensuring clarity and lightweight transmission of camera images. To make the watermark sensitive to malicious tampering and robust to bona fide attacks, the watermark embedding algorithm incorporates semi-fragile watermark characteristics on the basis of the least significant bit replacement method. In the image similarity measurement part, combining the initial image captured by the device and the image to be detected stored in the database, image metrics such as Euclidean distance, scale-invariant feature transform (SIFT), pixel matching algorithm (PixelMatch), and normalized mutual information (NMI) are calculated for subsequent measurement of image similarity. In the automatic abnormal tampering detection part, a trained image tampering automatic detection deep learning model is used for judgment. The model is trained by combining multiple similarity metrics and convolutional neural networks to achieve automatic detection of abnormal tampering. The present invention's method can automatically detect malicious tampering of camera video images through a pre-trained model. This technology can also be applied to various new media video types such as short videos, providing a solution for the supervision of the authenticity of various video images.

[0086] It should be noted that the above content merely illustrates the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, various improvements and modifications can be made without departing from the principle of the present invention, and all such improvements and modifications fall within the scope of protection of the claims of the present invention.

Claims

1. A method for automatic detection of image anomaly tampering based on deep learning, characterized in that, The steps include the following: S1. Digital watermark embedding: Keyframes in the video stream are extracted using a video stream keyframe extraction algorithm. The sequence number of the keyframe in the video stream is recorded. The digital watermark is embedded in the keyframe by fusing the semi-fragile digital watermark characteristics based on the least significant bit substitution method in the spatial domain. After the watermark is embedded, the image is stored in the database. For a single image, a digital watermark is directly embedded, and an index is recorded and stored in a database. The method for extracting keyframes using a video stream keyframe extraction algorithm is as follows: each frame in the video stream is converted from the original color space to the LUV color space, the difference feature value of the current frame is calculated, a sequence of difference feature values ​​containing all frames is generated, the sequence is smoothed, and the local maximum value of the smoothed difference value sequence is the keyframe. The difference feature value is the sum of the absolute differences of all pixels in the current frame and the previous frame in the three components of the LUV color space, divided by the total number of pixels. S2. Image Similarity Measurement: Set evaluation metrics. Using the sequence number or single image index recorded during keyframe extraction in step S1, find the stored initial image in the database and compare it with the image to be detected to determine the similarity between the images. The evaluation metrics include at least Euclidean distance, scale-invariant feature transformation, pixel matching algorithm, and normalized mutual information. S3. Establish a detection model: The detection model is a deep learning model based on a convolutional neural network, including at least an adaptive feature fusion module, a fully connected layer, and a global average pooling layer. The adaptive feature fusion module adaptively fuses high-level visual features of the image and similarity metrics. The fully connected layer maps the fused features to the output. The global average pooling layer performs global average pooling on the input feature map. In model optimization, a binary cross-entropy loss function is used to calculate the loss value between the output feature map and the true label. An optimizer is introduced to obtain the gradient value of each weight through backpropagation, and an adaptive learning rate algorithm is used to dynamically adjust the weights. In each iteration, the model calculates the weight gradient, updates the first and second moment estimates, and completes the weight update based on the corrected estimates until all training rounds are completed. S4. Abnormal Tampering Detection: Establish a dataset for training the detection model. The dataset contains multiple sets of images, each set containing at least one original image and an image after attack simulation. Input the dataset into the detection model established in step S3 for pre-training of the detection model. Apply the pre-trained detection model to the actual image tampering detection task to determine whether the image to be detected has been maliciously tampered with.

2. The image anomaly tampering automatic detection method based on deep learning as described in claim 1, characterized in that: The specific calculation methods for each evaluation index in step S2 are as follows: The Euclidean distance is obtained by: reading two images and converting them into grayscale images, converting the image data into NumPy arrays, and binarizing the pixel values ​​in the two NumPy arrays according to a preset default threshold: if the pixel value is higher than the threshold, it is set to 1; otherwise, it is set to 0; and taking the square root of the sum of the squares of the differences between each pixel in the binarized image to obtain the Euclidean distance between the two images. The scale-invariant feature transformation involves applying Gaussian blur to two grayscale images, initializing a SIFT detector to extract key points and descriptors from the two images, initializing a FLANN matcher, and using a KD tree algorithm for matching. During the matching process, the FLANN matcher returns the two nearest neighbor matches for each key point, and the matching results are filtered by applying a ratio test. Finally, the filtered matching results are returned. The pixel matching algorithm uses the Pixelmatch library to compare the pixel color values ​​of corresponding positions in two images pixel by pixel, and sets a color difference threshold to determine the similarity between pixels; for pixels that are determined to be mismatched, the color difference is quantized using Euclidean distance, and finally the total number of mismatched pixels is returned. The normalized mutual information is obtained by converting two images from the BGR color space to the Lab color space, flattening the pixel values ​​of the two Lab images into a two-dimensional array, with each row representing the three channel values ​​of a pixel, using the K-means++ clustering algorithm to cluster the pixels of the two images, converting the continuous color values ​​of the pixels into discrete category labels, calculating the mutual information based on the category labels obtained from the clustering and normalizing it, and finally returning the normalized mutual information value.

3. The image anomaly tampering automatic detection method based on deep learning as described in claim 1, characterized in that: Step S3, establishing the detection model, specifically includes the following steps: S31. Feature Extraction: The high-level visual features of the image extracted by the convolutional neural network, together with the similarity metric calculated in step S2, are entered into the detection model. S32. Feature Fusion: After feature extraction in step S31, high-level visual features and similarity metrics of the image are input into the adaptive feature fusion module. They are concatenated along the channel dimension to form a fused feature tensor, which is then fed into the convolutional layer of the adaptive feature fusion module. One half of the convolutional layer is used to extract intermediate features from the fused feature tensor, and the other half maps the intermediate features to single-channel feature maps. After normalization, the single-channel feature maps generate weight tensors. By multiplying the single-channel feature maps and weight tensors element-wise in the adaptive feature fusion stage, the fused feature tensor is obtained. S33, Global Average Pooling: Input the feature tensor obtained in step S32 into the global average pooling layer of the detection model to perform a compression operation on the spatial dimension of the feature tensor; the global average pooling transforms the two-dimensional feature map corresponding to each feature channel into a single value and removes redundant information. S34: Fully connected layer processing: The feature vector after global average pooling in step S33 is input into the fully connected layer of the detection model. The fully connected layer performs weighted summation and non-linear mapping on the input features through linear transformation and Sigmoid activation function.

4. The image anomaly tampering automatic detection method based on deep learning as described in claim 1, characterized in that: The abnormal tampering detection step S4 specifically includes the following steps: S41: Construct a dataset, label each image, determine whether it needs to be identified as tampered with, perform standardized preprocessing, calculate the similarity metric for each pair of original images and attacked images through step S2, and integrate the preprocessed images, similarity metrics and corresponding labels to construct a batch dataset. S42: Input the dataset into the detection model to perform pre-training of the detection model; The images in the dataset undergo feature extraction to obtain high-level visual features, which are then fused with similarity metrics. Then, the detection model uses the binary cross-entropy loss function to calculate the loss value between the mapping result obtained by the fully connected layer and the true label. The gradient value of each weight is obtained through backpropagation of the optimizer, and the weights are dynamically adjusted using the adaptive learning rate algorithm until all training rounds are completed, the loss function value is minimized, and the optimal weights are obtained. S43: Apply the pre-trained detection model to the actual image tampering detection task to determine whether the image to be detected has been maliciously tampered with.

5. The image anomaly tampering automatic detection method based on deep learning as described in claim 4, characterized in that: The preprocessing of the dataset in step S41 includes at least adjusting the image size and performing normalization operations.

6. A computer-readable storage medium, characterized in that: It stores a computer program that is executed by a processor to implement the deep learning-based automatic detection method for image anomaly tampering as described in any one of claims 1-5.

7. A computer device, characterized in that: include: Memory, used to store instructions; A processor is configured to execute the instructions, causing the computer device to perform the deep learning-based automatic image anomaly tampering detection method as described in any one of claims 1-5.

Images