Deep learning-based image abnormal tampering automatic detection method, storage medium and equipment

Abstract

The invention discloses an image abnormal tampering automatic detection method based on deep learning, a storage medium and equipment, and the method comprises the four steps: digital watermark embedding, image similarity measurement, detection model establishment and abnormal tampering detection. A digital watermark is embedded into a key frame by fusing semi-fragile watermark characteristics on the basis of a least significant bit replacement (LSB) method, so that the sensitivity of an image to tiny changes is improved; in the image similarity measurement, all-directional judgment on abnormal tampering is realized through measurement index screening; in establishment of the detection model, a basis of an abnormal tampering automatic detection deep learning model is constructed; and finally, in abnormal tampering detection, training is carried out in combination with a plurality of similarity measurement indexes and a convolutional neural network (CNN), and automatic detection of abnormal tampering is realized. The method realizes rapid and accurate abnormal tampering identification, can be used in various new media video types such as short videos and the like, and provides a solution for authenticity supervision of various video pictures.

Description

Technical Field

[0001] The present invention belongs to the technical field of information security, and relates to new applications of deep learning and digital watermark-related technologies in real-time detection of the authenticity of various new media video images such as camera images and short videos. It mainly involves a method, storage medium and device for automatic detection of image anomaly tampering based on deep learning. Background Art

[0002] Cameras, ubiquitous in our daily lives, are essential tools for capturing digital image information. The image data they capture is widely disseminated and applied across numerous industries. Therefore, protecting image data from unauthorized tampering and ensuring its security during acquisition, storage, processing, and transmission is crucial.

[0003] Numerous studies are currently underway on image protection for various types of video streams and images, but most focus on defending against passive attacks, neglecting the defense and alerting of active attacks. Some studies have successfully detected subtle changes in text and images by training deep learning models, but these have limited practical applications. Other studies focus on encrypting camera images to ensure privacy is not compromised during transmission, but they cannot guarantee the authenticity of the camera images received by users. In areas such as evidence identification, methods for verifying video streams and camera images focus on post-extraction verification, but lack the ability to detect whether evidence has been destroyed or altered before extraction, making it difficult to prevent malicious human intervention.

[0004] A secure and efficient camera image transmission mechanism must meet three requirements: low-latency transmission, high-quality video streams, and accurate and rapid detection of abnormal tampering. Low-latency transmission means that the transmission time from the camera device to the user terminal is short, meeting the camera product's requirement for providing real-time detection of target scenes. High-quality video streams mean that the video stream remains clear and smooth after all intermediate processing steps. Accurate and rapid detection of abnormal tampering means that the detection function can quickly filter out abnormal frames in the video stream captured by the camera in real time and continuously output detection results.

[0005] The camera video stream is easily interfered with during transmission, resulting in minor changes in the image content, such as color changes and rotation. This increases the difficulty of detecting minor content changes to a certain extent, and requires new solutions.

[0006] Original works on new media platforms can be maliciously altered and reposted, harming the original creators' interests. The diverse nature of reposting can lead to geometric transformations in reposted videos without altering their content. Distinguishing between geometric and content transformations is challenging, requiring the introduction of new technologies such as deep learning. Summary of the Invention

[0007] The present invention is aimed at the problems of the lack of active attack defense and detection of small content changes in the existing technology, and proposes a method, storage medium and device for automatic detection of abnormal image tampering based on deep learning, including four steps: digital watermark embedding, image similarity measurement, establishment of detection model and abnormal tampering detection. In digital watermark embedding, the digital watermark is embedded in the key frame by integrating the semi-fragile watermark characteristics based on the least significant bit replacement method (LSB), thereby improving the sensitivity of the image to small changes; in image similarity measurement, the metric index screening is used to achieve a full range of judgment on abnormal tampering; in the establishment of the detection model, a deep learning model for automatic detection of abnormal tampering is constructed; finally, in abnormal tampering detection, multiple similarity measurement indicators and convolutional neural network (CNN) are combined for training to achieve automatic detection of abnormal tampering. The method of the present invention realizes accurate and rapid abnormal tampering identification, which can be used in various new media video types such as short videos, and provides a solution for the authenticity supervision of various video images.

[0008] To achieve the above objectives, the present invention adopts a technical solution: a method for automatically detecting abnormal image tampering based on deep learning, comprising the following steps:

[0009] S1. Digital watermark embedding: The video stream key frame extraction algorithm extracts the key frames in the video stream, records the key frame sequence number in the video stream, and embeds the digital watermark into the key frame based on the least significant bit replacement method in the spatial domain combined with the semi-fragile digital watermarking characteristics. After the watermark is embedded, the image is stored in the database; for a single image, the digital watermark is directly embedded, and the index is recorded and stored in the database;

[0010] S2. Image similarity measurement: setting evaluation indicators, using the serial number or single image index recorded during key frame extraction in step S1, finding the stored initial image from the database, comparing and evaluating it with the image to be detected, and determining the similarity of the images; the evaluation indicators include at least Euclidean distance, scale-invariant feature transform, pixel matching algorithm, and normalized mutual information;

[0011] S3. Establish a detection model: The detection model is a deep learning model based on a convolutional neural network, which at least includes an adaptive feature fusion module, a fully connected layer and a global average pooling layer; wherein the adaptive feature fusion module is used to adaptively fuse high-level visual features of the image and similarity metrics; the global average pooling layer is used to perform global average pooling on the input feature map; the fully connected layer is used to map the fused features to the output; in model optimization, a binary cross entropy loss function is used to calculate the loss value between the output feature map and the true label, an optimizer is introduced to obtain the gradient value of each weight through back propagation, and the weight is dynamically adjusted using an adaptive learning rate algorithm. In each iteration, the model calculates the weight gradient, updates the first-order and second-order moment estimates, and completes the weight update based on the corrected estimate until all training rounds are completed;

[0012] S4. Abnormal tampering detection: Establish a data set for detection model training. The data set contains multiple image sets, and each set contains at least one original image and its image after attack simulation. Input the data set into the detection model established in step S3 to pre-train the detection model. Apply the pre-trained detection model to the actual image tampering detection task to determine whether the image to be detected has been maliciously tampered with.

[0013] As an improvement of the present invention, in step S1, the video stream key frame extraction algorithm converts each frame in the video stream from the original color space to the LUV color space, and then for each frame, compares it with the previous frame, calculates the sum of the absolute differences of the three components of all pixels of the current frame and the previous frame in the LUV color space, and divides it by the total number of pixels to obtain the average value of the difference, and stores the average value as the difference feature value of the current frame; as the video is read frame by frame, a sequence containing the difference feature values ​​of all frames will be generated, and the weighted sliding window smoothing method will be used to process the difference feature value sequence, and the local maximum of the smoothed difference value sequence will be found, and the frame corresponding to the value will be selected as the key frame.

[0014] As another improvement of the present invention, the index calculation in step S2 includes the following specific process:

[0015] When calculating the Euclidean distance metric, the two images are first read and converted into grayscale images. The image data is then converted into NumPy arrays. The pixel values ​​in the two NumPy arrays are binarized based on a preset default threshold: if the pixel value is above the threshold, it is set to 1; otherwise, it is set to 0. The square root of the sum of the differences between the two binarized images is then taken pixel by pixel to obtain the Euclidean distance between the two images, quantifying the degree of difference between the two images.

[0016] When calculating the scale-invariant feature transform, Gaussian blur is first applied to the two grayscale images to reduce noise and details, make edges clearer, and enhance the robustness and accuracy of feature detection. The SIFT detector is initialized to extract key points and descriptors from the two images, and the FLANN matcher is initialized and configured to use the KD tree algorithm. Five KD trees are constructed with a search parameter of 50 to quickly match these descriptors. During the matching process, the FLANN matcher returns the two nearest neighbor matches for each key point and filters the matching results by applying a ratio test with a ratio test threshold set to 0.7. The filtered matching results are finally returned.

[0017] When calculating pixel matching metrics, the Pixelmatch library is first used to compare the color values ​​of corresponding pixels in the two images, setting a color difference threshold to determine pixel similarity. During this process, the includeAA parameter is set to True to strictly account for the impact of anti-aliasing on pixel comparisons, ensuring a rigorous comparison process. For pixels determined to be mismatched, the Euclidean distance is used to quantify their color differences, and the total number of mismatched pixels is ultimately returned.

[0018] When calculating the normalized mutual information indicator, first read the two images and convert them from the BGR color space to the Lab color space. The pixel values ​​of the two Lab images are flattened into a two-dimensional array, where each row represents the three channel values ​​of a pixel. The pixels of the two images are clustered using the K-means++ clustering algorithm. The continuous color values ​​of the pixels are converted into discrete category labels. The mutual information is calculated and normalized based on the category labels obtained by clustering. Finally, the normalized mutual information value is returned.

[0019] As another improvement of the present invention, the step S3 of establishing the detection model specifically includes the following steps:

[0020] S31, Feature Extraction: The high-level visual features of the image extracted by the convolutional neural network and the calculated similarity metrics are fed into the detection model together;

[0021] S32, feature fusion: After the feature extraction in step S31, the high-level visual features of the image and the calculated similarity metrics are input into the adaptive feature fusion module, spliced ​​in the channel dimension to form a fused feature tensor, which is then sent to the convolution layer of the adaptive feature fusion module. Half of the convolution layer is used to extract intermediate features from the fused feature tensor, and the other half of the convolution layer maps the intermediate features into a single-channel feature map; the single-channel feature map is normalized by the Softmax function to generate a weight tensor in which each weight value is in the range of [0,1]. The fused feature tensor is obtained by element-by-element multiplication of the single-channel feature map and the weight tensor in the adaptive feature fusion stage;

[0022] S33, global average pooling: input the feature tensor obtained in step S32 into the global average pooling layer of the detection model, and perform a compression operation on the spatial dimension of the feature tensor; the global average pooling converts the two-dimensional feature map corresponding to each feature channel into a single value and eliminates redundant information;

[0023] S34: Fully connected layer processing: The feature vector after the global average pooling processing in step S33 is input to the fully connected layer of the detection model. The fully connected layer performs weighted summation and nonlinear mapping on the input features through linear transformation and Sigmoid activation function.

[0024] As an improvement of the present invention, the data set construction process in step S4 includes the following steps:

[0025] S41: During the dataset construction process, the original image is assigned a label of 0, and the label of the tampered image is assigned according to different detection requirements; the detection requirements specifically include: in different application scenarios and network conditions, the type of tampering attack corresponding to the content determined to be maliciously tampered is different. When marking the dataset images, the images that are not determined to be maliciously tampered in the given situation are marked as 0, and the remaining images that need to be determined to have been maliciously tampered are marked as 1; then, the image is loaded with the help of the PIL library and subjected to standardized preprocessing, which at least includes adjusting the image size and performing normalization operations; for each pair of original images and attacked images, the similarity measurement index described in S2 is calculated; the preprocessed images, the calculated similarity measurement index and the corresponding labels are integrated to construct a batched dataset, which is encapsulated as a TensorDataset, and then a DataLoader is created.

[0026] S42: The detection model is pre-trained on the constructed dataset. After the dataset undergoes feature extraction in S31 and feature fusion in S32, the detection model uses a binary cross-entropy loss function to calculate the loss between the mapping result obtained by the fully connected layer and the true label in 10 training rounds. The gradient value of each weight is obtained through backpropagation of the optimizer, and the weight is dynamically adjusted using an adaptive learning rate algorithm until all training rounds are completed, minimizing the loss function and obtaining the optimal weight.

[0027] S43: Apply the pre-trained detection model to the image tampering detection task. The detection model first obtains the index corresponding to the image to be detected, and then uses the index to query and obtain the corresponding initial image from the database. The detection model calculates the similarity evaluation index through step S2 and the fused feature tensor through step S3. The saved optimal weight is loaded and multiplied by the fused feature tensor to obtain the tampering probability. This is compared with the threshold of 0.5. If it is greater than 0.5, it is judged as malicious tampering.

[0028] In order to achieve the above-mentioned purpose, the present invention also adopts the following technical solution: a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the deep learning-based automatic detection method for image abnormality tampering as described in any one of claims 1-6.

[0029] In order to achieve the above object, the present invention also adopts a technical solution: a computer device comprising:

[0030] a memory for storing instructions;

[0031] A processor is used to execute the instructions so that the computer device performs the deep learning-based automatic detection method for abnormal image tampering as described in any one of claims 1 to 6.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] (1) The least significant bit (LSB) digital watermarking technology can embed watermarks without significantly affecting the visual quality of the image. Semi-fragile digital watermarking has the characteristics of both robustness and fragility. It can tolerate a certain degree of normal signal processing operations (such as compression, slight noise interference, etc.) while remaining highly sensitive to malicious tampering. In the digital watermark embedding step of step S1 of the present invention, the above two digital watermarking technologies are combined to embed watermarks in video frames. The sensitivity of video frames or images to malicious tampering can be significantly improved, and the camera video stream can also meet the low-latency, high-quality transmission requirements.

[0034] (2) The method of the present invention combines a variety of different evaluation indicators when measuring image similarity, solving the problem of being limited by a single evaluation indicator.

[0035] (3) Convolutional neural networks can achieve efficient feature learning and pattern recognition by simulating the structure of biological visual systems and extracting local features of images using convolutional layers and pooling layers. The detection model of the present invention can train an abnormal tampering detection model based on deep learning and convolutional neural networks by constructing targeted image data sets, thereby achieving accurate and rapid abnormal tampering recognition.

[0036] (4) When automatically determining abnormal image tampering, the method of the present invention, through the selection of measurement indicators and the training of deep learning models, can eliminate the interference of normal geometric transformations, improve sensitivity to malicious tampering such as content transformation, and prompt users to detect abnormal transformations that are truly necessary. In addition, the method of the present invention can adjust the detection intensity according to the application scenario, effectively reducing the false detection rate and improving the user experience.

[0037] (5) The method system of the present invention is complete, and it constructs a comprehensive, innovative and highly integrated solution, and designs a seamless connection system for users from video stream or image acquisition to the output of the final judgment result. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a simulation flow chart of the camera image security transmission mechanism in the present invention;

[0039] Figure 2 Schematic diagram of the dataset construction for the image abnormality tampering automatic detection model in the present invention;

[0040] Figure 3 Schematic diagram of the structure of the automatic detection model for abnormal image tampering in the present invention;

[0041] Figure 4 Schematic diagram of the automatic detection method for image abnormality tampering based on deep learning in the present invention. DETAILED DESCRIPTION

[0042] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0043] Example 1

[0044] The main problem solved by this method is the design of the camera image security transmission process and the automatic detection of malicious tampering of the video image, so as to ensure the authenticity of the image received by the camera product user and realize a seamless connection system from video stream acquisition to the final judgment result output. Figure 1 As shown, Figure 1 It simulates the mechanism process of secure transmission of camera images, can focus on malicious tampering attacks such as minor content tampering, and automatically output detection results to provide users with abnormality detection services in real time. Therefore, a deep learning-based automatic detection method for image abnormality tampering, such as Figure 4 As shown, the specific steps include:

[0045] Step S1: Digital watermark embedding stage.

[0046] The simulated camera image security transmission process in this embodiment begins with capturing a video stream from a camera device, and then applying a video key frame extraction algorithm to the video stream to accurately extract the most "critical" partial frames. The most "critical" partial frames have the characteristic of having the largest inter-frame differences among all video frames over a period of time. Subsequently, these key frames are watermarked to enhance their sensitivity to subtle changes. After the watermark embedding is completed, the key frame images with the watermark will be uploaded to the storage system for archiving. After the camera video stream is transmitted normally to the client, it is again accurately identified and located from the continuous video stream. The image abnormality tampering automatic detection method based on deep learning is used to determine the image abnormality tampering. If the video stream is complete and has not been tampered with, the real-time video image from the camera will be directly displayed to the user. If signs of tampering are detected in the video stream, the method will give an abnormal prompt.

[0047] During the digital watermark embedding phase, a least significant bit replacement method based on the spatial domain is integrated with the semi-fragile digital watermarking feature to embed the digital watermark into key frames. These key frames are extracted using a video stream key frame extraction algorithm. The algorithm first converts each frame in the video stream from the original color space to the LUV color space. Each frame is then compared with the previous frame, calculating the sum of the absolute differences in the three components of the LUV color space for all pixels between the current and previous frames. This difference is then divided by the total number of pixels to obtain the average value, which is stored as the difference feature value for the current frame. As the video is read frame by frame, a sequence containing the difference feature values ​​of all frames is generated. A weighted sliding window smoothing method is used to process the difference feature value sequence, finding the local maximum of the smoothed difference value sequence, and selecting the frame corresponding to this value as the key frame.

[0048] Subsequently, a spatial-domain least significant bit (LSB) digital watermarking technique was combined with semi-fragile watermarking to embed digital watermarks into keyframes. LSB digital watermarking, a spatial-domain digital watermarking method, embeds watermark information by modifying the least significant bit (LSB) of an image pixel. Because the human eye is insensitive to changes in the LSB, this technique allows for watermark embedding without significantly affecting the visual quality of the image. Semi-fragile watermarks offer a cross between robust and fragile watermarking, combining both robustness and fragility. After embedding, a semi-fragile watermark has minimal impact on the visual quality of the image, rendering it invisible. Furthermore, it can tolerate a certain degree of normal signal processing (such as compression and minor noise interference) while remaining highly sensitive to malicious tampering. By combining these two digital watermarking techniques to embed watermarks into video frames, the video frames' sensitivity to malicious tampering is significantly improved, enabling the camera video stream to meet low-latency, high-quality transmission requirements.

[0049] Step S2: Image similarity measurement.

[0050] The initial image stored in the database is found using the serial number recorded during keyframe extraction and compared with the image to be detected. Similarity metrics evaluate the similarity of image content. The indicators selected for evaluation are required to be able to exclude interference from geometric transformations, filter additions, etc., and have geometrically invariant features and scale-invariant features while ensuring sensitivity to local content changes. Therefore, multiple evaluation indicators represented by Euclidean distance (Euclidean Distance), scale-invariant feature transform (SIFT), pixel matching algorithm (PixelMatch), and normalized mutual information (NMI) are selected for joint measurement to avoid the limitation of a single indicator when facing complex tampering methods.

[0051] The Euclidean distance metric calculates the straight-line distance between any two points in multidimensional space. It's based on the distance formula between two points in Euclidean geometry and is calculated by taking the square root of the sum of the squares of their differences in each dimension. While this metric is simple to use and intuitive, it suffers from limitations such as sensitivity to scale and outliers.

[0052] The core principle of the scale-invariant feature transform (SIFT) metric is to detect and describe local features in an image by simulating the human visual system, making these features invariant under different scales, rotations, and lighting conditions. The specific method includes four main steps: first, scale space extremum detection is performed, using the Gaussian difference (DoG) function to find extreme points in the space of different scales as feature points; then, the feature points are accurately located, excluding points with low contrast and edge response; then, the main direction of the feature points is determined, and a direction value is assigned to each feature point using a Gaussian-weighted gradient direction histogram; finally, a feature descriptor is generated by dividing the area around the feature point into multiple sub-blocks and calculating the gradient direction histogram within each sub-block to form a unique vector descriptor with good scale invariance, rotation invariance, and robustness to changes in lighting and viewing angle.

[0053] The PixelMatch metric is a direct comparison method based on the grayscale or color values ​​of image pixels. Its basic principle is to compare the pixels of two images one by one, calculate the degree of difference between corresponding pixels, and then combine the differences of all pixels to obtain an overall match metric. While the PixelMatch algorithm is simple and intuitive, it is computationally intensive and can affect matching accuracy when dealing with complex situations such as noise, lighting changes, and geometric deformation.

[0054] The normalized mutual information (NMI) is a key metric for measuring the correlation between two images. Optimizing the selection of initial cluster centers in cluster analysis using the K-Means++ algorithm can enhance the reliability of NMI calculations. The K-Means++ algorithm can reduce the similarity between initial cluster centers, improve clustering quality, and accelerate the convergence of clustering results. Mutual information is calculated and normalized based on the clustering results to obtain a normalized mutual information value, which has the advantage of being insensitive to changes in lighting.

[0055] Step S3: Establish a detection model.

[0056] Detection models must focus on malicious tampering attacks, excluding normal transformations like rotation and light and shadow changes, and accurately identifying malicious tampering, including subtle content manipulation. Therefore, to achieve the functional requirements of the detection model, the dataset used for pre-training the detection model must be carefully designed, and the image similarity measurement method must be carefully selected.

[0057] Define a custom deep learning model, such as Figure 3 As shown in the figure, the model includes the main model structures such as the adaptive feature fusion (ASFF) module, the fully connected layer, and the global average pooling layer. The adaptive feature fusion (ASFF) module is used to adaptively fuse the high-level visual features and similarity metrics of the image, the fully connected layer is used to map the fused features to the output, and the global average pooling layer is used to perform global average pooling on the input feature map.

[0058] The adaptive feature fusion (ASFF) module of the detection model is used to adaptively fuse input features. The core idea is to dynamically adjust the importance of different feature channels by learning weights. The specific structure is divided into two parts: initialization and forward propagation.

[0059] During the initialization phase, the goal is to build a network structure that can efficiently process input features and reduce channel dimensionality. To this end, multiple convolutional layers are defined. By properly designing the parameters of these convolutional layers, such as kernel size, number of channels, and activation functions, we can ensure that features are reduced in dimensionality while retaining sufficient information for subsequent processing.

[0060] In the forward propagation stage, the main task is to achieve adaptive fusion of features. First, the input features are subjected to multiple layers of convolution to extract a more compact feature representation, which represents the initial dimensionality reduction and semantic extraction of the input features. These features then enter other layers of convolution to further reduce the number of channels and generate a set of single-channel feature maps. The single-channel feature maps are normalized to obtain a weight tensor. The weight tensor is then combined with the single-channel feature map to complete the weighted fusion of features. The fused features retain the core information of the input features through the adjustment of adaptive weights;

[0061] The global average pooling layer of the detection model is used to perform global average pooling on the adaptively fused feature maps, ultimately compressing the feature maps of each channel into a single value. The input feature map is the multidimensional data input to the convolutional neural network (CNN), including batch size, number of channels, feature map size, etc.

[0062] The fully connected layer of the detection model is used to map the pooled features to an output representing the probability that the image has been tampered with. The fully connected layer uses linear transformations and nonlinear mappings to map the input feature dimensions from the sum of the image features and the metric dimensions to 1, outputting a numerical value representing the probability that the image has been tampered with.

[0063] In this example, during the feature extraction process, high-level visual features of the image are extracted using a convolutional neural network (CNN) with the shape [batch size, 3, 256, 256]. These features are then fed into the detection model along with the calculated similarity metric (batch size, 4, 1, 1). The detection model then scales the similarity metric to the same spatial dimensions of 256×256 as the high-level visual features to ensure that they can be fused.

[0064] The high-level visual features of the image and the calculated similarity metrics are input into a custom feature fusion module, where they are concatenated along the channel dimension to form a fused feature tensor with a shape of [batch size, 7, 256, 256]. This fused feature tensor is then fed into the core of the adaptive feature fusion module, namely the convolutional layers. Half of the convolutional layers extract more representative intermediate features from the fused feature tensor, mapping the input's seven channels to four channels and changing the tensor shape to [batch size, 4, 256, 256] to reduce feature redundancy. The other half of the convolutional layers further map the intermediate features into single-channel feature maps with a shape of [batch size, 1, 256, 256]. The single-channel feature maps are then normalized using the Softmax function, generating a weight tensor whose weights are in the range [0, 1]. Finally, the fused feature tensor is obtained by element-wise multiplication of the single-channel feature maps with the weight tensor in the adaptive feature fusion stage.

[0065] The feature tensor processed by the feature fusion module is input to the global average pooling layer. This layer performs compression on the spatial dimensions of the feature tensor, namely its height and width. Global average pooling converts the two-dimensional feature map corresponding to each feature channel into a single value, preserving the global information of the feature map while removing local details and redundant information contained in the spatial dimensions. The global average pooling layer calculates the arithmetic mean of all pixel values ​​in each feature channel to generate a 1×1 feature vector, changing the shape of the feature vector to [batch size, 1, 1, 1].

[0066] The feature vector after global average pooling is input to the fully connected layer of the detection model. The fully connected layer performs a weighted summation and nonlinear mapping of the input features through a series of linear transformations and sigmoid activation functions. In this model, the output dimension of the fully connected layer is set to 1, meaning it generates a single numerical output representing the probability that the image has been attacked.

[0067] During the detection model optimization phase, the model's output feature maps are compared with the true labels using a binary cross-entropy loss function to calculate the loss value. After obtaining the loss value, the optimizer calculates the gradient of each weight through backpropagation, which is then used to update the weights.

[0068] The detection model uses an adaptive learning rate optimization algorithm to calculate weights, aiming for optimal convergence. The optimizer defines and initializes parameters, including the learning rate and hyperparameters. These parameters are used to dynamically adjust the learning rate during training to accommodate varying degrees of gradient variation. During each iteration of the algorithm, the detection model first calculates the gradient of each weight, then updates the first- and second-order moment estimates. Based on the number of iterations, these estimates are calculated to correct for bias. Finally, the gradient update and weight adjustment are completed based on these corrected estimates. This iterative process repeats until all training rounds are complete, and the trained model weights are saved in a file format. This algorithm allows the detection model to gradually adjust the model weights to minimize the loss function, thereby improving its learning performance.

[0069] Step S4: Abnormal tampering detection.

[0070] A pre-trained deep learning model is used for automatic identification. The detection model is trained by combining multiple similarity metrics and convolutional neural networks (CNN). Through feature selection, data enhancement and other methods, it focuses on capturing changes in image content, eliminating interference from geometric transformations, filter additions, etc., and providing accurate and effective protection for camera users.

[0071] The process of constructing the dataset used for model training is as follows Figure 2 As shown. The dataset used for pre-training of the abnormal tampering detection model contains multiple sets of images, each set contains an original image and its image after attack simulation. The attack simulation includes attack methods such as minor changes, filters, rotations, and cropping to simulate malicious tampering that may occur in real situations. The dataset undergoes processes such as label marking, training set division, and image preprocessing to form a training set and a test set. In the label marking process, the judgment results of the images in the dataset are defined, and the abnormal tampering that needs to be detected and the normal transformation that does not need to be detected are distinguished and marked with 1 and 0 respectively; in the training set division process, half of all image sets are divided into training sets and the other half are used as test sets;

[0072] The dataset was constructed using a wide range of original images, including natural landscapes, urban street scenes, portraits, and photographs of various objects, to accurately simulate real-world scenarios. Simulated attacks were then performed on the original images to ensure that the generated tampered images both matched the characteristics of malicious attacks and maintained a certain visual relevance to the original images, providing high-quality data samples for subsequent model training.

[0073] The simulated attacks include a variety of specific methods, such as carefully adjusting image parameters such as pixel values ​​and color channel intensities to ensure that the changes are subtle and difficult to detect, but sufficient to potentially affect the authenticity of the image; adding filters of different styles, such as Gaussian blur and color enhancement, to simulate the situation where malicious tamperers use filters to cover up the original information of the image or create misleading visual effects.

[0074] The original and tampered images in the dataset are assigned corresponding labels to clearly define their classification. Specifically, the original image is assigned the label 0, while the tampered image is assigned a label based on the detection requirements. These detection requirements include: content judged as maliciously tampered corresponds to different tampering attack types in different application scenarios and network conditions. When labeling the dataset images, images that are not judged as maliciously tampered are marked as 0, while other images that are judged as maliciously tampered are marked as 1. Based on this, the images are loaded using the PIL library and subjected to standardization preprocessing. This process includes resizing and normalization to ensure that the image data meets the input requirements of the detection model. For each pair of images, whether the original image is associated with the attacked image or the original image is associated with itself, various similarity metrics are calculated. The preprocessed high-level visual features, the calculated similarity metrics, and the corresponding labels are integrated to construct a batched dataset, which is then encapsulated as a TensorDataset and then a DataLoader is created. This series of operations provides standardized input data for model training and completes the conversion from original images to a format acceptable to the model. The dataset that has undergone the above operations will be input into the automatic detection model for image anomaly tampering. After the S31 feature extraction step and the S32 feature fusion step, in 10 training rounds, the detection model uses the binary cross entropy loss function to calculate the loss value between the mapping result of the S34 step and the true label. The gradient value of each weight is obtained through the optimizer backpropagation, and the weight is dynamically adjusted using the adaptive learning rate algorithm until all training rounds are completed, so that the value of the loss function is minimized and the optimal weight is obtained. At this point, the pre-trained automatic detection model for image anomaly tampering is obtained.

[0075] The pre-trained image anomaly tampering automatic detection model is then applied to the actual detection task. First, the initial key frame image stored in the database and the actual image received by the user end are obtained, and the two are jointly measured by multiple indicators through step S2. Then, the results of the measured indicators such as Euclidean distance (Euclidean Distance), scale-invariant feature transform (SIFT), pixel matching algorithm (PixelMatch), normalized mutual information (NMI) are combined to form a similarity measurement index, which is input into the image anomaly tampering automatic detection model together with the image, and the fused feature tensor is calculated through step S3. The saved optimal weight is loaded and multiplied by the fused feature tensor to obtain the tampering probability, which is compared with the threshold of 0.5. When it is greater than 0.5, it is judged as malicious tampering.

[0076] Example 2

[0077] This embodiment is a computer device, including a processor and a memory, wherein the memory stores codes for executing the method in the above embodiment.

[0078] The processor may be a multi-core processor or may include multiple processors. In some embodiments, the processor may include a general-purpose main processor and one or more special coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, the processor may be implemented using customized circuits, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0079] The memory may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that does not lose stored instructions and data even if the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all instructions and data required by the processor during operation. In addition, the memory may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0080] The memory stores executable codes thereon, and when the executable codes are executed by the processor, the processor is enabled to perform the above method.

[0081] Example 3

[0082] This embodiment provides a non-transitory machine-readable memory having executable code stored thereon. When the executable code is executed by a processor of an electronic device, the processor is caused to execute the method in the above embodiment.

[0083] A non-transitory machine-readable memory (or computer-readable memory, or machine-readable memory) having executable code (or computer program, or computer instruction code) stored thereon, which, when executed by a processor of an electronic device (or computing device, server, etc.), enables the processor to perform the various steps of the above-mentioned method according to the present invention.

[0084] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both.

[0085] In summary, in the digital watermark embedding part, the method of the present invention selects the least significant bit replacement method as the basis for embedding watermarks according to the requirements of ensuring clarity and lightweight after the camera image is transmitted. In order to make the watermark sensitive to malicious tampering and robust to good-intentioned attacks, the embedding watermark algorithm integrates the semi-fragile watermark characteristics based on the least significant bit replacement method; in the image similarity measurement part, the device captures the initial image and the image to be detected in combination with the database storage, and calculates image indicators such as Euclidean Distance, Scale Invariant Feature Transform (SIFT), Pixel Matching Algorithm (PixelMatch), Normalized Mutual Information (NMI), etc., which are used to subsequently measure the similarity between the front and back images; in the abnormal tampering automatic judgment part, the image tampering automatic detection deep learning model after training is used for judgment. The model is trained in combination with multiple similarity measurement indicators and convolutional neural networks to achieve automatic detection of abnormal tampering. The method of the present invention can automatically detect malicious tampering of camera video images through a pre-trained model. This technology can also be used in various new media video types such as short videos, providing a solution for the authenticity supervision of various video images.

[0086] It should be noted that the above content merely illustrates the technical idea of ​​the present invention and cannot be used to limit the scope of protection of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications all fall within the scope of protection of the claims of the present invention.

Claims

1. The automatic detection method of image abnormality tampering based on deep learning is characterized by: The steps include: S1. Digital watermark embedding: The key frames in the video stream are extracted through the video stream key frame extraction algorithm, and the sequence numbers of the key frames in the video stream are recorded. The digital watermark is embedded in the key frames based on the least significant bit replacement method in the spatial domain and the semi-fragile digital watermarking characteristics. After the watermark is embedded, the image is stored in the database; For a single image, directly embed the digital watermark, record the index, and store it in the database; S2. Image similarity measurement: setting evaluation indicators, using the serial number or single image index recorded during key frame extraction in step S1, finding the stored initial image from the database, comparing and evaluating it with the image to be detected, and determining the similarity of the images; the evaluation indicators include at least Euclidean distance, scale-invariant feature transform, pixel matching algorithm, and normalized mutual information; S3. Establish a detection model: The detection model is a deep learning model based on a convolutional neural network, which at least includes an adaptive feature fusion module, a fully connected layer and a global average pooling layer; wherein the adaptive feature fusion module is used to adaptively fuse high-level visual features of the image and similarity metrics; the fully connected layer is used to map the fused features to the output; the global average pooling layer is used to perform global average pooling on the input feature map; in model optimization, a binary cross entropy loss function is used to calculate the loss value between the output feature map and the true label, an optimizer is introduced to obtain the gradient value of each weight through back propagation, and an adaptive learning rate algorithm is used to dynamically adjust the weight. In each iteration, the model calculates the weight gradient, updates the first-order and second-order moment estimates, and completes the weight update based on the corrected estimate until all training rounds are completed; S4. Abnormal tampering detection: Establish a data set for detection model training, where the data set contains multiple image sets, and each set contains at least one original image and its image after attack simulation; input the data set into the detection model established in step S3 to pre-train the detection model, and apply the pre-trained detection model to the actual image tampering detection task to determine whether the image to be detected has been maliciously tampered with.

2. The deep learning-based automatic detection method for image anomaly tampering according to claim 1, characterized in that: In step S1, the method for extracting key frames through the video stream key frame extraction algorithm is specifically as follows: each frame in the video stream is converted from the original color space to the LUV color space, the difference feature value of the current frame is calculated, a difference feature value sequence containing all frames is generated, and the sequence is smoothed. The local maximum of the smoothed difference value sequence is the key frame; the difference feature value is the sum of the absolute differences of the three components of all pixels of the current frame and the previous frame in the LUV color space, divided by the average value of the total number of pixels.

3. The deep learning-based automatic detection method for image anomaly tampering according to claim 1, characterized in that: The calculation method of each evaluation index in step S2 is as follows: The Euclidean distance method reads two images and converts them into grayscale images, converts the image data into NumPy arrays, and binarizes the pixel values ​​in the two NumPy arrays according to a preset default threshold: if the pixel value is higher than the threshold, it is set to 1; otherwise, it is set to 0; the square root of the sum of the differences between the binarized images is calculated pixel by pixel, and the Euclidean distance between the two images is obtained; The scale-invariant feature transform: applies Gaussian blur to two grayscale images, initializes a SIFT detector to extract key points and descriptors of the two images, and initializes a FLANN matcher to perform matching using a KD tree algorithm. During the matching process, the FLANN matcher returns two nearest neighbor matches for each key point, and filters the matching results by applying a ratio test, and finally returns the filtered matching results; The pixel matching algorithm uses the Pixelmatch library to compare the pixel color values ​​of corresponding positions in the two images pixel by pixel, and sets a color difference threshold to determine the similarity between the pixels. For pixels that are determined to be mismatched, the Euclidean distance is used to quantify their color differences, and the total number of mismatched pixels is finally returned. The normalized mutual information is as follows: two images are converted from BGR color space to Lab color space, and the pixel values ​​of the two Lab images are flattened into a two-dimensional array, where each row represents the three channel values ​​of a pixel. The pixels of the two images are clustered using the K-means++ clustering algorithm, and the continuous color values ​​of the pixels are converted into discrete category labels. The mutual information is calculated and normalized based on the category labels obtained by clustering, and finally the normalized mutual information value is returned.

4. The deep learning-based automatic detection method for image anomaly tampering according to claim 1, characterized in that: The step S3 of establishing the detection model specifically includes the following steps: S31, feature extraction: The high-level visual features of the image extracted by the convolutional neural network are fed into the detection model together with the similarity metric calculated in step S2; S32, feature fusion: After the feature extraction in step S31, the high-level visual features of the image and the similarity measurement index are input into the adaptive feature fusion module, spliced ​​in the channel dimension to form a fused feature tensor, which is then sent to the convolution layer of the adaptive feature fusion module. Half of the convolution layer is used to extract intermediate features from the fused feature tensor, and the other half of the convolution layer maps the intermediate features into a single-channel feature map; the single-channel feature map is normalized to generate a weight tensor, and the fused feature tensor is obtained by element-by-element multiplication of the single-channel feature map and the weight tensor in the adaptive feature fusion stage; S33, global average pooling: input the feature tensor obtained in step S32 into the global average pooling layer of the detection model, and perform a compression operation on the spatial dimension of the feature tensor; the global average pooling converts the two-dimensional feature map corresponding to each feature channel into a single value and eliminates redundant information; S34: Fully connected layer processing: The feature vector after the global average pooling processing in step S33 is input to the fully connected layer of the detection model. The fully connected layer performs weighted summation and nonlinear mapping on the input features through linear transformation and Sigmoid activation function.

5. The deep learning-based automatic detection method for image anomaly tampering according to claim 1, characterized in that: The abnormal tampering detection step S4 specifically includes the following steps: S41: Construct a dataset, label each image, determine whether it needs to be identified as tampered, perform standardized preprocessing, and calculate a similarity metric for each pair of original and attacked images through step S2; integrate the preprocessed images, similarity metrics, and corresponding labels to construct a batched dataset; S42: Input the data set into the detection model to perform pre-training of the detection model; The images in the dataset are subjected to feature extraction to obtain high-level visual features of the images, and then feature fusion is performed with similarity metrics; Afterwards, the detection model uses a binary cross-entropy loss function to calculate the loss between the mapping result obtained by the fully connected layer and the true label. The gradient value of each weight is obtained through backpropagation of the optimizer, and the weight is dynamically adjusted using an adaptive learning rate algorithm until all training rounds are completed and the loss function value is minimized, thus obtaining the optimal weight. S43: Apply the pre-trained detection model to the actual image tampering detection task to determine whether the image to be detected has been maliciously tampered with.

6. The deep learning-based automatic detection method for image anomaly tampering according to claim 5, characterized in that: The step S41 constructs the preprocessing of the data set, which at least includes adjusting the image size and performing normalization operations.

7. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is executed by a processor to implement the deep learning-based automatic detection method for image abnormality tampering as described in any one of claims 1-6.

8. A computer device, characterized in that: include: a memory for storing instructions; A processor is used to execute the instructions so that the computer device performs the deep learning-based automatic detection method for abnormal image tampering as described in any one of claims 1 to 6.

Images