A method and system for identifying network video source devices based on convolutional neural networks

Abstract

This invention discloses a method and system for identifying network video source devices based on convolutional neural networks. The method involves segmenting the video source to be identified into keyframes, extracting the largest information block from each keyframe, filtering the largest information block to obtain a filtered image, and inputting the filtered image into a trained classification network model for classification and outputting the classification result. This invention can learn image fingerprint features directly from data without relying on manual or prior features; simultaneously, the use of constrained convolutional layers can jointly suppress image content and adaptively learn image fingerprint features. This invention can fully utilize the high-frequency noise information in the largest information block of the keyframe, improving the accuracy of video source identification.

Description

Technical Field

[0001] This invention belongs to the field of deep learning and video source forensics, and relates to a method and system for identifying network video source devices based on convolutional neural networks. Background Technology

[0002] With the rapid development of multimedia technology, the application of digital video has become increasingly widespread. For example, people can conduct business activities through live video streaming or share their lives using social video apps. However, the widespread use of digital video has also promoted the continuous development of video editing software. This software allows malicious attackers to easily edit and tamper with video information to carry out various illegal activities. Therefore, identifying the authenticity of digital videos is of significant research importance.

[0003] Digital video is essentially composed of multiple frames arranged in a fixed temporal sequence. Therefore, digital video forensics algorithms can also draw upon some methods from digital image forensics. Digital video forensics algorithms are divided into active and passive methods. Digital video source device identification algorithms belong to the passive category. These algorithms directly identify the source device based on the information inherent in the digital video itself, without requiring any preprocessing. Traditional video source device identification algorithms primarily compare imaging artifacts (such as different CFA interpolation algorithms), sensor defects (such as random noise caused by the inhomogeneity of photosensitive materials' photoresponse), and video post-processing (such as video frame compression algorithms) within the video frames to determine the source device. Among these, the most widely used method is sensor mode noise based on photoresponse inhomogeneity. This noise arises from defects in the sensor manufacturing process, leading to varying light sensitivities in different CMOS / CCD chips. This defect is unique to each device, thus sensor mode noise can be used for video source identification.

[0004] Video source device identification algorithms are based on digital image source device identification algorithms, but they are more challenging than general digital image source device identification algorithms, mainly in two aspects. Firstly, because digital video has stronger compression than digital images, primarily in intra-frame coding and inter-frame prediction, it results in greater loss of noise information. This reduces the noise information used for source identification, thus lowering the identification accuracy. Secondly, most current video source identification methods identify the center block of the video frame. These algorithms do not consider the loss of center pixel values ​​during video compression, which also leads to a decrease in identification accuracy. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for identifying network video source devices based on convolutional neural networks, so as to overcome the problem of decreased recognition accuracy caused by existing traditional algorithms that only use the center block pixels of the video.

[0006] A method for identifying network video source devices based on convolutional neural networks includes the following steps:

[0007] S1, divide the video source to be identified into frames to obtain key frames;

[0008] S2, extract the maximum information block from the obtained keyframes, and filter the obtained maximum information block to obtain the filtered image;

[0009] S3 inputs the filtered image into the trained classification network model, performs classification, and outputs the classification result.

[0010] Preferably, the video source to be identified is segmented into frames using the FFmpeg library to obtain keyframes.

[0011] Preferably, the keyframes after video framing are divided into 4x4 blocks, these blocks are then discretely segmented, and a cosine transform is performed on each block to conduct a frequency domain transformation operation. Then, a search is performed to obtain the largest information block.

[0012] The preferred formula for the cosine transform is as follows:

[0013]

[0014] In the above formula, f(x, y) represents the pixel value at position (x, y) in the image.

[0015] Preferably, after cosine transformation, if If F(u, v) = 0 and F(0, 0) ≠ 0, then set all corresponding positions of the block in the frame to 0; otherwise, set all corresponding positions to 1. After processing all 4x4 blocks, a 0 / 1 matrix M is obtained. 01 The size of this matrix is ​​the same as the size of the frame.

[0016] Preferably, an integral graph-based search algorithm is used in M. 01 The block containing the most 1s is searched. In this embodiment, the block size is fixed at 256×256, and the block with the largest sum is the largest information block.

[0017] Preferably, matrix M is calculated first. 01 The integral graph is obtained by drawing the integral graph S. 01 :

[0018]

[0019] An enumeration search algorithm is used to search from top to bottom and left to right, and the sum of each block is calculated using the integral graph formula.

[0020] Preferred, B x，yThis indicates that the coordinates (x, y) of the top left corner in matrix M 01 Block at the location:

[0021] B x，y =S 01 (x+256,y+256)-S 01 (x+256,y)-S 01 (x, y+256)+S 01 (x, y)

[0022] After obtaining the optimal (x, y) coordinates using the above formula, use the (x, y) coordinates in the Frame. i，j，k Extract a 256x256 block B from the middle max This block is the largest information block.

[0023] Preferably, for each Frame i，j，k The maximum information block extraction algorithm is used to extract the maximum information block, and high-pass filtering is performed on the maximum information block to obtain the filtered image.

[0024] A network video source device recognition system based on convolutional neural networks includes a preprocessing module and a recognition module.

[0025] The preprocessing module is used to divide the video source to be identified into frames to obtain key frames, extract the largest information block from the obtained key frames, and filter the obtained largest information block to obtain the filtered image.

[0026] The recognition module compares the filtered image with the corresponding category in the trained classification network model and outputs the classification result.

[0027] Compared with the prior art, the present invention has the following beneficial technical effects:

[0028] This invention discloses a network video source device identification method based on convolutional neural networks. The method involves segmenting the video source to be identified into keyframes, extracting the largest information block from each keyframe, filtering the largest information block to obtain a filtered image, and inputting the filtered image into a trained classification network model for classification and outputting the classification result. This invention can learn image fingerprint features directly from data without relying on manual or prior features; simultaneously, the use of constrained convolutional layers can jointly suppress image content and adaptively learn image fingerprint features. This invention can fully utilize high-frequency noise information in the largest information block of the keyframe, improving the accuracy of video source identification. Attached Figure Description

[0029] Figure 1 This illustrates the distribution of different types of frames in a video file according to an embodiment of the present invention.

[0030] Figure 2 This is a basic flowchart of the maximum information block extraction algorithm based on DCT coefficients in an embodiment of the present invention.

[0031] Figure 3 This is a schematic diagram comparing the performance of the method of the present invention with other block extraction algorithms in an embodiment of the present invention.

[0032] Figure 4 This diagram illustrates a performance comparison between the method of this invention and currently popular algorithms. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0034] A method for identifying network video source devices based on convolutional neural networks includes the following steps:

[0035] S1, divide the video source to be identified into frames to obtain key frames;

[0036] Specifically, the video source to be identified is segmented into I-frames using the FFmpeg library. I-frames do not undergo inter-frame prediction and retain the most useful information. Only I-frames are retained after segmentation.

[0037] The I-frames after video framing are divided into 4x4 blocks, and these blocks are then discretely segmented. In each block, a cosine transform (DCT) is performed to transform it in the frequency domain.

[0038] If the coefficients after DCT transformation contain only DC components, the position is marked as 0; otherwise, it is marked as 1, thus obtaining a labeled image A with the same size as the original image. Finally, a sliding window from image A is used to search for a sub-image B of a specified size containing at most 1s. This invention uses integral images to optimize its time complexity and improve computational efficiency. For grayscale images, the value of any point (,y) in the integral image refers to the sum of the grayscale values ​​of all points in the rectangular region from the top left corner of the image to this point. In this way, the computational complexity of the area within the sliding window can be optimized to O(1) each time it slides, and the time complexity of the entire search can be reduced to O(n). 2 ), Figure 2 The process of extracting the maximum information block based on DCT transform is demonstrated.

[0039] After obtaining the largest information block, to avoid interference from scene information during the training process of the convolutional neural network, this invention pre-filters the largest information block using a wavelet-based high-pass filter to eliminate all scene information, thus obtaining a training set G. n ={noise n :label n}, which contains high-frequency noisy images with n largest information blocks, each image having a corresponding category label, i.e., the source camera to which it belongs, the category label being one-hot encoded.

[0040] Finally, the preprocessed training set G is used. n The network is trained to obtain a well-trained classification network model; the input size of the convolutional neural network is 511×511×3, and each input data of the network is from G. n Random sampling was used; the network loss function used was standard cross-entropy, the optimizer was selected as ADma, the initial learning rate was set to 0.001, and the learning rate was reduced to 0.5 times the original rate every 30 epochs; the total number of iterations for network training was set to 100.

[0041] S2, extract the maximum information block from the obtained keyframes, and filter the obtained maximum information block to obtain the filtered image;

[0042] S3. Input the filtered image into the trained classification network model, classify it, and output the classification result to obtain the device used to identify the video source.

[0043] This invention determines whether two input videos originate from the same source device by comparing their intrinsic features. By fusing frame-level similarity features into video-level similarity features, the robustness of the algorithm and the accuracy of source device identification are further improved.

[0044] Example:

[0045] For 50 cameras, each camera recorded 100 videos.

[0046] The video is represented as Where i represents the i-th camera, and j represents the j-th video captured by camera i.

[0047] Firstly, regarding the video To perform frame segmentation using FFmpeg, the following frame segmentation commands are used:

[0048] ffmpeg-l video_name-Vf select=′eq(pict_type\\,l)′-vsync 2-f image2folder / video_name-%02d.png

[0049] After the above commands are executed, each video is divided into several l-frames, i.e., keyframes, using the Frame... i，j，k Let l represent the segmented frame, where i represents the i-th camera, j represents the j-th video, and k represents the k-th frame in the video.

[0050] The frame is then divided into several disjoint 4x4 blocks from top to bottom and left to right. The Discrete Cosine Transform (DCT) is then used to perform a frequency domain transformation on each 4x4 block. The formula for the DCT transformation is shown below:

[0051]

[0052] In the above formula, f(x, y) represents the pixel value at position (x, y) in the image, where N represents the length and width of the region undergoing DCT transformation. In this embodiment, the transformation region is a 4x4 square region, so N = 4. u, v represent the coordinates of any point in the transformation region. In this embodiment, u, v ∈ {0, 1, 2, 3}, and π is a constant with a value of approximately 3.1415926. After DCT transformation, if And if F(0,0)≠0, it means that the block no longer contains any useful information, so set all corresponding positions of the block in the frame to 0; otherwise, set all corresponding positions to 1. After processing all 4x4 blocks, a 0 / 1 matrix M is obtained. 01 The size of this matrix is ​​the same as the size of the frame.

[0053] Then, an integral graph-based search algorithm is used in M. 01 The block containing the most 1s is searched, and in this embodiment, the block size is fixed at 256×256; first, matrix M is calculated. 01 The integral graph is obtained by drawing the integral graph S. 01 Its definition is:

[0054]

[0055] Where p and q represent M respectively 01 In this embodiment, the coordinate positions of p, q ∈ [0, 255] and p, q ∈ N. + .

[0056] An enumeration search method is used to search from top to bottom and left to right. The sum of each block is calculated using the integral graph formula, and the block with the largest sum is the largest information block.

[0057] B x，y This indicates that the coordinates (x, y) of the top left corner in matrix M 01 Block at the location:

[0058] B x，y =S 01 (x+256,y+256)-S 01 (x+256,y)-S 01 (x, y+256)+S 01 (x, y)

[0059] After obtaining the optimal (x, y) coordinates using the above formula, use the (x, y) coordinates in the Frame. i，j，k Extract a 256x256 block B from the middle max This block is the largest information block.

[0060] For each Frame i，j，k The maximum information block extraction algorithm is used to extract the maximum information block, which is then used as the representative of the frame.

[0061] High-pass filtering is performed on the largest information block. Specifically, B is first filtered... max The DB4 wavelet is used to perform level 4 decomposition, then LL is set to 0, and Wiener filtering is performed in the remaining wavelet domains to obtain the filtered image. The classification result sequence of the filtered image is fused, and the fused information is input into the trained classification network for classification and output of the classification result.

[0062] like Figure 3 As shown, the dataset uses a subset of the VISION dataset. The figure shows the ROC curves obtained by testing three different frame combinations at three different resolutions. These three types are: (1) using the maximum information block proposed in this invention as the classification criterion; (2) using only the center block of the I-frame as the classification criterion; and (3) using both the center block of the I-frame and the center block of the P-frame as the classification criterion. It can be seen from the figure that the AUC value of each algorithm gradually increases with the increase of the block size. When the block size is 128*128, the performance of the three frame combination modes is comparable, and it is impossible to distinguish which algorithm performs best. However, as the block size increases, when it becomes 256*256, the algorithm proposed in this invention significantly outperforms the other two algorithms. At this point, the performance of the I-frame center block and the I-frame center block + P-frame center block combination modes is comparable. When the block size continues to increase to 512*512, the algorithm proposed in this invention still outperforms the other two algorithms.

[0063] like Figure 4As shown, to verify the performance of the proposed method, a portion of the data in the VISION dataset was processed to generate a new dataset for algorithm testing. All videos in this dataset are 1080P in resolution and re-encoded to MP4 format by FFMPEG, with the video bitrate consistent with the original videos. This new dataset contains 114 videos from 6 cameras, of which 60% (66 videos) were used to train the network model, and 40% (48 videos) were used to test the model's performance. As can be seen from the figure, the proposed algorithm outperforms other popular algorithms by comparing it with the classic PRNU model and a mask-based PRNU model at different resolutions (128*128, 256*256, 512*512).

[0064] This invention differs from traditional methods that use CNNs (such as AlexNet and ResNet50) for image source identification tasks. These networks more easily learn low-dimensional semantic features of images, leading to a classifier that identifies scene content relevant to the training data, rather than learning noise introduced by image content tampering. This invention learns image fingerprint features directly from the data without relying on manual or prior features; simultaneously, constrained convolutional layers jointly suppress image content and adaptively learn image fingerprint features. This invention fully utilizes high-frequency noise information in the largest information block within an I-frame, improving the accuracy of video source identification. This invention is suitable for forensic personnel.

Claims

1. A method for identifying network video source devices based on convolutional neural networks, characterized in that, Includes the following steps: S1, divide the video source to be identified into frames to obtain key frames; S2, extract the maximum information block from the obtained keyframes, and filter the obtained maximum information block to obtain the filtered image; The keyframes after video framing are divided into 4x4 blocks. These blocks are then discretized, and a cosine transform is applied to each block for frequency domain transformation. The block with the largest information is then searched for. The formula for the cosine transform is as follows: In the above formula Representing an image The pixel value of the location; After cosine transformation, if , ,and The process involves setting all corresponding positions of the block in the frame to 0, or setting all corresponding positions to 1. After processing all 4x4 blocks, a 0 / 1 matrix is ​​obtained. The size of this matrix is ​​the same as the size of the frame; Using an integral graph-based search algorithm in The search finds the block containing the most 1s, with a fixed block size. The largest and most retained block is the largest information block; S3 inputs the filtered image into the trained classification network model, performs classification, and outputs the classification result.

2. The method for identifying network video source devices based on convolutional neural networks according to claim 1, characterized in that, The video source to be identified is segmented into frames using the FFmpeg library to obtain keyframes.

3. The method for identifying network video source devices based on convolutional neural networks according to claim 1, characterized in that, First, calculate the matrix. The integral graph is obtained. : An enumeration search algorithm is used to search from top to bottom and left to right, and the sum of each block is calculated using the integral graph formula.

4. The method for identifying network video source devices based on convolutional neural networks according to claim 3, characterized in that, Represents the coordinates of the top left corner In the matrix Block at the location: The optimal solution is obtained using the above formula. After coordinates, use Coordinates at Extract a 256x256 block from the middle This block is the largest information block.

5. The method for identifying network video source devices based on convolutional neural networks according to claim 4, characterized in that, For each The maximum information block extraction algorithm is used to extract the maximum information block, and high-pass filtering is performed on the maximum information block to obtain the filtered image.

6. A network video source device recognition system based on convolutional neural networks, characterized in that, Includes a preprocessing module and a recognition module: The preprocessing module is used to divide the video source to be identified into frames to obtain key frames, extract the largest information block from the obtained key frames, and filter the obtained largest information block to obtain the filtered image. The keyframes after video framing are divided into 4x4 blocks. These blocks are then discretized, and a cosine transform is applied to each block for frequency domain transformation. The block with the largest information is then searched for. The formula for the cosine transform is as follows: In the above formula Representing an image The pixel value of the location; After cosine transformation, if , ,and The process involves setting all corresponding positions of the block in the frame to 0, or setting all corresponding positions to 1. After processing all 4x4 blocks, a 0 / 1 matrix is ​​obtained. The size of this matrix is ​​the same as the size of the frame; Using an integral graph-based search algorithm in The search finds the block containing the most 1s, with a fixed block size. The largest and most retained block is the largest information block; The recognition module compares the filtered image with the corresponding category in the trained classification network model and outputs the classification result.

Images