Digital intellectual property comparison detection method and system based on deep neural network

Abstract

The invention provides a digital intellectual property comparison detection method and system based on a deep neural network, and a human-computer interaction interface is used for achieving a digital intellectual property comparison detection type receiving step. The face similarity detection step, the image tampering detection step, the portrait type video similarity detection step and the audio type video similarity detection step are completed by a processor. The face feature extraction model is a facenet neural network model; the abnormal feature extraction model is a VGG network model. According to the invention, the video similarity analysis is carried out by respectively combining the video content style and feature with the face feature and the audio feature, the video similarity can be analyzed from multiple dimensions, and the method has the advantages of flexibility, high performance, simple structure and high practicability; meanwhile, similarity detection of images and videos is integrated into one GUI application, the integration degree is high, and the method is simpler, more convenient and more practical.

Description

Technical Field

[0001] The present invention relates to the fields of audio and video processing, machine learning and computer vision, and in particular to digital intellectual property comparison and detection technology. Background Art

[0002] Driven by the wave of digitalization, digital intellectual property protection has become an important cornerstone for maintaining the healthy development of the creative industry. With the rapid development of information technology, the forms of digital content are becoming increasingly diverse, including but not limited to high-definition video, high-resolution images, stereo audio, complex software programs, interactive e-books, and immersive games. These digital works have greatly enriched people's cultural life with their unique artistic value, technical content and commercial value, and promoted the vigorous development of related industries. However, the easy duplication and wide dissemination of digital content have also brought unprecedented challenges to its intellectual property protection. The frequent unauthorized duplication, tampering, distribution and secondary creation of digital content have not only seriously infringed the legitimate rights and interests of creators, such as copyright, trademark rights, patent rights, etc., but also disrupted the market order and hindered the sustainable development of the creative industry. In particular, multimedia content such as video, images and audio, due to their intuitiveness and high dissemination efficiency, have become the hardest hit areas for infringement.

[0003] For video similarity detection, although there have been many studies and technologies dedicated to improving the accuracy and efficiency of detection, most studies still focus on specific goals and often only target a specific aspect of video similarity assessment. Q.Yin et al. designed a dynamic fine-grained difference capture module to deeply explore the difference area between frames, and used fine-grained denoising operations to eliminate the interference caused by dynamic factors such as facial motion, thereby achieving effective detection of deep similarity of video content. Similarly, Mohammed Aloraini et al. regarded video sequences as a mixture of random processes and positive abnormal patches. By cleverly combining sequence analysis and patch analysis methods, they were able to identify abnormal changes in similarity of specific targets or objects in video frames, which is of great significance for detecting local tampering or editing in videos. However, although the above studies have achieved remarkable results in their focused fields, their scope of application is relatively limited, and it is difficult to fully cope with the diverse needs of video similarity detection. In order to broaden the coverage of detection, X.Jin et al. proposed a framework for object-based similarity detection of different types in videos. The framework aims to reveal object-based similarity changes such as splicing, VCMFD, and object removal, thereby solving the problem of single detection type to a certain extent. However, its complex implementation process has limited its promotion in practical applications to a certain extent.

[0004] The current state of research in audio similarity detection is a shift from traditional feature extraction methods to deep learning techniques. Although traditional methods such as cepstral transform and wavelet transform are widely used, they are difficult to represent deep features and are not suitable for massive samples. Although artificial features such as Mel-frequency cepstral coefficients (MFCC) perform well in speech recognition, their limitations lie in their reliance on large amounts of data and the inefficiency of manual annotation. Although some studies have combined MFCC and CNN audio similarity algorithms, there are still problems with insufficient model depth, inaccurate feature extraction, and the need to improve performance and efficiency.

[0005] Similar problems also exist in the field of image tampering detection. Many studies and technologies on image tampering detection and positioning only focus on a certain type of tampering technology, rather than being applicable to most types of tampering. For example, Salloum et al. proposed an image splicing positioning algorithm based on a fully convolutional network, but this method only focuses on the image splicing positioning problem, and has poor results for other types of tampering, such as Copy-Move.

[0006] Therefore, it is particularly important to develop a digital intellectual property protection technology with high performance, simple structure and wide application range. Summary of the invention

[0007] The technical problem to be solved by the present invention is to realize a method and system for comparison and detection of digital content by introducing deep learning technology, especially using neural networks to perform global and local similarity comparison analysis on digital content such as videos, images, and audio.

[0008] The technical means adopted by the present invention to solve the above technical problems is a digital intellectual property comparison and detection method based on a deep neural network, comprising:

[0009] Digital intellectual property comparison detection type receiving step: when a face similarity detection instruction is received, the face similarity detection step is triggered; when an image tampering detection instruction is received, the image tampering detection step is triggered; when a portrait video similarity detection instruction is received, the portrait video similarity detection step is triggered; when an audio video similarity detection instruction is received, the audio video similarity detection step is triggered;

[0010] Face similarity detection step: receiving an input face image group to be compared, locating and cropping the face area in the image of the face image group to be compared by using face detection and cropping technology, extracting a face feature vector group corresponding to two face areas in the face image group to be compared by using a pre-trained face feature extraction model, obtaining the face similarity of the two face feature vectors in the face feature vector group, comparing the face similarity with the similarity threshold and outputting the similarity comparison result;

[0011] Image tampering detection step: receiving the input image to be detected, preprocessing the image to be detected, using the pre-trained abnormal feature extraction model to extract abnormal features in the image to be detected, and then analyzing and processing the abnormal features to obtain the abnormal score of each pixel in the image to be detected, and judging the pixel with an abnormal score greater than the preset abnormal value as a tampered pixel to complete the positioning of the tampered point;

[0012] The steps of similarity detection of portrait videos are as follows: receiving two input video files to be compared, extracting video frames from the video files to be compared respectively, and then, on the one hand, extracting key frames containing the faces of the characters from the video frames of the two video files respectively to form a face image group to be compared, and locating and cropping the face area in the picture of the face image group to be compared using face detection and cropping technology in the key frames containing the faces of the characters, extracting face feature vectors using a pre-trained face feature extraction model, and removing abnormal data using an isolation forest algorithm to obtain the final face feature vectors of the two video files to be compared; on the other hand, extracting video content feature vectors from the video frames of the two video files respectively using a video feature extraction model; using cosine similarity to measure the face feature vectors of the two video files to be compared to obtain a face cosine similarity matrix, using cosine similarity to measure the video content feature vectors of the two video files to be compared to obtain a video content cosine similarity matrix, and combining the face cosine similarity matrix and the video content cosine similarity matrix to output the final video similarity;

[0013] Audio-video similarity detection steps: receive two input video files to be compared, extract video frames from each of the video files to be compared, and then, on the one hand, use the video feature extraction model to extract video content feature vectors from the video frames of the two video files respectively; on the other hand, extract the Mel-frequency cepstral coefficients MFCC in the audio information in the video frames as audio features; use the Euclidean distance to measure the video content feature vectors of the two video files to be compared to obtain image similarity, use the cosine similarity to measure the audio features of the two video files to be compared to obtain audio similarity, and combine the image similarity and audio similarity to output the final video similarity.

[0014] At the same time, a system for implementing the method is provided, including a human-computer interaction interface and a processor, wherein the human-computer interaction interface is used to implement the digital intellectual property comparison detection type receiving step, and other detection steps are completed by the processor. The human-computer interaction interface is specifically a multi-page graphical interface GUI.

[0015] Specifically, the face feature extraction model is the FaceNet neural network model; the abnormal feature extraction model is the VGG network model; and the video feature extraction model is the ResNet-18 neural network model.

[0016] Specifically, for face similarity detection, the pre-trained facenet neural network model is used to extract high-dimensional face feature vectors, and the Euclidean distance between face feature vectors is calculated, and the similarity comparison result is output by comparing with the trained threshold. Two functions are provided in the graphical user interface GUI application: face similarity detection of two images and face similarity detection of two groups of images.

[0017] Specifically, for the image tampering detection module, Mantra-Net is used to identify local abnormal features to detect forged pixels and locate the tampered areas.

[0018] Specifically, for similarity comparison of portrait videos, the FaceNet neural network model is used to extract high-dimensional facial feature vectors, and the isolation forest algorithm is used to remove abnormal data; the ResNet-18 neural network model is used to extract high-dimensional video content feature vectors from video frames. Then, cosine similarity is used to measure the similarity between video content and facial features, respectively, to form a cosine similarity matrix. Through comprehensive analysis of the similarity, the video similarity score is finally output to determine whether the video is plagiarized or tampered with.

[0019] Specifically, for audio-video similarity comparison, the ResNet-18 neural network is used to extract the feature vector of the video frame and calculate the Euclidean distance. MFCC is used to extract the audio features in the video and the cosine distance of the feature vector is calculated to obtain the video similarity and audio similarity respectively. Finally, each is multiplied by a weight factor to obtain the joint similarity, and the similarity and similar position of the video clips are judged based on this.

[0020] For the multi-interface GUI, the present invention provides five interfaces for face similarity detection of a set of image pairs, face similarity detection between multiple images, image tampering detection, portrait video similarity detection and audio video similarity detection.

[0021] The beneficial effect of the present invention is that the video content style and features are combined with facial features and audio features to perform video similarity analysis, and video similarity can be analyzed from multiple dimensions. It has the advantages of flexibility, high performance, simple structure and strong practicality. At the same time, the present invention integrates the similarity detection of images and videos into a GUI application, which has a high degree of integration and is more simple and practical. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is a schematic diagram of the overall system architecture;

[0023] Figure 2 This is a schematic diagram of face similarity detection;

[0024] Figure 3 Schematic diagram of image tampering detection;

[0025] Figure 4 This is a schematic diagram for comparing video similarity and face similarity;

[0026] Figure 5 This is a schematic diagram for comparing video similarity and sound similarity;

[0027] Figure 6 This is a schematic diagram of the GUI interface, where: Figure 6-1 Face image pair detection interface; Figure 6-2 It is the face file pair detection interface; Figure 6-3 It is an image tampering detection interface; Figure 6-4 It is the portrait video detection interface; Figure 6-5 It is the audio video detection interface. DETAILED DESCRIPTION

[0028] The present invention relates to a digital intellectual property comparison detection method in the aspects of face similarity detection, image tampering detection, video similarity detection and audio similarity comparison. The processing efficiency is improved by combining optimization technologies such as multithreading to meet the needs of fast response and large-scale data processing in practical applications. The implementation of the present invention will provide new solutions and technical support for the effective protection of digital intellectual property.

[0029] Example 1

[0030] Digital intellectual property anti-counterfeiting detection includes four functions: face similarity detection, image tampering detection, portrait video similarity detection, and audio video similarity detection. Among them, face similarity detection includes two sub-functions: face image pair detection and face file pair detection. Face image pair detection is used for face similarity comparison of a group of image pairs. Face file pair detection is used for face similarity comparison of multiple groups of image pairs.

[0031] 1. Face Similarity Detection

[0032] 1. Method for comparing the similarity of faces between two images:

[0033] 1) Obtain a set of original image pairs to be compared and preprocess them, including face recognition and cropping of the original images to be compared, and converting them into a size and format suitable for model input. The face image set to be compared is two images containing suspected identical faces. Face detection and cropping technology is used to locate and crop the face area in the image to improve the accuracy of face feature extraction.

[0034] Specifically, when performing face detection and cropping, the dlib library in pytorch is used to perform face recognition on the original image, and the target face image is cropped after scaling by boundary parameters; the cropped face image is converted to a size and format suitable for model input.

[0035] 2) Use the pre-trained facenet deep neural network model to extract the feature vectors of the cropped face areas.

[0036] Specifically, an open source facenet neural network model pre-trained with the ImageNet dataset was used. The backbone feature extraction network was InceptionResnetV1, which consisted of 10 explicit convolutional layers, 5 maximum pooling layers, a residual module, and a ReLU activation function. An adaptive average pooling layer and a fully connected layer were used. The dropout was set to 0.5, and the loss function used the cross entropy loss function. A cropped face image was used as the input of the neural network model to obtain a 128-dimensional face feature vector.

[0037] 3) Evaluate the facial similarity of the two images to be compared based on the obtained facial feature vectors, and then obtain the comparison result by comparing with the similarity threshold. If the Euclidean distance of the facial feature vectors of the image group is less than the facial similarity threshold, the output facial similarity comparison result is similar faces, otherwise the output comparison result is dissimilar faces.

[0038] Specifically, the face similarity in this process is reflected by calculating the Euclidean distance between the high-dimensional feature vectors of the two face regions. The similarity threshold is obtained by training the commonly used LFW dataset.

[0039] Specifically, let any two face feature vectors be A = {a1, a2, …, a 128}, B={b1,b2,…,b 128}, the calculation formula of vector Euclidean distance is as follows:

[0040]

[0041] Among them, a i and b i are the i-th components of A and B respectively.

[0042] 2. Face similarity comparison method between two sets of images:

[0043] 1) Obtain two files containing the original face similarity images to be compared. Each file (i.e., each image group) can contain multiple face images to be compared. Preprocess all original images, including face recognition and cropping of the original images to be compared, and convert them into a size and format suitable for model input. The face images to be compared are composed of two groups, each with multiple face images to be compared. Then, facial detection and cropping techniques are used to locate and crop the face area in the image to improve the accuracy of facial feature extraction.

[0044] Specifically, when performing face detection and cropping, the dlib library in pytorch is used to perform face recognition on the original image, and the target face image is cropped after scaling by boundary parameters; the cropped face image is converted to a size and format suitable for model input to form two face image groups to be compared.

[0045] 2) Use the pre-trained facenet deep neural network model to extract the feature vector groups of the cropped face areas in the two image groups respectively.

[0046] Specifically, the facenet deep neural network model used here is the same as the face similarity detection part of a set of image pairs; the face image group to be compared is used as the input of the model neural network model to obtain two face feature vector groups, which are respectively composed of m 128-dimensional face feature vectors and n 128-dimensional face feature vectors, where m and n are the number of face images contained in each face image group to be compared.

[0047] 3) Evaluate the facial similarity of the two images to be compared based on the obtained facial feature vector group, and then compare with the threshold to get the result. This process calculates the Euclidean distance between each pair of facial feature vectors in the two image groups, obtains the Euclidean distance matrix, and then compares it with the facial similarity threshold obtained by training the LFW dataset to obtain the facial similarity comparison result of the facial image group to be compared. If the Euclidean distance matrix is ​​greater than the facial similarity threshold, it is judged as dissimilar, and if it is less than the threshold, it is judged as similar, and the result is represented by a heat map. Each square in the heat map represents the facial similarity detection result of the corresponding two horizontal and vertical image names.

[0048] Specifically, in the heat map, the squares with similar results are filled with yellow, and the rest of the dissimilar ones are filled with purple, making it easier to visually see the comparison results.

[0049] 2. Image tampering detection:

[0050] 1) Obtain the image to be detected, preprocess the original image, and convert the original image to be detected into a size and format suitable for model input.

[0051] 2) Use the pre-trained VGG network model as the abnormal feature extraction module. The VGG network is very suitable for feature extraction with its deep and multi-layer convolutional structure. Its deep network can capture subtle features in the image. The test image is passed through the VGG network model to extract abnormal features from the test image.

[0052] Specifically, the VGG deep learning network here consists of a 5*5 convolutional layer Conv, a constrained convolutional layer BayerConv, an SRMConv layer, and twelve 3*3 convolutional layers Conv. The SRMConv layer refers to a style-based convolutional neural network recalibration module, which is used to carry the convolution of the style-based recalibration module, which dynamically evaluates the importance of each style, allowing the convolutional network to focus more on the meaningful parts.

[0053] 3) The abnormal features are passed through the Zpool of windows of different sizes to obtain the scores Zscore of different window sizes. The Zscores calculated by windows of different sizes are synthesized into a sequence to generate a four-dimensional feature, and the four-dimensional feature is analyzed using the ConvLSTM2D layer to finally obtain the abnormal score of each pixel. The abnormal score is compared with the preset value to obtain the tampering detection and positioning result.

[0054] Specifically, windows of sizes 7*7, 15*15, and 31*31 are used respectively, and the Zscore of each pixel is calculated according to the following formula:

[0055]

[0056] Among them, n×n is the measurement window size, n is the measurement window side length, F represents the image to be tested, H and W are the height and width of the image to be tested, i is the row number of the pixel, j is the column number of the pixel, and F[i,j] is the abnormal feature value of the pixel point [i,j]. is the average value of the abnormal characteristic value in the n×n area, are all intermediate values, σ F is the variance of the abnormal feature value. The Zscores calculated by windows of different sizes are synthesized into a sequence to generate a four-dimensional feature of size (n+1)×H×W×L, where L is the number of channels. Using the ConvLSTM2D layer, the Zscores of different window sizes are analyzed in sequence to finally obtain the abnormal score of each pixel. The four-dimensional feature here is combined by the three-dimensional features of different window sizes, so analyzing it is essentially analyzing the Zscores of different window sizes. Pixels with a score greater than 0.2 are considered to be tampered pixels, so tampering is located.

[0057] 3. Similarity comparison method for portrait videos:

[0058] 1) Get the original video data set to be compared, create two independent threads, and extract frames from the videos to be compared respectively: the videos can include different types of content such as film and television works, short videos, etc., and each two videos to be compared are considered as a group. Through the video frame extraction technology, representative frames are extracted from the video at a fixed frequency.

[0059] Specifically, representative frames are extracted from the video at a frequency of 1 frame per 10 frames.

[0060] 2) Preprocess the extracted representative frames in two threads, including converting them into a format suitable for model input, and using multi-threading technology to improve frame processing efficiency. By creating a thread pool, face detection of multiple video frames is processed in parallel. In the preprocessed frames, key frames containing the person's face are extracted.

[0061] 3) When extracting the key frames containing the human face, the face detection and cropping techniques are used to locate and crop the face area in the video in two threads respectively.

[0062] Specifically, the dlib library in pytorch is used to perform face recognition on the extracted video frames. x1, y1, x2, y2 represent the left, upper, right, and lower boundaries of the recognized face rectangle, respectively. Margin represents the scaling factor of the cropped face rectangle, which is used to extract more comprehensive face information. The specific face cropping process is as follows:

[0063] dx=margin×(x2-x1)

[0064] dy=margin×(y2-y1)

[0065] d1=x1-dx

[0066] p1=y1-dy

[0067] d2=x2+dx

[0068] p2=y2+dy

[0069] Where dx and dy represent the number of pixels to be expanded in the left and right directions and the top and bottom directions respectively, and d1, p1, d2, and p2 represent the left boundary, top boundary, right boundary, and bottom boundary of the expanded face rectangle box respectively.

[0070] 4) The pre-trained FaceNet deep neural network model is used to extract facial feature vectors from the face regions cropped from the video frames in the two threads. At the same time, feature forest is used to remove abnormal features of a specified proportion to ensure the accuracy and consistency of the feature data input to the model.

[0071] Specifically, a pre-trained facenet deep neural network model is used to extract facial feature vectors from the face regions cropped from the video frames in the two threads respectively; the facenet deep neural network model used here is the same as the face similarity detection part; the cropped face image is used as the input of the neural network model to obtain a 128-dimensional facial feature vector;

[0072] Then, the isolation forest algorithm is used to detect and filter abnormal features with a ratio of 0.2, remove faces with low frequency and abnormal detection, and form a face feature vector group; let the formed face feature vector group be X = {x1, x2, ... x i ,…x n}, where each eigenvector x i ∈R 128 is a 128-dimensional face feature vector; the following scoring function s is used to evaluate each feature vector x i To rate:

[0073]

[0074] Where E(h(x i )) is x i The average split depth in the isolated tree, h(x i ) represents the data point x i The depth in an isolated tree, E(h(x i )) is h(x i ) is the mathematical expectation; c(n) is the adjustment term for the dataset size n, s(x i ,n) is the point x i The abnormality score of x is obtained by judging whether the point is abnormal according to the following prediction function: i Corresponding status value

[0075]

[0076] Finally, the feature vector group X′ of the normal data point is returned:

[0077]

[0078] 5) Using the pre-trained resnet-18 model, extract the video frame feature vectors from the video frames extracted from the videos in the two threads, end the multi-threading, and obtain the face feature vector group and video feature vector group of the two videos respectively.

[0079] Specifically, the resnet-18 pre-trained model in PyTorch is used, and the output of the average pooling layer avgpool is used as the feature vector of the video frame to form a video feature vector group.

[0080] 6) Use cosine similarity to calculate the similarity between the feature vectors of two video frames and between the feature vectors of faces.

[0081] Specifically, based on the similarity calculation method of video content and facial features, the facial similarity and video content similarity of two videos are evaluated; the cosine similarity between the two vectors in the facial feature vector group and the video feature vector group is calculated respectively to form a cosine similarity matrix. Assume that the two feature vectors in any vector group are A = {a1, a2, ..., a i ,…,a j ,…a m},B={b1,b2,…,b i ,…,b j ,…b m}, the calculation formula of cosine similarity is as follows:

[0082]

[0083] Where m is the dimension of the two feature vectors and || || represents the L2 norm.

[0084] 7) Combine the face similarity and video frame similarity to comprehensively output the video similarity.

[0085] Specifically, the maximum value of each row and the maximum value of each column of the cosine similarity matrix are taken as the average value to obtain the similarity between the video content and the facial features. The video content and facial feature similarities are averaged to obtain the video similarity score.

[0086] 8) The video similarity is then compared with a similarity threshold obtained through training of the training data set or a threshold set manually to obtain a comparison result.

[0087] Specifically, the threshold value is manually set at 0.8. Videos with a video similarity score greater than 0.8 are judged as plagiarized works, and videos with a similarity score less than 0.8 are judged as not plagiarized.

[0088] 4. Audio and video similarity comparison method:

[0089] 1) Read two video file paths to be compared, where the video file paths to be compared contain at least one video to be compared and an original video. Extract video frames at regular intervals and obtain audio information of the video at the same time.

[0090] Specifically, the OpenCV library is used to read two video files to be compared. The video to be compared corresponds to the suspected pirated video, and the original video corresponds to the original video.

[0091] Specifically, representative frames are extracted from the video to be compared at fixed time intervals. By setting a sliding window in the video, the entire video content is covered, and key frames containing important information are extracted. At the same time, a wav file is extracted from the video to obtain the audio information of the video to be compared. The sliding window operation is also performed to extract frames, and then the Librosa library is used to extract the Mel-frequency cepstral coefficients (MFCC) of the audio.

[0092] 2) Use the pre-trained ResNet-18 model to extract image features from the extracted video frames. At the same time, obtain the audio features of the video.

[0093] Specifically, a pre-trained ResNet-18 model is used to extract features from each video clip and output the feature vector of the video clip. At the same time, the MFCC features of the audio clip are calculated as audio features.

[0094] 3) The similarity of the image and audio features between the video to be compared and the original video is calculated using the Euclidean distance and cosine similarity algorithms respectively. Then, the total similarity of the two videos is obtained by combining the similarities of the audio and video in a weighted manner.

[0095] Specifically, Image Similarity uses the Euclidean Distance formula:

[0096]

[0097] Here, A and B are the feature vectors of the image, and n is the dimension of the feature. It is converted to the similarity percentage by the following formula:

[0098]

[0099] Audio Similarity is calculated using cosine similarity:

[0100]

[0101] Then, the similarity of the audio and video is combined in a weighted manner to obtain the joint similarity of the two videos. The formula for calculating the joint similarity is as follows:

[0102] Total Similarity

[0103] =Image Similarity*ω image +Audio Similarity*ω audio

[0104] Among them, ωimage and ω audio are the weights of image and audio similarity respectively.

[0105] 4) Based on the joint similarity score, determine whether there are clips similar to the original video in the suspected pirated video; set a similarity threshold, and the clips exceeding the threshold are considered pirated clips.

[0106] Example 2

[0107] A system for implementing the method of Example 1 is provided, and each function is called using a graphical user interface GUI, and the user interface layout adopts a modular design. The GUI includes five modules: face image pair detection, face file pair detection, image tampering detection, portrait video detection, and audio video detection. Figure 6-1 arrive Figure 6-5 As shown on the left side of the page. The interface icons and buttons are concise and clear, ensuring that users can quickly identify and accurately operate each function. Two auxiliary tool buttons, "Threshold" and "Error Prompt", are also set on the left side of the page. The "Threshold" button allows users to customize the similarity judgment threshold in the face image pair detection and face file pair detection functions, and supports manual adjustment or default value; the "Error Prompt" button is used to display the user's error operation information to assist users in correcting operations.

[0108] Face image detection: Figure 6-1 As shown in the figure, two image import buttons are set on the right side of the interface, which are used to upload the images 1 and 2 to be compared. After successful upload, the images are automatically displayed in the corresponding TextLabel area. The "Similarity Calculation" button is set at the bottom right. Click it to start the detection, and the results are displayed in the "Calculation Results" area.

[0109] Face document pair detection: Figure 6-2 As shown, it is similar to the image pair detection, but the two image import buttons on this interface import files containing multiple images, and the two sets of images of the two files to be compared are file 1 and file 2. After successful upload, the file path is displayed in the corresponding area. Click the "Similarity Calculation" button to perform pairwise similarity detection of the images in the file, and the results are displayed in the "Calculation Results" area in the form of a graphical matrix.

[0110] Image tampering detection: Figure 6-3 As shown in the figure, the interface provides an "Import image to be detected" button. After uploading the image, the image is automatically displayed in the corresponding TextLabel area. Then click the "Image Tampering Detection" button to perform the detection, and the detection result is displayed in the "Detection Result" area.

[0111] Portrait video detection: Figure 6-4As shown in the figure, two "Import Video" buttons are used to upload Video 1 and Video 2, and the corresponding area displays the video path. After the video is loaded, click the "Start Detection" button to start the portrait video similarity detection. At the same time, the progress bar below displays the detection progress in real time. The detection results include similarity values ​​and judgment results, which are displayed in the "Detection Results" box.

[0112] Audio video detection: such as Figure 6-5 As shown, two import buttons are used to upload the video file to be detected and the comparison video file to complete "Select the video to be detected" and "Select the comparison video". At the same time, click "Select the location to save the results" to select the path to save the Excel file of the detection results. After the file is loaded, click the "Start Detection" button to start the audio video similarity detection. The detection results include the names of the two videos involved in the detection, the similarity judgment results and the maximum similarity period, which are displayed in the "Detection Results" box. The Excel file also contains the specific values ​​of the video and audio similarity and the joint similarity in different video clips.

[0113] In terms of video similarity comparison detection, the present invention takes into account the diversity of video styles and adopts the detection modes of "combining portraits with videos" and "combining audio with videos" to improve the accuracy of similarity detection. When the user selects a video type with multiple portrait styles, the portrait video detection mode can be selected; when the selected video type is less portrait-based and mainly voice and scenery, the audio video detection mode can be selected. This design can not only better adapt to the diversity of video styles and improve the accuracy of detection, but also provide a more flexible, convenient and personalized user experience. At the same time, the present invention integrates multi-dimensional similarity analysis of faces, images, videos and sounds, and provides a comprehensive similarity detection tool. Compared with the prior art, this system not only improves the detection accuracy, but also optimizes the user operation experience through a graphical user interface (GUI). The GUI application integrates multiple functions such as picture and video similarity detection and tampering detection, which not only effectively improves the efficiency and accuracy of anti-counterfeiting detection, but also fully meets the diverse comparison and detection needs of users, so that both professionals and ordinary users can use it easily. The present invention is particularly suitable for fields such as copyright protection and content verification, and provides a high-precision, high-efficiency, automated and widely applicable digital intellectual property protection solution.

Claims

1. A digital intellectual property rights comparison and detection method based on deep neural network, characterized in that: Includes steps: Digital intellectual property comparison detection type receiving step: when a face similarity detection instruction is received, the face similarity detection step is triggered; when an image tampering detection instruction is received, the image tampering detection step is triggered; when a portrait video similarity detection instruction is received, the portrait video similarity detection step is triggered; when an audio video similarity detection instruction is received, the audio video similarity detection step is triggered; Face similarity detection step: receiving an input face image group to be compared, locating and cropping the face area in the image of the face image group to be compared by using face detection and cropping technology, extracting a face feature vector group corresponding to two face areas in the face image group to be compared by using a pre-trained face feature extraction model, obtaining the face similarity of the two face feature vectors in the face feature vector group, comparing the face similarity with the similarity threshold and outputting the similarity comparison result; Image tampering detection step: receiving the input image to be detected, preprocessing the image to be detected, using the pre-trained abnormal feature extraction model to extract abnormal features in the image to be detected, and then analyzing and processing the abnormal features to obtain the abnormal score of each pixel in the image to be detected, and judging the pixel with an abnormal score greater than the preset abnormal value as a tampered pixel to complete the positioning of the tampered point; The steps of similarity detection of portrait videos are as follows: receiving two input video files to be compared, extracting video frames from the video files to be compared respectively, and then, on the one hand, extracting key frames containing human faces from the video frames of the two video files respectively to form a human face image group to be compared, and using face detection and cropping technology to locate and crop the human face area in the picture of the human face image group to be compared in the key frames containing human faces, extracting human face feature vectors using a pre-trained human face feature extraction model, and removing abnormal data using an isolation forest algorithm to obtain the final human face feature vector group of the two video files to be compared; on the other hand, extracting video content feature vector groups from the video frames of the two video files respectively using a video feature extraction model; using cosine similarity to measure the human face feature vector groups of the two video files to be compared to obtain a human face cosine similarity matrix, using cosine similarity to measure the video content feature vector groups of the two video files to be compared to obtain a video content cosine similarity matrix, and combining the human face cosine similarity matrix and the video content cosine similarity matrix to output the final video similarity; Audio-video similarity detection steps: receiving two input video files to be compared, extracting video frames from the video files to be compared, and then, on the one hand, extracting video content feature vectors from the video frames of the two video files respectively using a video feature extraction model; on the other hand, extracting Mel frequency cepstral coefficients MFCC from the audio information in the video frames as audio features; The Euclidean distance is used to measure the video content feature vector group of the two video files to be compared to obtain the image similarity, and the cosine similarity is used to measure the audio features of the two video files to be compared to obtain the audio similarity. The image similarity and audio similarity are combined to output the final video similarity.

2. The method according to claim 1, characterized in that: The face feature extraction model is the FaceNet neural network model; the abnormal feature extraction model is the VGG network model; and the video feature extraction model is the ResNet-18 neural network model.

3. The method according to claim 1, characterized in that: In the face similarity detection step, when performing face similarity detection on two images, two input images are received as a face image group to be compared; when performing face similarity detection between two groups of images, two input files are received, one file contains more than two face images to be compared as a group, and the two groups of images are used as face image groups to be compared.

4. The method according to claim 1, characterized in that: In the face similarity detection step and the portrait video similarity detection step, the face similarity of two face feature vectors in the face feature vector group is measured by Euclidean distance.

5. The method according to claim 1, characterized in that: The specific method of analyzing and processing the abnormal features to obtain the abnormal score of each pixel in the image to be tested is: the abnormal features are passed through detection windows of different sizes to obtain the corresponding scores, and the scores corresponding to all detection windows are synthesized into a sequence and then input into ConvLSTM2D for analysis and processing to obtain the abnormal score of each pixel in the image to be tested; The score corresponding to the detection window is calculated as follows: Among them, n×n is the measurement window size, n is the measurement window side length, F represents the image to be tested, H and W are the height and width of the image to be tested, i is the row number of the pixel, j is the column number of the pixel, and F[i,j] is the abnormal feature value of the pixel point [i,j]. is the average value of the abnormal characteristic value in the n×n area, are all intermediate values, σ F is the variance of the abnormal eigenvalues.

6. The method according to claim 1, characterized in that: The specific method of combining the face cosine similarity matrix and the video content cosine similarity matrix to output the final video similarity is: The average of the maximum value of each row and the maximum value of each column in the face cosine similarity matrix is ​​taken as the face similarity, and the average of the maximum value of each row and the maximum value of each column in the video content cosine similarity matrix is ​​taken as the video content similarity; then the average of the face similarity and the video content similarity is calculated to obtain the final video similarity.

7. The method according to claim 1, characterized in that: The specific method of combining image similarity and audio similarity to output the final video similarity is: The video similarity and audio similarity are multiplied by the corresponding weight factors and then added together to obtain the final video similarity.

8. In the method as described in claim 1, in the portrait video similarity detection step and the audio video similarity detection step, if the final video similarity is greater than the preset similarity threshold, it is determined that there is an infringement of digital intellectual property rights.

9. A digital intellectual property comparison and detection system based on deep neural network, characterized in that: Including human-computer interaction interface and processor; The human-computer interaction interface is used to trigger the processor to perform the face similarity detection step and display the face similarity detection result when receiving the face similarity detection instruction; When an image tampering detection instruction is received, the image tampering detection step of the processor is triggered, and the image tampering detection result is displayed; when a portrait video similarity detection instruction is received, the portrait video similarity detection step of the processor is triggered and the portrait video similarity detection result is displayed; when an audio video similarity detection instruction is received, the audio video similarity detection step of the processor is triggered, and the audio video similarity detection result is displayed; The processor is used to: When performing the face similarity detection step: receiving an input face image group to be compared, locating and cropping the face area in the image of the face image group to be compared by using face detection and cropping technology, extracting a face feature vector group corresponding to two face areas in the face image group to be compared by using a pre-trained face feature extraction model, obtaining the face similarity of two face feature vectors in the face feature vector group, comparing the face similarity with the similarity threshold, and outputting the similarity comparison result; Perform image tampering detection steps: receive the input image to be detected, pre-process the image to be detected, use the pre-trained abnormal feature extraction model to extract abnormal features in the image to be detected, and then analyze and process the abnormal features to obtain the abnormal score of each pixel in the image to be detected and output it. The pixel with an abnormal score greater than the preset abnormal value is judged as a tampered pixel, and the tampering point is located; The steps of detecting similarity of portrait videos are as follows: receiving two input video files to be compared, extracting video frames from the video files to be compared respectively, and then, on the one hand, extracting key frames containing human faces from the video frames of the two video files respectively to form a human face image group to be compared, and locating and cropping the human face area in the picture of the human face image group to be compared by using face detection and cropping technology in the key frames containing human faces, extracting human face feature vectors by using a pre-trained human face feature extraction model, and obtaining the final human face feature vector group of the two video files to be compared after eliminating abnormal data by using an isolation forest algorithm; on the other hand, extracting video content feature vector groups from the video frames of the two video files respectively by using a video feature extraction model; using cosine similarity to measure the human face feature vector groups of the two video files to be compared to obtain a human face cosine similarity matrix, using cosine similarity to measure the video content feature vector groups of the two video files to be compared to obtain a video content cosine similarity matrix, and combining the human face cosine similarity matrix and the video content cosine similarity matrix to output the final video similarity; Performing audio-video similarity detection steps: receiving two input video files to be compared, extracting video frames from the video files to be compared, and then, on the one hand, extracting video content feature vectors from the video frames of the two video files respectively using a video feature extraction model; on the other hand, extracting Mel frequency cepstral coefficients MFCC from the audio information in the video frames as audio features; The Euclidean distance is used to measure the video content feature vector group of the two video files to be compared to obtain the image similarity, and the cosine similarity is used to measure the audio features of the two video files to be compared to obtain the audio similarity. The image similarity and audio similarity are combined to output the final video similarity.

Images