Digital intellectual property comparison detection method and system based on deep neural network

Abstract

The application provides a kind of digital intellectual property comparison detection method and system based on deep neural network, human-computer interaction interface is used to realize digital intellectual property comparison detection type receiving step, face similarity detection step, image tampering detection step, portrait video similarity detection step and audio video similarity detection step are completed using processor.Face feature extraction model is facenet neural network model;Abnormal feature extraction model is VGG network model.The video content style and feature of the application are combined with face feature and audio feature respectively to analyze video similarity, which can analyze video similarity from multiple dimensions, has the advantages of flexibility, high performance, simple structure and strong practicability;At the same time, the similarity detection of image and video is integrated into a GUI application, with high integration degree, more convenient and practical.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of audio-video processing, machine learning and computer vision, and in particular to a digital intellectual property comparison detection technology. BACKGROUND

[0002] Under the impetus of digitalization, digital intellectual property protection has become an important cornerstone for maintaining the healthy development of the creative industry. With the rapid development of information technology, the forms of digital content are becoming increasingly diverse, including but not limited to high-definition videos, high-resolution images, stereo audio, complex software programs, interactive e-books, and immersive games, etc. These digital works greatly enrich people's cultural life and promote the vigorous development of related industries with their unique artistic value, technical content and commercial value. However, the ease of copying and widespread dissemination of digital content also poses unprecedented challenges to its intellectual property protection. Unauthorized copying, tampering, distribution and secondary creation of digital content occur frequently, which not only seriously infringes the legitimate rights and interests of creators such as copyright, trademark rights and patent rights, but also disrupts the market order and hinders the sustainable development of the creative industry. In particular, multimedia content such as videos, images and audio, due to their intuitive nature and high transmission efficiency, has become a hot spot for infringement.

[0003] For video similarity detection, although there have been numerous researches and technologies dedicated to improving the accuracy and efficiency of detection, most of the researches still focus on specific targets and often only address a specific aspect of video similarity evaluation. Q. Yin et al. designed a dynamic fine-grained difference capture module to deeply mine the inter-frame difference area, and used fine-grained denoising operation to eliminate the interference caused by dynamic factors such as facial movement, thereby achieving effective detection of deep similarity of video content. Similarly, Mohammed Aloraini et al. treated video sequences as a mixture of random processes and positive abnormal patches, and by skillfully combining sequence analysis and patch analysis, they were able to identify abnormal changes in the similarity of specific targets or objects in video frames, which was of great significance for detecting local tampering or editing in videos. However, although the above researches have made remarkable achievements in their focused areas, their scope of application is relatively limited, and it is difficult to fully meet the diverse needs of video similarity detection. In order to broaden the coverage of detection, X. Jin et al. proposed a framework for object-based similarity detection in videos of different types. This framework aims to reveal object-based similarity changes such as splicing, VCMFD and target removal, thereby to some extent solving the problem of single detection type. However, the complex implementation process limits its promotion in practical applications to some extent.

[0004] The current state of research in audio similarity detection is characterized by a shift from traditional feature extraction methods to deep learning techniques. While traditional methods such as cepstral transform and wavelet transform are widely used, they struggle to represent deep features and are unsuitable for massive datasets. Although artificial features such as Mel-frequency cepstral coefficients (MFCCs) perform well in speech recognition, their limitations lie in their reliance on large datasets and the inefficiency of manual annotation. Some studies, while combining MFCC and CNN-based audio similarity algorithms, still suffer from insufficient model depth, imprecise feature extraction, and a need for further performance and efficiency improvements.

[0005] Similar problems exist in the field of image tampering detection. Many studies and techniques for image tampering detection and localization focus only on a specific type of tampering technique, rather than being applicable to most types of tampering. For example, Salloum et al. proposed an image stitching localization algorithm based on fully convolutional networks, but this method only focuses on the image stitching localization problem; it is ineffective for other types of tampering, such as copy-move.

[0006] Therefore, it is particularly important to develop a high-performance, simple-structured, and widely applicable digital intellectual property protection technology. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to realize a method and system for comparison and detection of digital content by introducing deep learning technology, especially by using neural networks to perform global and local similarity comparison analysis on digital content such as videos, images, and audio.

[0008] The technical means adopted by this invention to solve the above-mentioned technical problems is a digital intellectual property comparison and detection method based on deep neural networks, comprising:

[0009] Digital intellectual property comparison and detection type receiving steps: When a face similarity detection instruction is received, the face similarity detection step is triggered; when an image tampering detection instruction is received, the image tampering detection step is triggered; when a portrait video similarity detection instruction is received, the portrait video similarity detection step is triggered; when an audio video similarity detection instruction is received, the audio video similarity detection step is triggered.

[0010] Face similarity detection steps: Receive the input group of face images to be compared, use face detection and cropping technology to locate and crop the face regions in the images of the group of face images to be compared, use a pre-trained face feature extraction model to extract face feature vector groups corresponding to two face regions in the group of face images to be compared, obtain the face similarity between the two face feature vector groups, compare the face similarity with the similarity threshold, and output the similarity comparison result;

[0011] Image tampering detection steps: Receive the input image to be detected, preprocess the image, use a pre-trained abnormal feature extraction model to extract abnormal features from the image, analyze and process the abnormal features to obtain the abnormal score of each pixel in the image, and judge the abnormal score greater than the preset abnormal value as the tampered pixel, thus completing the location of the tampered point.

[0012] The steps for human-image video similarity detection are as follows: Two input video files to be compared are received. Video frames are extracted from each video file. Then, on one hand, keyframes containing human faces are extracted from the video frames of both video files to form a set of face images to be compared. Face detection and cropping techniques are used to locate and crop the face regions in the images of the face image set. A pre-trained face feature extraction model is used to extract face feature vectors, and an isolated forest algorithm is used to remove outliers to obtain the final face feature vectors of the two video files to be compared. On the other hand, a video feature extraction model is used to extract video content feature vectors from the video frames of both video files. Cosine similarity is used to measure the face feature vectors of the two video files to obtain a face cosine similarity matrix, and cosine similarity is used to measure the video content feature vectors of the two video files to obtain a video content cosine similarity matrix. The final video similarity is output by combining the face cosine similarity matrix and the video content cosine similarity matrix.

[0013] The audio-video similarity detection steps are as follows: Two input video files to be compared are received. Video frames are extracted from each video file. Then, on the one hand, a video feature extraction model is used to extract video content feature vectors from the video frames of the two video files respectively. On the other hand, Mel-frequency cepstral coefficients (MFCCs) in the audio information of the video frames are extracted as audio features. Euclidean distance is used to measure the video content feature vectors of the two video files to obtain image similarity, and cosine similarity is used to measure the audio features of the two video files to obtain audio similarity. The final video similarity is output by combining the image similarity and audio similarity.

[0014] Simultaneously, a system for implementing this method is provided, including a human-computer interaction interface and a processor. The human-computer interaction interface is used to implement the digital intellectual property comparison and detection type receiving step, while other detection steps are completed by the processor. Specifically, the human-computer interaction interface is a multi-page graphical user interface (GUI).

[0015] Specifically, the face feature extraction model is the FaceNet neural network model; the anomaly feature extraction model is the VGG network model; and the video feature extraction model is the ResNet-18 neural network model.

[0016] Specifically, for face similarity detection, a pre-trained FaceNet neural network model is used to extract high-dimensional face feature vectors, and the Euclidean distance between these feature vectors is calculated. The similarity comparison result is then output by comparing this distance with a trained threshold. The graphical user interface (GUI) application provides two functions: face similarity detection between two images and face similarity detection between two sets of images.

[0017] Specifically, for the image tampering detection module, Mantra-Net is used to identify local abnormal features to detect forged pixels and locate the tampered area.

[0018] Specifically, for facial image video similarity comparison, the FaceNet neural network model is used to extract high-dimensional facial feature vectors, and an outlier is removed using the Isolation Forest algorithm. The ResNet-18 neural network model is used to extract high-dimensional video content feature vectors from video frames. Then, cosine similarity is used to measure the similarity between video content and facial features, forming a cosine similarity matrix. Through comprehensive analysis of the similarity scores, a final similarity score is output to determine whether the video has been plagiarized or tampered with.

[0019] Specifically, for audio-video similarity comparison, the ResNet-18 neural network is used to extract the feature vectors of video frames and calculate the Euclidean distance; the MFCC is used to extract the audio features in the video and calculate the cosine distance of the feature vectors to obtain the video similarity and audio similarity respectively. Finally, each is multiplied by a weight factor to obtain the joint similarity, and the degree of similarity and similar position of video segments are judged accordingly.

[0020] For multi-interface GUI, this invention provides five interfaces: face similarity detection for a set of image pairs, face similarity detection between multiple images, image tampering detection, portrait video similarity detection, and audio video similarity detection.

[0021] The beneficial effects of this invention are that by combining video content style and features with facial features and audio features respectively for video similarity analysis, it is possible to analyze video similarity from multiple dimensions, and has the advantages of flexibility, high performance, simple structure and strong practicality. At the same time, this invention integrates image and video similarity detection into a GUI application, which has a high degree of integration and is more convenient and practical. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall system architecture;

[0023] Figure 2 A diagram illustrating face similarity detection;

[0024] Figure 3 This is a schematic diagram of image tampering detection;

[0025] Figure 4 This is a diagram comparing video similarity with facial similarity.

[0026] Figure 5 This is a diagram illustrating the comparison between video similarity and audio similarity.

[0027] Figure 6 This is a schematic diagram of the GUI interface, where, Figure 6-1 Face image detection interface; Figure 6-2 This is the interface for detecting face files. Figure 6-3 This is the interface for image tampering detection. Figure 6-4 An interface for detecting human portrait videos; Figure 6-5 This is an interface for audio and video detection. Detailed Implementation

[0028] This invention relates to digital intellectual property comparison and detection methods in areas such as face similarity detection, image tampering detection, video similarity detection, and audio similarity comparison. It incorporates multi-threading and other optimization techniques to improve processing efficiency, meeting the demands of rapid response and large-scale data processing in practical applications. The implementation of this invention will provide new solutions and technical support for the effective protection of digital intellectual property rights.

[0029] Example 1

[0030] Digital intellectual property anti-counterfeiting detection includes four functions: face similarity detection, image tampering detection, portrait video similarity detection, and audio video similarity detection. Face similarity detection further includes two sub-functions: face image pair detection and face file pair detection. Face image pair detection is used for comparing the face similarity of a set of image pairs. Face file pair detection is used for comparing the face similarity of multiple sets of image pairs.

[0031] I. Face Similarity Detection

[0032] 1. Methods for comparing facial similarity between two images:

[0033] 1) Obtain a set of original image pairs to be compared, and preprocess them, including face recognition and cropping of the original images to be compared, and convert them to a size and format suitable for model input. The set of face images to be compared consists of two images containing suspected identical faces. Face detection and cropping techniques are used to locate and crop the face regions in the images to improve the accuracy of face feature extraction.

[0034] Specifically, when performing face detection and cropping, the dlib library in PyTorch is used to perform face recognition on the original image, and the target face image is cropped after scaling by boundary parameters; the cropped face image is then converted into a size and format suitable for the model input.

[0035] 2) The pre-trained FaceNet deep neural network model is used to extract the feature vectors of the cropped face regions.

[0036] Specifically, the open-source FaceNet neural network model, pre-trained on the ImageNet dataset, was used. The backbone feature extraction network was InceptionResnetV1, consisting of 10 explicit convolutional layers, 5 max pooling layers, residual modules, and ReLU activation functions. An adaptive average pooling layer and one fully connected layer were also used, with dropout set to 0.5 and cross-entropy loss function employed. A cropped face image was used as input to the neural network model to obtain a 128-dimensional face feature vector.

[0037] 3) Evaluate the facial similarity between the two images to be compared based on the obtained facial feature vectors, and then compare the results with the similarity threshold. If the Euclidean distance between the facial feature vectors of the image group is less than the facial similarity threshold, the output facial similarity comparison result is similar faces; otherwise, the output comparison result is dissimilar faces.

[0038] Specifically, facial similarity in this process is represented by calculating the Euclidean distance between the high-dimensional feature vectors of two facial regions. The similarity threshold is obtained through training on the commonly used LFW dataset.

[0039] Specifically, let any two facial feature vectors be A = {a1, a2, ..., a...} 128}, B={b1,b2,…,b 128 The formula for calculating the Euclidean distance between vectors is as follows:

[0040]

[0041] Among them, a i and b i These are the i-th components of A and B, respectively.

[0042] 2. Method for comparing facial similarity between two sets of images:

[0043] 1) Obtain two files containing the original face similarity images to be compared. Each file (i.e., each image group) can contain multiple face images to be compared. Preprocess all original images, including face recognition and cropping of the original comparison images, and convert them to a size and format suitable for model input. The comparison face images consist of two groups, each containing multiple face images. Then, face detection and cropping techniques are used to locate and crop the face regions in the images to improve the accuracy of face feature extraction.

[0044] Specifically, when performing face detection and cropping, the dlib library in PyTorch is used to perform face recognition on the original image, and the target face image is cropped after scaling by boundary parameters. The cropped face image is then converted into a size and format suitable for the model input, forming two sets of face images to be compared.

[0045] 2) Using a pre-trained FaceNet deep neural network model, feature vector groups of the cropped face regions in the two image groups are extracted respectively.

[0046] Specifically, the FaceNet deep neural network model used here is the same as the face similarity detection part of a set of image pairs; the set of face images to be compared is used as the input of the neural network model to obtain two sets of face feature vectors. The two sets of vectors are composed of m 128-dimensional face feature vectors and n 128-dimensional face feature vectors, respectively, where m and n are the number of face images contained in each set of face images to be compared.

[0047] 3) Evaluate the facial similarity between two images to be compared based on the obtained facial feature vector sets, and then compare the results with a threshold. This process calculates the Euclidean distance between each pair of facial feature vectors in the two image sets, obtaining a Euclidean distance matrix. This matrix is ​​then compared with a facial similarity threshold trained using the LFW dataset to obtain the facial similarity comparison results for the image sets to be compared. If the Euclidean distance matrix is ​​greater than the facial similarity threshold, the images are considered dissimilar; if it is less than the threshold, they are considered similar. The results are represented using a heatmap. Each square in the heatmap represents the facial similarity detection result of the corresponding image names in the horizontal and vertical directions.

[0048] Specifically, in the heatmap, similar squares are filled in yellow, while dissimilar squares are filled in purple, making it easy to see the comparison results intuitively.

[0049] II. Image Tampering Detection:

[0050] 1) Obtain the image to be detected, preprocess the original image, and convert the original image to be detected into a size and format suitable for the model input.

[0051] 2) A pre-trained VGG network model is used as the anomaly feature extraction module. The VGG network, with its deep and multi-layered convolutional structure, is well-suited for feature extraction. Its deep network can capture subtle features in images, allowing the VGG network model to extract anomaly features from the test image.

[0052] Specifically, the VGG deep learning network here consists of a 5x5 convolutional layer (Conv), a constrained convolutional layer (BayerConv), an SRMConv layer, and twelve 3x3 convolutional layers (Conv). The SRMConv layer refers to a style-based convolutional neural network recalibration module, which dynamically evaluates the importance of each style, allowing the convolutional network to focus more on meaningful parts.

[0053] 3) The abnormal features are processed through Z-pooling with windows of different sizes to obtain Z-scores for different window sizes. The Z-scores calculated from windows of different sizes are combined into a sequence to generate a four-dimensional feature. This four-dimensional feature is then analyzed using a ConvLSTM2D layer to obtain the abnormal score for each pixel. The abnormal score is compared with a preset value to obtain the tamper detection and localization result.

[0054] Specifically, using windows of sizes 7*7, 15*15, and 31*31 respectively, the Z-score of each pixel is calculated according to the following formula:

[0055]

[0056] Where n×n is the measurement window size, n is the side length of the measurement window, F represents the image to be tested, H and W are the height and width of the image to be tested, i is the row number of the pixel, j is the column number of the pixel, and F[i,j] is the outlier value of pixel [i,j]. The average value of the outlier features in the n×n region. All are intermediate values, σ F The variance of the abnormal feature values ​​is represented by . The Z-scores calculated from windows of different sizes are combined into a sequence, generating a four-dimensional feature of size (n+1)×H×W×L, where L is the number of channels. Using a ConvLSTM2D layer, the Z-scores belonging to different window sizes are analyzed sequentially to obtain the abnormal score for each pixel. This four-dimensional feature is a combination of three-dimensional features based on different window sizes; therefore, analyzing it is essentially analyzing the Z-scores of different window sizes. Pixels with scores greater than 0.2 are considered tampered with, thus allowing for tamper location.

[0057] III. Similarity Comparison Methods for Portrait Videos:

[0058] 1) Obtain the original set of video data to be compared, create two independent threads, and extract frames from the videos to be compared separately: the videos can include different types of content such as movies and short videos, and each pair of videos to be compared is grouped together. Representative frames are extracted from the videos at a fixed frequency using video frame extraction technology.

[0059] Specifically, representative frames are extracted from the video at a frequency of 1 frame every 10 frames.

[0060] 2) The extracted representative frames are preprocessed in two separate threads, including converting them to a format suitable for model input. Multithreading is used to improve frame processing efficiency by creating a thread pool to process face detection of multiple video frames in parallel. Keyframes containing human faces are extracted from the preprocessed frames.

[0061] 3) In extracting keyframes containing human faces, face detection and cropping techniques are used to locate and crop the face regions in the video in two separate threads.

[0062] Specifically, the dlib library in PyTorch is used to perform face recognition on the extracted video frames. x1, y1, x2, and y2 represent the left, top, right, and bottom boundaries of the recognized face bounding box, respectively. The margin represents the scaling factor for cropping the face bounding box, used to extract more comprehensive facial information. The specific face cropping process is as follows:

[0063] dx = margin × (x2 - x1)

[0064] dy = margin × (y2 - y1)

[0065] d1 = x1 - dx

[0066] p1 = y1 - dy

[0067] d2=x2+dx

[0068] p2=y2+dy

[0069] Where dx and dy represent the number of pixels to be expanded in the left and right and up and down directions, respectively, and d1, p1, d2, and p2 represent the left, top, right, and bottom boundaries of the expanded face rectangle, respectively.

[0070] 4) A pre-trained FaceNet deep neural network model is used to extract facial feature vectors from the face regions cropped from video frames in two threads. Simultaneously, a feature forest is used to remove a specified proportion of anomalous features, ensuring the accuracy and consistency of the feature data input to the model.

[0071] Specifically, a pre-trained FaceNet deep neural network model is used to extract facial feature vectors from the face regions cropped from video frames in two threads. The FaceNet deep neural network model used here is the same as the face similarity detection part. The cropped face image is used as input to the neural network model to obtain a 128-dimensional facial feature vector.

[0072] Then, the Isolation Forest algorithm was used to detect and filter out abnormal features at a ratio of 0.2, removing faces with low frequency of occurrence and those with abnormal detection, and forming a face feature vector group; let the formed face feature vector group be X={x1,x2,…x i ,…x n}, where each feature vector x i ∈R 128 Given a 128-dimensional face feature vector; use the following scoring function s to evaluate each feature vector x. i Rate it:

[0073]

[0074] Where E(h(x) i )) is x i The average segmentation depth in an isolated tree, h(x) i ) represents data point x i The depth of an isolated tree, E(h(x) i h(x) is i The mathematical expectation of ), c(n) is the adjustment term for the dataset size n, s(x) i (n) represents point x i Anomaly scoring. The following prediction function is used to determine whether a point is anomaly, yielding x. i Corresponding state value

[0075]

[0076] Finally, the feature vector set X′ of the normal data points is returned:

[0077]

[0078] 5) Using a pre-trained ResNet-18 model, video frame feature vectors are extracted from video frames extracted from the videos in the two threads. Multithreading ends, and face feature vector groups and video feature vector groups for each of the two videos are obtained.

[0079] Specifically, using the ResNet-18 pre-trained model in PyTorch, the output of the average pooling layer (avgpool) is used as the feature vector of the video frame, forming a video feature vector group.

[0080] 6) Use cosine similarity to calculate the similarity between the feature vectors of two video frames and between the feature vectors of faces.

[0081] Specifically, a similarity calculation method based on video content and facial features is used to evaluate the facial similarity and video content similarity between two videos. The cosine similarity between each pair of vectors in the facial feature vector group and the video feature vector group is calculated separately, forming a cosine similarity matrix. Let A = {a1, a2, ..., a...} be any two feature vectors in a vector group. i ,…,a j ,…a m}, B={b1,b2,…,b i ,…,b j ,…b m The formula for calculating cosine similarity is as follows:

[0082]

[0083] Where m is the dimension of the two feature vectors, and || represents the L2 norm.

[0084] 7) Combine facial similarity and video frame similarity to output a comprehensive video similarity score.

[0085] Specifically, the average of the maximum values ​​in each row and column of the cosine similarity matrix is ​​taken to obtain the similarity between the video content and facial features. The average of the similarities between the video content and facial features is then calculated to obtain the video similarity score.

[0086] 8) Compare the video similarity with the similarity threshold obtained from training the training dataset or a manually set threshold to obtain the comparison result.

[0087] Specifically, the manually set threshold is 0.8. Videos with a similarity score greater than 0.8 are considered plagiarized works, while videos with a similarity score less than 0.8 are considered not plagiarized.

[0088] IV. Audio-video similarity comparison methods:

[0089] 1) Read the paths of two video files to be compared. Each path contains at least one video to be compared and the original video. Extract video frames at regular time intervals, and simultaneously obtain the audio information of the video.

[0090] Specifically, the OpenCV library is used to read two video files to be compared. The video to be compared corresponds to the suspected pirated video, and the original video corresponds to the original video.

[0091] Specifically, representative frames are extracted from the video to be compared at fixed time intervals. By setting a sliding window in the video, the entire video content is covered, and key frames containing important information are extracted. At the same time, a WAV file is extracted from the video to obtain the audio information of the video to be compared. The sliding window operation is also performed to extract frames, and then the Mel-frequency cepstral coefficients (MFCCs) of the audio are extracted using the Librosa library.

[0092] 2) A pre-trained ResNet-18 model is used to extract image features from the extracted video frames. Simultaneously, the audio features of the video are obtained.

[0093] Specifically, a pre-trained ResNet-18 model is used to extract features from each video segment and output the feature vector of the video segment; at the same time, the MFCC features of the audio segment are calculated as audio features.

[0094] 3) Calculate the similarity of image and audio features between the video to be compared and the original video using Euclidean distance and cosine similarity algorithms respectively. Then, combine the audio and video similarities in a weighted manner to obtain the total similarity between the two videos.

[0095] Specifically, image similarity uses the Euclidean distance formula:

[0096]

[0097] Here, A and B are the feature vectors of the image, and n is the dimension of the feature vectors. The similarity percentage (Similarity) is converted using the following formula:

[0098]

[0099] Audio similarity is calculated using cosine similarity:

[0100]

[0101] Then, the joint similarity between the audio and video is calculated by weighting the similarity of the two videos. The formula for calculating the joint similarity is as follows:

[0102] Total Similarity

[0103] =Image Similarity*ω image +Audio Similarity*ω audio

[0104] Where, ωimage and ω audio These are the weight values ​​for image and audio similarity, respectively.

[0105] 4) Based on the joint similarity score, determine whether there are segments in the suspected pirated video that are similar to the original video; set a similarity threshold, and segments that exceed the threshold are considered pirated segments.

[0106] Example 2

[0107] A system for implementing the method of Embodiment 1 is provided, which uses a graphical user interface (GUI) to call various functions. The user interface layout adopts a modular design. The GUI includes five main modules: face image pair detection, face file pair detection, image tampering detection, portrait video detection, and audio video detection. Figure 6-1 arrive Figure 6-5 As shown on the left side of the page, the interface icons and buttons are simple and clear, ensuring that users can quickly identify and accurately operate each function. Two auxiliary tool buttons, "Threshold" and "Error Message," are also located on the left side of the page. The "Threshold" button allows users to customize the similarity threshold for face image pair detection and face file pair detection functions, supporting manual adjustment or using the default value; the "Error Message" button displays information about user errors to help users correct their actions.

[0108] Face image detection: such as Figure 6-1 As shown, two image import buttons are located on the right side of the interface, used to upload image 1 and image 2 to be compared, respectively. After successful upload, the images are automatically displayed in the corresponding TextLabel areas. A "Similarity Calculation" button is located in the lower right corner; clicking it starts the detection, and the results are displayed in the "Calculation Results" area.

[0109] Face file pair detection: such as Figure 6-2 As shown, similar to image pair detection, but the two image import buttons on this interface import files containing multiple images, namely file 1 and file 2, representing the two sets of images to be compared. After successful upload, the file path is displayed in the corresponding area. Clicking the "Similarity Calculation" button performs pairwise similarity detection on the images within the file, and the results are displayed in the "Calculation Results" area as a graphical matrix.

[0110] Image tampering detection: such as Figure 6-3 As shown, the interface provides an "Import Image to be Detected" button. After uploading an image, it is automatically displayed in the corresponding TextLabel area. Clicking the "Image Tampering Detection" button then performs the detection, and the results are displayed in the "Detection Results" area.

[0111] Human image video detection: such as Figure 6-4As shown, two "Import Video" buttons are used to upload Video 1 and Video 2, with the video paths displayed in the corresponding areas. After the videos are loaded, clicking the "Start Detection" button initiates the similarity detection for portrait videos, while a progress bar below displays the detection progress in real time. The detection results, including similarity scores and judgments, are displayed in the "Detection Results" box.

[0112] Audio and video detection: such as Figure 6-5 As shown, two import buttons are used to upload the video file to be detected and the comparison video file, completing the "Select Video to be Detected" and "Select Comparison Video" steps. Simultaneously, clicking "Select Result Save Location" allows you to choose the path to save the Excel file of the detection results. After the file is loaded, clicking the "Start Detection" button initiates the audio-video similarity detection. The detection results include the names of the two videos involved in the detection, the similarity judgment result, and the maximum similarity time segment, displayed in the "Detection Results" box. The Excel file also contains specific numerical values ​​for video and audio similarity and joint similarity in different video segments.

[0113] This invention, in video similarity comparison and detection, takes into account the diversity of video styles and adopts detection modes combining "image and video" and "audio and video" to improve the accuracy of similarity detection. When the user selects a video type with multiple images, the image-based video detection mode can be selected; when the selected video type has fewer images and mainly consists of voice and scenery, the audio-based video detection mode can be selected. This design not only better adapts to the diversity of video styles and improves the accuracy of detection, but also provides a more flexible, convenient, and personalized user experience. Simultaneously, this invention integrates multi-dimensional similarity analysis of faces, images, videos, and sounds, providing a comprehensive similarity detection tool. Compared with existing technologies, this system not only improves detection accuracy but also optimizes the user experience through a graphical user interface (GUI). This GUI application integrates multiple functions such as image and video similarity detection and tampering detection, effectively improving the efficiency and accuracy of anti-counterfeiting detection and fully meeting the diverse comparison and detection needs of users, making it easy for both professionals and ordinary users to use. This invention is particularly applicable to fields such as copyright protection and content verification, providing a high-precision, high-efficiency, automated, and widely applicable digital intellectual property protection solution.

Claims

1. A digital intellectual property comparison and detection method based on deep neural networks, characterized in that, Including the following steps: Digital intellectual property comparison and detection type receiving steps: When a face similarity detection instruction is received, the face similarity detection step is triggered; when an image tampering detection instruction is received, the image tampering detection step is triggered; when a portrait video similarity detection instruction is received, the portrait video similarity detection step is triggered; when an audio video similarity detection instruction is received, the audio video similarity detection step is triggered. Face similarity detection steps: Receive the input group of face images to be compared, use face detection and cropping technology to locate and crop the face regions in the images of the group of face images to be compared, use a pre-trained face feature extraction model to extract face feature vector groups corresponding to two face regions in the group of face images to be compared, obtain the face similarity between the two face feature vector groups, compare the face similarity with the similarity threshold, and output the similarity comparison result; Image tampering detection steps: Receive the input image to be detected, preprocess the image, use a pre-trained abnormal feature extraction model to extract abnormal features from the image, analyze and process the abnormal features to obtain the abnormal score of each pixel in the image, and judge the abnormal score greater than the preset abnormal value as the tampered pixel, thus completing the location of the tampered point. The steps for human-image video similarity detection are as follows: Two input video files to be compared are received. Video frames are extracted from each video file. Then, on one hand, keyframes containing human faces are extracted from the video frames of both video files to form a set of face images to be compared. Face detection and cropping techniques are used to locate and crop the face regions in the images of the face image set. A pre-trained face feature extraction model is used to extract face feature vectors, and an isolated forest algorithm is used to remove outliers to obtain the final set of face feature vectors for the two video files to be compared. On the other hand, a video feature extraction model is used to extract video content feature vectors from the video frames of the two video files. Cosine similarity is used to measure the face feature vectors of the two video files to obtain a face cosine similarity matrix, and cosine similarity is used to measure the video content feature vectors of the two video files to obtain a video content cosine similarity matrix. The final video similarity is output by combining the face cosine similarity matrix and the video content cosine similarity matrix. The audio-video similarity detection steps are as follows: Receive two input video files to be compared, extract video frames from each video file, and then, on the one hand, use a video feature extraction model to extract video content feature vectors from the video frames of the two video files respectively; on the other hand, extract the Mel-frequency cepstral coefficients (MFCCs) from the audio information in the video frames as audio features. Image similarity is obtained by measuring the video content feature vectors of two video files to be compared using Euclidean distance, and audio similarity is obtained by measuring the audio features of the two video files to be compared using cosine similarity. The final video similarity is then output by combining the image similarity and audio similarity.

2. The method as described in claim 1, characterized in that, The face feature extraction model is the FaceNet neural network model; the anomaly feature extraction model is the VGG network model; and the video feature extraction model is the ResNet-18 neural network model.

3. The method as described in claim 1, characterized in that, In the face similarity detection step, when performing face similarity detection between two images, the two input images are received as a group of face images to be compared; when performing face similarity detection between two groups of images, two input files are received, one file contains two or more face images to be compared as a group, and the two groups of images are received as a group of face images to be compared.

4. The method as described in claim 1, characterized in that, In the face similarity detection step and the portrait video similarity detection step, the Euclidean distance is used to measure the face similarity between two face feature vectors in a face feature vector group.

5. The method as described in claim 1, characterized in that, The specific method for obtaining the anomaly score of each pixel in the image under test by analyzing and processing the anomaly features is as follows: the anomaly features are passed through detection windows of different sizes to obtain the corresponding scores, the scores corresponding to all detection windows are combined into a sequence and then input into ConvLSTM2D for analysis and processing to obtain the anomaly score of each pixel in the image under test. The score corresponding to the detection window is calculated as follows: Where n×n is the measurement window size, n is the side length of the measurement window, F represents the image to be tested, H and W are the height and width of the image to be tested, i is the row number of the pixel, j is the column number of the pixel, and F[i,j] is the outlier value of pixel [i,j]. The average value of the outlier features in the n×n region. All are intermediate values, σ F The variance of the outlier eigenvalues.

6. The method as described in claim 1, characterized in that, The specific method for combining the face cosine similarity matrix and the video content cosine similarity matrix to output the final video similarity is as follows: The face similarity is calculated by taking the average of the maximum values ​​in each row and column of the face cosine similarity matrix, and the video content similarity is calculated by taking the average of the maximum values ​​in each row and column of the video content cosine similarity matrix. Finally, the average of the face similarity and the video content similarity is calculated to obtain the final video similarity.

7. The method as described in claim 1, characterized in that, The specific method for combining image similarity and audio similarity to output the final video similarity is as follows: The final video similarity score is obtained by multiplying the video similarity score and audio similarity score by their respective weighting factors and then summing them.

8. In the method described in claim 1, if the final video similarity is greater than a preset similarity threshold in the human image video similarity detection step and the audio video similarity detection step, it is determined that there is an infringement of digital intellectual property rights.

9. A digital intellectual property comparison and detection system based on deep neural networks, characterized in that, Including human-computer interaction interface and processor; The human-computer interaction interface is used to trigger the processor to perform face similarity detection steps and display the face similarity detection results when a face similarity detection command is received. When an image tampering detection command is received, the processor's image tampering detection step is triggered, and the image tampering detection result is displayed; when a human face video similarity detection command is received, the processor's human face video similarity detection step is triggered, and the human face video similarity detection result is displayed; when an audio video similarity detection command is received, the processor's audio video similarity detection step is triggered, and the audio video similarity detection result is displayed. The processor is used for: When performing face similarity detection: the input group of face images to be compared is received, face detection and cropping techniques are used to locate and crop the face regions in the images of the group of face images to be compared, a pre-trained face feature extraction model is used to extract face feature vector groups corresponding to two face regions in the group of face images to be compared, the face similarity of the two face feature vectors in the face feature vector group is obtained, and the face similarity is compared with the similarity threshold to output the similarity comparison result. The steps for image tampering detection are as follows: Receive the input image to be detected, preprocess the image, use a pre-trained abnormal feature extraction model to extract abnormal features from the image, analyze and process the abnormal features to obtain an abnormal score for each pixel in the image and output it, and judge the pixel whose abnormal score is greater than the preset abnormal value as the tampered pixel, thus completing the location of the tampered point. The steps for performing facial image video similarity detection are as follows: Two input video files to be compared are received. Video frames are extracted from each video file. Then, on one hand, keyframes containing facial features are extracted from the video frames of both video files to form a set of facial image groups to be compared. Facial detection and cropping techniques are used to locate and crop the facial regions from the keyframes containing facial features. A pre-trained facial feature extraction model is used to extract facial feature vectors, and an isolated forest algorithm is used to remove outliers to obtain the final set of facial feature vectors for the two video files to be compared. On the other hand, a video feature extraction model is used to extract video content feature vectors from the video frames of both video files. Cosine similarity is used to measure the facial feature vectors of the two video files to obtain a face cosine similarity matrix, and cosine similarity is used to measure the video content feature vectors of the two video files to obtain a video content cosine similarity matrix. The final video similarity is output by combining the face cosine similarity matrix and the video content cosine similarity matrix. The steps for audio-video similarity detection are as follows: Receive two input video files to be compared, extract video frames from each video file, and then, on the one hand, use a video feature extraction model to extract video content feature vectors from the video frames of the two video files respectively; on the other hand, extract the Mel-frequency cepstral coefficients (MFCCs) from the audio information in the video frames as audio features. Image similarity is obtained by measuring the video content feature vectors of two video files to be compared using Euclidean distance, and audio similarity is obtained by measuring the audio features of the two video files to be compared using cosine similarity. The final video similarity is then output by combining the image similarity and audio similarity.

Images