Video detection method, apparatus, device, storage medium, and program product

Abstract

The application provides a video detection method, device, equipment, storage medium and program product, the method comprises the following steps: obtaining a to-be-processed video; visual feature extraction is carried out on the image of the to-be-processed video, and visual feature information is obtained; audio feature extraction is carried out on the voice of the to-be-processed video, and audio feature information is obtained; whether the to-be-processed video is a safe video is determined based on the visual feature information and the audio feature information. The video detection method, device, equipment, storage medium and program product provided by the application are used for improving the accuracy of safety detection on the to-be-processed video.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and speech recognition, and in particular to a video detection method, apparatus, device, storage medium, and program product. Background Technology

[0002] In the field of content (such as video) security, it is an important task to conduct security testing on content to determine whether it contains sensitive information.

[0003] In related technologies, video is typically input into a pre-trained encoder-decoder (transformer) network, which performs security checks on the video and outputs the detection results of whether the video contains sensitive information.

[0004] In the aforementioned related technologies, video detection and processing are performed solely through transformer networks, resulting in poor accuracy in video security detection. Summary of the Invention

[0005] This invention provides a video detection method, apparatus, device, storage medium, and program product to address the shortcomings of poor accuracy in video security detection in existing technologies, thereby improving the accuracy of video security detection.

[0006] This invention provides a video detection method, comprising:

[0007] Get the video to be processed;

[0008] Visual feature extraction is performed on the images of the video to be processed to obtain visual feature information;

[0009] Audio feature extraction is performed on the speech of the video to be processed to obtain audio feature information;

[0010] Based on visual and audio features, determine whether the video to be processed is a safe video.

[0011] According to a video detection method provided by the present invention, visual feature extraction is performed on the image of the video to be processed to obtain visual feature information, including:

[0012] By using a semantic feature extraction network, semantic features are extracted from the image to obtain visual semantic feature information;

[0013] By using a syntactic feature extraction network, syntactic features are extracted from the image to obtain visual syntactic feature information;

[0014] A salient object detection network is used to extract salient object features from an image, thereby obtaining salient object feature information.

[0015] Visual feature information includes visual semantic feature information, visual syntactic feature information, and salient object feature information.

[0016] According to a video detection method provided by the present invention, audio feature extraction is performed on the speech of the video to be processed to obtain audio feature information, including:

[0017] Speech feature extraction network is used to extract speech features and obtain speech feature information;

[0018] By using a multi-frequency domain reconstruction network, frequency domain features are extracted from speech feature information to obtain frequency domain feature information;

[0019] By using a multi-frequency domain relationship learning network, frequency domain relationship features are extracted from frequency domain feature information to obtain audio feature information.

[0020] According to a video detection method provided by the present invention, based on visual feature information and audio feature information, it determines whether a video to be processed is a safe video, including:

[0021] Based on visual and audio feature information, the target depth hash code of the video to be processed is determined;

[0022] Based on the target depth hash code, determine whether the video to be processed is a safe video.

[0023] According to a video detection method provided by the present invention, the target depth hash code of the video to be processed is determined based on visual feature information and audio feature information, including:

[0024] By using graph convolutional neural networks, visual and audio feature information is processed to obtain visual semantic features including object relationships, visual syntactic features including object relationships, and auditory features including object relationships.

[0025] Multi-source feature fusion processing is performed on visual semantic features including object relationships, visual syntactic features including object relationships, and auditory features including object relationships to obtain attention index information and word vector information;

[0026] Multi-source feature fusion processing is performed on visual semantic features, syntactic feature extraction networks, and audio features included in visual feature information to obtain attention weight information;

[0027] The attention index information, word vector information, and attention weight information are processed through an encoding and decoding network to obtain the encoding and decoding results of the video to be processed.

[0028] The target depth hash code is obtained by hashing the encoding and decoding results through a hash encoding network.

[0029] According to a video detection method provided by the present invention, based on a target depth hash code, it determines whether a video to be processed is a safe video, including:

[0030] Retrieve a pre-stored blacklist, which includes multiple preset deep hash codes. The videos corresponding to the preset deep hash codes are unsafe videos.

[0031] If a preset depth hash code exists in the blacklist with a Hamming distance less than or equal to the target depth hash code, the video to be processed is determined to be an insecure video; if no preset depth hash code exists in the blacklist with a Hamming distance less than or equal to the target depth hash code, the video to be processed is determined to be a secure video.

[0032] The present invention also provides a video detection device, comprising:

[0033] The acquisition module is used to acquire the video to be processed;

[0034] The processing module is used to extract visual features from the images of the video to be processed, and obtain visual feature information.

[0035] The processing module is also used to extract audio features from the speech of the video to be processed, and obtain audio feature information;

[0036] The processing module is also used to determine whether the video to be processed is a safe video based on visual and audio feature information.

[0037] According to the video detection device provided by the present invention, the processing module is specifically used for:

[0038] By using a semantic feature extraction network, semantic features are extracted from the image to obtain visual semantic feature information;

[0039] By using a syntactic feature extraction network, syntactic features are extracted from the image to obtain visual syntactic feature information;

[0040] A salient object detection network is used to extract salient object features from an image, thereby obtaining salient object feature information.

[0041] Visual feature information includes visual semantic feature information, visual syntactic feature information, and salient object feature information.

[0042] According to the video detection device provided by the present invention, the processing module is further specifically used for:

[0043] Speech feature extraction network is used to extract speech features and obtain speech feature information;

[0044] By using a multi-frequency domain reconstruction network, frequency domain features are extracted from speech feature information to obtain frequency domain feature information;

[0045] By using a multi-frequency domain relationship learning network, frequency domain relationship features are extracted from frequency domain feature information to obtain audio feature information.

[0046] According to the video detection device provided by the present invention, the processing module is further specifically used for:

[0047] Based on visual and audio feature information, the target depth hash code of the video to be processed is determined;

[0048] Based on the target depth hash code, determine whether the video to be processed is a safe video.

[0049] According to the video detection device provided by the present invention, the processing module is further specifically used for:

[0050] By using graph convolutional neural networks, visual and audio feature information is processed to obtain visual semantic features including object relationships, visual syntactic features including object relationships, and auditory features including object relationships.

[0051] Multi-source feature fusion processing is performed on visual semantic features including object relationships, visual syntactic features including object relationships, and auditory features including object relationships to obtain attention index information and word vector information;

[0052] Multi-source feature fusion processing is performed on visual semantic features, syntactic feature extraction networks, and audio features included in visual feature information to obtain attention weight information;

[0053] The attention index information, word vector information, and attention weight information are processed through an encoding and decoding network to obtain the encoding and decoding results of the video to be processed.

[0054] The target depth hash code is obtained by hashing the encoding and decoding results through a hash encoding network.

[0055] According to the video detection device provided by the present invention, the processing module is further specifically used for:

[0056] Retrieve a pre-stored blacklist, which includes multiple preset deep hash codes. The videos corresponding to the preset deep hash codes are unsafe videos.

[0057] If a preset depth hash code exists in the blacklist with a Hamming distance less than or equal to the target depth hash code, the video to be processed is determined to be an insecure video; if no preset depth hash code exists in the blacklist with a Hamming distance less than or equal to the target depth hash code, the video to be processed is determined to be a secure video.

[0058] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the video detection methods described above.

[0059] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the video detection methods described above.

[0060] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the video detection methods described above.

[0061] The video detection method, apparatus, equipment, storage medium, and program products provided by this invention determine whether a video to be processed is a safe video by using visual and audio feature information. This achieves the goal of determining whether a video to be processed is a safe video through multi-source features, thereby improving the accuracy of security detection of the video to be processed. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0063] Figure 1 This is a flowchart illustrating the video detection method provided by the present invention;

[0064] Figure 2 This is a schematic diagram of the structure of the first training network provided by the present invention;

[0065] Figure 3 This is a schematic diagram of the structure of the second training network provided by the present invention;

[0066] Figure 4 This is a schematic diagram of the structure of the third training network provided by the present invention;

[0067] Figure 5 This is a schematic diagram of the structure of the fourth training network provided by the present invention;

[0068] Figure 6 This is a schematic diagram of the structure of the fifth training network provided by the present invention;

[0069] Figure 7 This is a schematic diagram of the structure of the sixth training network provided by the present invention;

[0070] Figure 8This is a flowchart of the video detection method provided by the present invention;

[0071] Figure 9 This is a schematic diagram of the video detection device provided by the present invention;

[0072] Figure 10 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0074] In related technologies, video detection processing using transformer networks focuses only on global features of images in the video, resulting in poor accuracy in video security detection.

[0075] In this invention, in order to improve the accuracy of security detection of videos, the inventors conceived of extracting the visual and audio features of the video to obtain the target depth hash code, and determining whether the video is a secure video based on the target depth hash code. This approach not only focuses on the image features of the video but also on its audio features, thereby improving the accuracy of security detection of videos.

[0076] The video detection method provided by the present invention is described below with reference to specific embodiments.

[0077] Figure 1 This is a flowchart illustrating the video detection method provided by the present invention. Figure 1 As shown, the method includes:

[0078] Step 101: Obtain the video to be processed.

[0079] Optionally, the execution subject of this embodiment of the invention is an electronic device, which can be a video detection device installed in the electronic device. The video detection device can be implemented through a combination of software and / or hardware.

[0080] The video to be processed can be any video that needs to be security tested.

[0081] For example, the video to be processed is one that needs to be published on an online platform. This could be a video downloaded from an online platform, or a video recorded by an electronic device. The source of the video to be processed is not limited here.

[0082] Step 102: Extract visual features from the images of the video to be processed to obtain visual feature information.

[0083] The image is a video image included in the video to be processed.

[0084] Visual feature information can reflect both local and global features of an image. Local and global features are complementary.

[0085] Step 103: Extract audio features from the speech of the video to be processed to obtain audio feature information.

[0086] The voice is the voice configured for the video to be processed.

[0087] Step 104: Based on visual and audio feature information, determine whether the video to be processed is a safe video.

[0088] exist Figure 1 In the video detection method provided in the embodiment, visual feature information and audio feature information are used to determine whether the video to be processed is a safe video. This achieves the goal of determining whether the video to be processed is a safe video through multi-source features, thereby improving the accuracy of security detection of the video to be processed.

[0089] Optionally, the visual feature extraction model includes a semantic feature extraction network, a syntactic feature extraction network, and a salient object detection network.

[0090] Specifically, step 102 includes:

[0091] By using a semantic feature extraction network, semantic features are extracted from the image to obtain visual semantic feature information;

[0092] By using a syntactic feature extraction network, syntactic features are extracted from the image to obtain visual syntactic feature information;

[0093] A salient object detection network is used to extract salient object features from an image, thereby obtaining salient object feature information.

[0094] Visual feature information includes visual semantic feature information, visual syntactic feature information, and salient object feature information.

[0095] Unlike existing technologies that rely solely on transformer networks for video detection, which lack multi-source understanding and characterization, resulting in poor accuracy in security detection, transformer networks only learn global features and fail to characterize key local content and objects within the image. This leads to missed detections and misjudgments of similar content from different sources. For example, inserting normal and abnormal content into the same image background can yield similar detection results. In contrast, this invention utilizes a semantic feature extraction network to obtain visual semantic features, a syntactic feature extraction network to obtain visual syntactic features, and a salient object detection network to obtain salient object features. This enables multi-source understanding and characterization of the video, improving the accuracy of security detection. Furthermore, the complementary nature of visual semantic and visual syntactic features allows for the learning of global image features and key local content, enabling object characterization within the image and avoiding missed detections and misjudgments of similar content from different sources, thus enhancing the accuracy of video detection.

[0096] In this invention, it can be achieved through the appendix as shown. Figure 2 The first training network shown yields a semantic feature extraction network. The following section combines... Figure 2 The first training network will be described.

[0097] Figure 2 This is a schematic diagram of the structure of the first training network provided by the present invention. (See diagram below.) Figure 2 As shown, it includes: an active learning module, a feature extraction network, and a prediction and loss function module.

[0098] The active learning module is used to label unlabeled sample images and then adaptively adjust the labeling threshold.

[0099] The main structure of a feature extraction network can be a deep learning network such as ResNet (e.g., ResNet50) or EfficientNet. The input of the feature extraction network is an image, and the output is visual depth features (e.g., a 512-dimensional feature vector).

[0100] The prediction and loss function module is used for prediction based on the loss function Ls semantic Adjust the model parameters of the first training network until the model parameters in the first training network converge.

[0101] Ls semantic =L CE (X lab ,Y lab )+L Semi-CE (X semi ,Y semi )+ KL(f(X unlab ),f(aug(Xunlab )));

[0102] Among them, L CE L represents the classification loss. Semi- X represents the semi-supervised classification loss, KL represents the semi-supervised distribution fitting loss, and X represents the semi-supervised distribution fitting loss. lab Y represents a labeled sample image. lab Represents sample image X lab The corresponding label, X semi Y represents the sample image that has been self-labeled through the active learning module. semi Represents sample image X semi The corresponding label, X unlab f(X) represents an unlabeled sample image. unlab This indicates that the feature extraction network is effective for the sample image X. unmlab Output feature results, f(aug(X) unlab )) represents the sample image X unlab The feature results are obtained through a feature extraction network after data augmentation.

[0103] From Ls semantic As can be seen from this, labeled sample images have an effect on L CE The unlabeled sample images are applied to KL, and the labeled sample images obtained after the unlabeled sample images are processed by the active learning module are applied to L. Semi- Above. In this invention, the labeled sample image acts on L. CE The unlabeled sample images are applied to KL, and the labeled sample images obtained after the unlabeled sample images are processed by the active learning module are applied to L. Semi-CE This ensures that the feature distributions of the disturbed and undisturbed images are as consistent as possible.

[0104] The training process of the first training network is described below: The active learning module is disabled, and the feature extraction network is trained using a labeled training set (including labeled samples and their labels) for one epoch of supervised iterative training. During this iteration, Ls semantic Only L is included CE (X lab ,Y labThe pre-trained feature extraction network is used to predict the labels of unlabeled samples in an unlabeled dataset. These unlabeled samples can be obtained from the open internet or through web scraping. The top 10% of unlabeled samples with the highest prediction confidence are selected from the unlabeled dataset. These 10% of unlabeled samples, along with their labels and the labeled training set, are used as the target training set. The active learning module is then enabled, and the target training set is used to perform multiple rounds of semi-supervised iterative training on the active learning module and the semantic feature extraction network. semantic Including L CE (X lab ,Y lab ), L Semi-CE (X semi ,Y semi ), KL(f(X) unlab ),f(aug(X unlab ))).

[0105] Optionally, if all model parameters in the first training network converge, the feature extraction network is determined to be the semantic feature extraction network of this invention.

[0106] Optionally, the syntactic feature extraction network can obtain visual syntactic feature information using either method 11 or method 12.

[0107] Method 11: Extract features from the image using the Histogram of Oriented Gradients (HoG) operator to obtain HoG features; extract features from the image using the Scale Invariant Feature Transform (SIFT) operator to obtain SIFT features; extract features from the image using the Local Maximal Occurrence (LoMo) model in the syntactic feature extraction network to obtain LoMo features; extract features from the image using the Local Binary Pattern (LBP) operator to obtain LBP features; process the HoG features, SIFT features, LoMo features, and LBP features through the attention-based Principal Component Analysis (PCA) feature fusion subnetwork in the syntactic feature extraction network to obtain visual syntactic feature information.

[0108] Method 12: Extract visual syntactic features from the image using a target operator or a target sub-network. The target operator can be any of the HoG, SIFT, or LBP operators, and the target sub-network can be a LoMo model. For example, if the target operator is the HoG operator, the HoG features are determined as the visual syntactic features.

[0109] In the specific implementation, the cross-platform computer vision and machine learning software library (OpenCV) based on graphics processing unit (GPU) acceleration is used to obtain HoG features, SIFT features, and LBP features; the LoMo model in the GPU-accelerated web crawler (Python) machine learning library (PyTorch) is used to obtain LoMo features.

[0110] An attention-based PCA feature fusion subnetwork is used to condense lengthy syntactic features into 512-dimensional visual syntactic feature information.

[0111] The attention-based PCA feature fusion subnetwork includes a squeeze-and-excitation (SE) block and a trainable PCA block.

[0112] Optionally, the trainable PCA module consists of two fully connected layers for dimensionality reduction. The output of the first fully connected layer is the dimensionality-reduced feature, and the second fully connected layer performs feature reconstruction on the dimensionality-reduced feature.

[0113] Optionally, the salient object detection network can be obtained based on the second trained network. The following section combines... Figure 3 The second training network is described in detail.

[0114] Figure 3 This is a schematic diagram of the structure of the second training network provided by the present invention. Figure 3 As shown, the second training network includes a salient object detection feature extraction module and a decoupling learning module.

[0115] The salient object detection feature extraction module takes an image as input and outputs a feature map corresponding to that image. This module can be a commonly used detection and segmentation network, such as the Mask R-CNN network.

[0116] The input to the decoupled learning module is the output of the salient object detection feature extraction module. The output of the decoupled learning module is a salient feature map, a non-salient feature map, and the final salient object location. The decoupled learning module consists of three convolutional layers.

[0117] Optionally, the decoupling learning module uses the downstream task loss function Ls based on the decoupling learning module. dis Conduct training.

[0118] Ls dis =(L sal (f1)-L non-s (f1))+(L non-sal (f2)-L sal (f2));

[0119] Where f1 represents the saliency feature map of the sample, f2 represents the non-saliency feature map of the sample, and L sal L represents the loss function used to improve the accuracy of salient feature map prediction; its role is to make the prediction of salient targets more accurate. non-s This represents the loss function used to improve the prediction accuracy of non-saliency feature maps, making the prediction of non-saliency objects more accurate.

[0120] It should be noted that f1 optimizes for salient objects while suppressing the performance of non-salient object detection; f2 has the opposite constraint.

[0121] Specifically, the loss function Ls can be adjusted based on the Stochastic Gradient Descent (SGD) method. dis Optimize the process.

[0122] Optionally, step 103 includes:

[0123] Speech feature extraction network is used to extract speech features and obtain speech feature information;

[0124] By using a multi-frequency domain reconstruction network, frequency domain features are extracted from speech feature information to obtain frequency domain feature information;

[0125] By using a multi-frequency domain relationship learning network, frequency domain relationship features are extracted from frequency domain feature information to obtain audio feature information.

[0126] Optionally, a speech feature extraction network, a multi-frequency domain reconstruction network, and a multi-frequency domain relation learning network can be obtained based on the third training network. The following section combines... Figure 4 The third training network will be explained.

[0127] Figure 4 This is a schematic diagram of the structure of the third training network provided by the present invention. (See diagram below.) Figure 4 As shown, the third training network includes: a Recurrent Neural Network (RNN) feature extraction module, a multi-frequency domain reconstruction module, and a multi-frequency domain relation learning module.

[0128] The RNN feature extraction module was built using an RNN module in PyTorch that includes Gate Recurrent Units (GRUs). The multi-frequency domain reconstruction module consists of a 3-layer Multilayer Perceptron (MLP). The multi-frequency domain relation learning network also consists of a 3-layer MLP.

[0129] The input to the RNN feature extraction module is speech, and the output is speech features.

[0130] The input to the multi-frequency domain reconstruction module is speech features, and the output is speech information in multiple frequency bands.

[0131] The input to the multi-frequency domain relation learning module is speech information from multiple frequency bands, and the output is speech features modulated by multi-frequency domain relations.

[0132] During the training of the second training network, based on the loss function Ls voice Update all model parameters in the second training network until all model parameters converge.

[0133] Ls voice =L CE +L RE ;

[0134] Among them, L CE L represents the classification loss. RE This indicates the losses incurred during reconstruction.

[0135] When all model parameters converge, the RNN feature extraction module is determined to be the speech feature extraction network; the multi-frequency domain reconstruction module is determined to be the multi-frequency domain reconstruction network; and the multi-frequency domain relation learning module is determined to be the multi-frequency domain relation learning network.

[0136] Optionally, determining whether the video to be processed is a safe video based on visual and audio feature information includes: determining the target depth hash code of the video to be processed based on visual and audio feature information; and determining whether the video to be processed is a safe video based on the target depth hash code.

[0137] Among these steps, the target depth hash code of the video to be processed is determined based on visual and audio feature information, including:

[0138] By using graph convolutional neural networks, visual and audio feature information is processed to obtain visual semantic features including object relationships, visual syntactic features including object relationships, and auditory features including object relationships.

[0139] Multi-source feature fusion processing is performed on visual semantic features including object relationships, visual syntactic features including object relationships, and auditory features including object relationships to obtain attention index information and word vector information;

[0140] Multi-source feature fusion processing is performed on visual semantic features, syntactic feature extraction networks, and audio features included in visual feature information to obtain attention weight information;

[0141] The attention index information, word vector information, and attention weight information are processed through an encoding and decoding network to obtain the encoding and decoding results of the video to be processed.

[0142] The target depth hash code is obtained by hashing the encoding and decoding results through a hash encoding network.

[0143] Optionally, a graph convolutional neural network (GCN) can be obtained based on the fourth trained network. The following section combines... Figure 5 The fourth training network will be explained.

[0144] Figure 5 This is a schematic diagram of the structure of the fourth training network provided by the present invention. (See diagram below.) Figure 5 As shown, the fourth training network includes a multi-source feature extraction module for salient objects and a graph modeling module.

[0145] The multi-source feature extraction module for salient objects uses the outputs of the first, second, and third training networks to extract features.

[0146] The graph modeling module is used to model the output of the multi-source feature extraction module for salient objects. It realizes that each feature of each object in the image (such as people, animals, and plants) is a node, and the multi-source features of an object are connected in pairs; the corresponding features of adjacent objects (the object B that is closest to object A in the same image is adjacent) are connected in pairs (for example, semantic features are connected to semantic features, syntactic features are connected to syntactic features, etc.).

[0147] Based on the classification loss of the images and nodes mentioned above, all model parameters of the fourth training network are updated until all model parameters of the fourth training network converge. When all model parameters of the fourth training network converge, the fourth training network is determined to be a graph convolutional neural network.

[0148] The encoder-decoder network can be obtained based on the fifth trained network. The following section combines... Figure 6 The fifth training network will be explained.

[0149] Figure 6 This is a schematic diagram of the structure of the fifth training network provided by the present invention. (See diagram below.) Figure 6 As shown, the fifth training network includes: a feature encoding module, a self-attention module, and a feature simplification module.

[0150] The feature encoding module consists of three residual blocks.

[0151] The input to the feature encoding module is the multi-source features of the salient object (including) and the multi-source features trained by GCN. The output is Q vector, K vector, and V vector (Q is produced by the multi-source features trained by GCN, and K and V are produced by the multi-source features of the salient object).

[0152] The self-attention module consists of a transformer network used to compute the attention matrices of the Q and K vectors, and then perform a self-attention operation with the V vector to obtain a feature map. The feature map contains the relationships between salient objects and multi-source feature information.

[0153] The feature simplification module uses a fully connected layer to reduce the dimensionality of the feature map after the self-attention operation, thereby reducing the feature dimension (in the specific implementation, 128-dimensional features can achieve a balance between performance and storage).

[0154] Optionally, the fifth training network is trained based on the classification loss for sensitive content detection until all model parameters in the fifth training network converge. The self-attention module and feature simplification module whose model parameters have converged are then identified as the encoder-decoder network.

[0155] Optionally, a hash encoding network can be obtained based on the sixth training network. The structure of the sixth training network is explained below with reference to Figure 7.

[0156] Figure 7 This is a schematic diagram of the structure of the sixth training network provided by the present invention. (See diagram below.) Figure 7 As shown, the sixth training network includes a basic feature extraction module and a dual attention module.

[0157] The basic feature extraction module takes the original floating-point features as input and outputs compressed 128-dimensional floating-point features. The basic feature extraction module consists of a 5-layer MLP.

[0158] The dual attention module takes as input the output of the basic feature extraction module and outputs the final floating-point features. The dual attention module consists of two SE blocks and a fusion submodule. Specifically, one SE block in the dual attention module performs a position-based attention operation to obtain the first floating-point feature, while the other SE block performs a channel-based attention operation to obtain the second floating-point feature. The fusion submodule adds the first and second floating-point features to obtain the final floating-point feature.

[0159] Optionally, based on the hash loss function, all model parameters of the sixth trained network are updated. If all model parameters converge, the dual attention module is determined to be a hash encoding network. Optionally, the hash loss function can be a commonly used quantization loss function or a classification loss function. It should be noted that after the above training process, the resulting video detection network is shown in the attached figure. Figure 8 As shown.

[0160] Alternatively, the safety of the video to be processed can be determined based on the target depth hash code using methods 21 and 22.

[0161] Method 21: Obtain a pre-stored blacklist, which includes multiple preset depth hash codes. The videos corresponding to the preset depth hash codes are unsafe videos. If there is a preset depth hash code in the blacklist whose Hamming distance to the target depth hash code is less than or equal to a preset threshold, then the video to be processed is determined to be an unsafe video. If there is no preset depth hash code in the blacklist whose Hamming distance to the target depth hash code is less than or equal to a preset threshold, then the video to be processed is determined to be a safe video.

[0162] Method 22: Obtain a pre-stored blacklist, which includes multiple preset depth hash codes. The videos corresponding to the preset depth hash codes are unsafe videos. If a preset depth hash code with the same value as the target depth hash code exists in the blacklist, the video to be processed is determined to be an unsafe video. If no preset depth hash code with the same value as the target depth hash code exists in the blacklist, the video to be processed is determined to be a safe video.

[0163] In this invention, the type of the preset depth hash code in the blacklist stored by the electronic device is int8, which can reduce the storage requirements and greatly reduce the time required to retrieve a preset depth hash code whose Hamming distance from the target depth hash code is less than or equal to a preset threshold, or a preset depth hash code that is the same as the target depth hash code.

[0164] Figure 8This is a flowchart of the video detection network provided by the present invention. Figure 8 As shown, the video detection network includes: semantic feature extraction network, syntactic feature extraction network, salient object detection network, audio feature extraction network, graph convolutional neural network, encoder-decoder network, and hash encoder network.

[0165] Figure 8 The video detection network shown can process multiple preset videos to obtain a preset depth hash code corresponding to each preset video in the blacklist. The preset depth hash code is of type int8. Figure 8 The video detection network shown can also process the video to be processed to obtain the target depth hash code.

[0166] Audio feature extraction networks include: speech feature extraction networks, multi-frequency domain reconstruction networks, and multi-frequency domain relation learning networks. These networks are used to extract audio features from the speech of the video being processed, obtaining audio feature information.

[0167] exist Figure 8 In this context, Q represents attention weight information, K represents attention index information, and V represents word vector information.

[0168] The video detection device provided by the present invention is described below. The video detection device described below and the video detection method described above can be referred to in correspondence.

[0169] Figure 9 This is a schematic diagram of the video detection device provided by the present invention. Figure 9 As shown, the video detection device includes:

[0170] Module 910 is used to acquire the video to be processed;

[0171] Processing module 920 is used to extract visual features from the images of the video to be processed to obtain visual feature information;

[0172] The processing module 920 is also used to extract audio features from the speech of the video to be processed, and obtain audio feature information;

[0173] The processing module 920 is also used to determine whether the video to be processed is a safe video based on visual feature information and audio feature information.

[0174] According to the video detection device provided by the present invention, the processing module 920 is specifically used for:

[0175] By using a semantic feature extraction network, semantic features are extracted from the image to obtain visual semantic feature information;

[0176] By using a syntactic feature extraction network, syntactic features are extracted from the image to obtain visual syntactic feature information;

[0177] A salient object detection network is used to extract salient object features from an image, thereby obtaining salient object feature information.

[0178] Visual feature information includes visual semantic feature information, visual syntactic feature information, and salient object feature information.

[0179] According to the video detection device provided by the present invention, the processing module 920 is further specifically used for:

[0180] Speech feature extraction network is used to extract speech features and obtain speech feature information;

[0181] By using a multi-frequency domain reconstruction network, frequency domain features are extracted from speech feature information to obtain frequency domain feature information;

[0182] By using a multi-frequency domain relationship learning network, frequency domain relationship features are extracted from frequency domain feature information to obtain audio feature information.

[0183] According to the video detection device provided by the present invention, the processing module 920 is further specifically used for:

[0184] Based on visual and audio feature information, the target depth hash code of the video to be processed is determined; based on the target depth hash code, it is determined whether the video to be processed is a safe video.

[0185] According to a video detection device provided by the present invention, the processing module 920 is further specifically configured to: process visual feature information and audio feature information through a graph convolutional neural network to obtain visual semantic features including object relationships, visual syntactic features including object relationships, and auditory features including object relationships; perform multi-source feature fusion processing on the visual semantic features including object relationships, the visual syntactic features including object relationships, and the auditory features including object relationships to obtain attention index information and word vector information; perform multi-source feature fusion processing on the visual semantic feature information and syntactic feature extraction network included in the visual feature information, as well as the audio feature information, to obtain attention weight information; process the attention index information, word vector information, and attention weight information through an encoding and decoding network to obtain the encoding and decoding result of the video to be processed; and perform hash encoding on the encoding and decoding result through a hash encoding network to obtain the target depth hash code.

[0186] According to the video detection device provided by the present invention, the processing module 920 is further specifically configured to: acquire a pre-stored blacklist, the blacklist including multiple preset depth hash codes, the videos corresponding to the preset depth hash codes being unsafe videos; if there is a preset depth hash code in the blacklist whose Hamming distance to the target depth hash code is less than or equal to a preset threshold, then the video to be processed is determined to be an unsafe video; if there is no preset depth hash code in the blacklist whose Hamming distance to the target depth hash code is less than or equal to a preset threshold, then the video to be processed is determined to be a safe video.

[0187] Figure 10 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. For example... Figure 10 As shown, the electronic device may include a processor 1010, a communications interface 1020, a memory 1030, and a communication bus 1040. The processor 1010, communications interface 1020, and memory 1030 communicate with each other via the communication bus 1040. The processor 1010 can call logical instructions in the memory 1030 to execute a video detection method. This method includes: acquiring a video to be processed; extracting visual features from the image of the video to be processed to obtain visual feature information; extracting audio features from the speech of the video to be processed to obtain audio feature information; and determining whether the video to be processed is a safe video based on the visual feature information and the audio feature information.

[0188] Furthermore, the logical instructions in the aforementioned memory 1030 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0189] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the video detection method provided by the above methods. The method includes: acquiring a video to be processed; extracting visual features from the image of the video to be processed to obtain visual feature information; extracting audio features from the speech of the video to be processed to obtain audio feature information; and determining whether the video to be processed is a safe video based on the visual feature information and the audio feature information.

[0190] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program is implemented to perform the video detection method provided by the above methods. The method includes: acquiring a video to be processed; extracting visual features from the image of the video to be processed to obtain visual feature information; extracting audio features from the speech of the video to be processed to obtain audio feature information; and determining whether the video to be processed is a safe video based on the visual feature information and the audio feature information.

[0191] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0192] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of various embodiments or some parts of embodiments.

[0193] This invention is not limited by hardware platform or programming language. Common programming languages ​​such as C, C++, or Python can be used to implement the methods described herein and achieve the desired results. In the following specific embodiments, this method provides a specific implementation environment for reference. In practical applications, it can be flexibly adjusted according to specific application scenarios. In this specific embodiment, the invention uses a server including a 3.1GHz 128-core central processing unit and 256GB of memory. The training and video detection methods involved in this invention are all processed on the eight Nvidia Tesla V100 GPUs mounted on the aforementioned server. On the GPUs, this invention uses PyTorch as the software platform for the development and implementation of specific functions.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video detection method, characterized in that, include: Get the video to be processed; Visual feature extraction is performed on the images of the video to be processed to obtain visual feature information; Audio features are extracted from the speech of the video to be processed to obtain audio feature information; Based on the visual feature information and the audio feature information, determine whether the video to be processed is a safe video; The step of determining whether the video to be processed is a safe video based on the visual feature information and the audio feature information includes: Based on the visual feature information and the audio feature information, the target depth hash code of the video to be processed is determined; Based on the target depth hash code, determine whether the video to be processed is a safe video; The step of determining the target depth hash code of the video to be processed based on the visual feature information and the audio feature information includes: The visual and audio feature information are processed by a graph convolutional neural network to obtain visual semantic features including object relationships, visual syntactic features including object relationships, and auditory features including object relationships. The graph convolutional neural network performs graph modeling on the input features in the following way: each feature of each object in the image is treated as a node, multi-source features of the same object are connected pairwise, and corresponding features between adjacent objects are connected pairwise. Multi-source feature fusion processing is performed on the visual semantic features including object relationships, the visual syntactic features including object relationships, and the auditory features including object relationships to obtain attention index information and word vector information; The visual semantic features and syntactic feature extraction network included in the visual feature information, as well as the audio feature information, are subjected to multi-source feature fusion processing to obtain attention weight information; The attention index information, word vector information, and attention weight information are processed through an encoding / decoding network to obtain the encoding / decoding result of the video to be processed. The encoding / decoding network includes a feature encoding module, a self-attention module, and a feature simplification module. The feature encoding module generates a Q-vector based on the multi-source features output by the graph convolutional neural network, and generates K-vectors and V-vectors based on the multi-source features of salient objects. The self-attention module calculates an attention matrix based on the Q-vector and the K-vector, and performs a self-attention operation between the attention matrix and the V-vector to obtain a feature map. The feature simplification module performs dimensionality reduction processing on the feature map to obtain the encoding / decoding result. The target depth hash code is obtained by hashing the encoding and decoding results through a hash encoding network.

2. The video detection method according to claim 1, characterized in that, The step of extracting visual features from the images of the video to be processed to obtain visual feature information includes: The image is subjected to semantic feature extraction using a semantic feature extraction network to obtain visual semantic feature information; The image is processed by a syntactic feature extraction network to extract syntactic features and obtain visual syntactic feature information. The image is processed by a salient object detection network to extract salient object features and obtain salient object feature information. The visual feature information includes the visual semantic feature information, the visual syntactic feature information, and the salient object feature information.

3. The video detection method according to claim 1, characterized in that, The step of extracting audio features from the speech of the video to be processed to obtain audio feature information includes: The speech features are extracted using a speech feature extraction network to obtain speech feature information. Frequency domain feature information is obtained by extracting frequency domain features from the speech feature information through a multi-frequency domain reconstruction network. The audio feature information is obtained by extracting the frequency domain relationship features of the frequency domain feature information through a multi-frequency domain relationship learning network.

4. The video detection method according to claim 3, characterized in that, The step of determining whether the video to be processed is a safe video based on the target depth hash code includes: Obtain a pre-stored blacklist, which includes multiple preset deep hash codes, and the videos corresponding to the preset deep hash codes are unsafe videos; If the blacklist contains a preset depth hash code whose Hamming distance to the target depth hash code is less than or equal to a preset threshold, then the video to be processed is determined to be an insecure video; if the blacklist does not contain a preset depth hash code whose Hamming distance to the target depth hash code is less than or equal to a preset threshold, then the video to be processed is determined to be a secure video.

5. A video detection device, characterized in that, include: The acquisition module is used to acquire the video to be processed; The processing module is used to extract visual features from the images of the video to be processed to obtain visual feature information; The processing module is also used to extract audio features from the speech of the video to be processed to obtain audio feature information; The processing module is also used to determine whether the video to be processed is a safe video based on the visual feature information and the audio feature information; The step of determining whether the video to be processed is a safe video based on the visual feature information and the audio feature information includes: Based on the visual feature information and the audio feature information, the target depth hash code of the video to be processed is determined; Based on the target depth hash code, determine whether the video to be processed is a safe video; The step of determining the target depth hash code of the video to be processed based on the visual feature information and the audio feature information includes: The visual and audio feature information are processed by a graph convolutional neural network to obtain visual semantic features including object relationships, visual syntactic features including object relationships, and auditory features including object relationships. The graph convolutional neural network performs graph modeling on the input features in the following way: each feature of each object in the image is treated as a node, multi-source features of the same object are connected pairwise, and corresponding features between adjacent objects are connected pairwise. Multi-source feature fusion processing is performed on the visual semantic features including object relationships, the visual syntactic features including object relationships, and the auditory features including object relationships to obtain attention index information and word vector information; The visual semantic features and syntactic feature extraction network included in the visual feature information, as well as the audio feature information, are subjected to multi-source feature fusion processing to obtain attention weight information; The attention index information, word vector information, and attention weight information are processed through an encoding / decoding network to obtain the encoding / decoding result of the video to be processed. The encoding / decoding network includes a feature encoding module, a self-attention module, and a feature simplification module. The feature encoding module generates a Q-vector based on the multi-source features output by the graph convolutional neural network, and generates K-vectors and V-vectors based on the multi-source features of salient objects. The self-attention module calculates an attention matrix based on the Q-vector and the K-vector, and performs a self-attention operation between the attention matrix and the V-vector to obtain a feature map. The feature simplification module performs dimensionality reduction processing on the feature map to obtain the encoding / decoding result. The target depth hash code is obtained by hashing the encoding and decoding results through a hash encoding network.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the video detection method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the video detection method as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the video detection method as described in any one of claims 1 to 4.

Images