Video detection method, device, equipment and computer-readable storage medium

Abstract

The present invention discloses a video detection method, apparatus, device, and computer-readable storage medium. The video detection method includes: obtaining an audio signal of a video to be tested, determining an audio depth feature of the audio signal, and detecting whether the audio signal is a pornographic audio signal; if so, determining a target video in the video to be tested that corresponds to the audio signal, and determining a visual depth feature of the target video; constructing a fusion feature matrix based on the audio depth feature and the visual depth feature; if the video to be tested is determined to contain pornographic content based on the fusion feature matrix, determining the target pornographic video content in the video to be tested. The present invention achieves efficient and accurate detection of pornographic videos.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a video detection method, device, equipment and computer-readable storage medium. Background Art

[0002] The rapid development of computer and internet technologies has facilitated information exchange between people. Online video is an important and widespread means of information exchange. Online videos primarily include news videos, film and television productions, live streams, and self-media videos. Due to the diverse nature of these videos, as well as the hidden nature of their content, online videos often become a vehicle for disseminating pornographic content, negatively impacting the physical and mental health of the public, especially young people. Therefore, detecting pornographic content in videos and preventing its spread online is crucial for purifying the online environment and maintaining social stability. The current mainstream video detection method involves extracting frames from videos to obtain images, then testing these images for pornographic content to determine if the video is legal. However, this method has the drawback that long videos contain a vast number of frames, and even extracting frames still requires processing a significant number of images, resulting in low efficiency. Furthermore, images only provide visual information, often misinterpreting images with large exposed areas as pornographic, while images with minimal clothing and excessive skin exposure often result in false detections, thus blocking legitimate videos. Therefore, efficient and accurate detection of pornographic videos is an urgent need. Summary of the Invention

[0003] The main purpose of the present invention is to provide a video detection method, device, equipment and computer-readable storage medium, aiming to solve the technical problem of how to efficiently and accurately detect pornographic videos.

[0004] To achieve the above object, the present invention provides a video detection method, comprising the following steps:

[0005] Acquire an audio signal of a video to be tested, determine an audio depth feature of the audio signal, and detect whether the audio signal is a pornographic audio signal;

[0006] If so, determining a target video corresponding to the audio signal in the video to be tested, and determining a visual depth feature of the target video;

[0007] Constructing a fusion feature matrix based on the audio depth feature and the visual depth feature;

[0008] If it is determined according to the fusion feature matrix that the video to be tested is a video containing pornographic content, then target pornographic content in the video to be tested is determined.

[0009] Optionally, if the video to be tested is determined to contain pornographic content according to the fusion feature matrix, the step of determining a target pornographic video segment in the video to be tested includes:

[0010] Determining whether there are pornographic features in the target video according to the fusion feature matrix;

[0011] If the target video contains pornographic features, the video to be tested is determined to be a pornographic video, and the target video is divided into multiple video segments to be tested with equal length, and the target pornographic video content in the video to be tested is determined based on each of the video segments to be tested.

[0012] Optionally, the step of determining target pornographic video content in the video to be tested based on each of the video clips to be tested includes:

[0013] Traversing each of the video segments to be tested, and determining whether the pornographic feature exists in the traversed video segments to be tested based on the fusion feature matrix;

[0014] If the pornographic feature exists in the traversed video segment to be tested, the time period corresponding to the traversed video segment to be tested is determined, and the video content in the video to be tested that matches the time period is used as the target pornographic video content.

[0015] Optionally, after the step of using the video content matching the time period in the video to be tested as the target pornographic video content, the method further includes:

[0016] Determining adjacent videos adjacent to the target video in the video to be tested, and determining whether there is a video segment with the pornographic feature in the adjacent videos;

[0017] If there is a video segment with the pornographic feature in the adjacent video, the video content corresponding to the video segment with the pornographic feature in the video to be tested is used as the target pornographic video content.

[0018] Optionally, the step of constructing a fusion feature matrix according to the audio depth feature and the visual depth feature includes:

[0019] Performing long short-term memory network training on the audio depth features to obtain audio training features;

[0020] Performing the long short-term memory network training on the visual depth feature to obtain a visual training feature;

[0021] Performing feature extraction and fusion on the audio training features and the visual training features to obtain preliminary fusion features;

[0022] Constructing an audio fusion feature based on the preliminary fusion feature and the audio training feature, and constructing a visual fusion feature based on the preliminary fusion feature and the visual training feature;

[0023] A fusion feature matrix is ​​constructed according to the audio fusion features and the visual training features.

[0024] Optionally, the step of determining the visual depth feature of the target video includes:

[0025] Calculating the visual attention weight of the target video according to the audio depth feature, and extracting the initial visual depth feature of the target video;

[0026] A visual depth feature is calculated according to the visual attention weight and the initial visual depth feature.

[0027] Optionally, the step of determining the audio depth feature of the audio signal includes:

[0028] The audio signal is converted into an audio spectrogram, depth information of the audio spectrogram is determined, and audio depth features in the audio spectrogram are extracted according to the depth information.

[0029] In addition, to achieve the above-mentioned purpose, the present invention further provides a video detection device, comprising:

[0030] an acquisition unit, configured to acquire an audio signal of a video to be tested, determine an audio depth feature of the audio signal, and detect whether the audio signal is a pornographic audio signal;

[0031] a determining unit, configured to, if yes, determine a target video corresponding to the audio signal in the video to be tested, and determine a visual depth feature of the target video;

[0032] A construction unit, configured to construct a fusion feature matrix based on the audio depth feature and the visual depth feature;

[0033] The detection unit is configured to determine target pornographic video content in the video to be tested if it is determined according to the fusion feature matrix that the video to be tested is a pornographic video.

[0034] In addition, to achieve the above-mentioned purpose, the present invention also provides a video detection device, which includes a memory, a processor, and a video detection program stored in the memory and runnable on the processor. When the video detection program is executed by the processor, the steps of the video detection method as described above are implemented.

[0035] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a video detection program is stored. When the video detection program is executed by a processor, the steps of the above-mentioned video detection method are implemented.

[0036] The present invention determines the visual depth features of the target video corresponding to the audio signal in the video to be tested only when the audio signal of the video to be tested is a pornographic audio signal, thereby avoiding the waste of computing resources caused by directly extracting frames from the video, and improving the efficiency of detecting pornographic videos. In addition, a fusion feature matrix is ​​constructed based on the audio depth features and visual depth features of the audio signal, and when the video to be tested is determined to be a pornographic video based on the fusion feature matrix, the specific target pornographic video content is determined, so that the pornographic content of the video can be detected and qualitatively analyzed by combining vision and audio, avoiding the misjudgment of situations such as excessive visual exposure area or similar sounds such as a baby crying, thereby improving the accuracy and efficiency of detecting pornographic videos. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 Schematic diagram of the terminal / device structure of the hardware operating environment involved in the embodiment of the present invention;

[0038] Figure 2 This is a flow chart of a first embodiment of a video detection method according to the present invention;

[0039] Figure 3 Schematic diagram of the device unit of the video detection device of the present invention;

[0040] Figure 4 Schematic diagram of the construction process of the fusion feature matrix in the video detection method of the present invention;

[0041] Figure 5 Schematic diagram of the pornographic video detection process in the video detection method of the present invention;

[0042] Figure 6 Schematic diagram of the feature fusion process in the video detection method of the present invention.

[0043] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings and in conjunction with the embodiments. DETAILED DESCRIPTION

[0044] It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] like Figure 1 As shown, Figure 1 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present invention.

[0046] The terminal in the embodiment of the present invention is a video detection device.

[0047] like Figure 1 As shown, the terminal may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0048] Optionally, the terminal may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. Among them, the sensors include light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor may turn off the display screen and / or backlight when the terminal device is moved to the ear. Of course, the terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, etc., which will not be repeated here.

[0049] Those skilled in the art will understand that Figure 1 The terminal structure shown in the figure does not constitute a limitation to the terminal, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0050] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a video detection program.

[0051] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the video detection program stored in the memory 1005 and perform the following operations:

[0052] Reference Figure 2The present invention provides a video detection method. In a first embodiment of the video detection method, the video detection method includes the following steps:

[0053] Step S10, obtaining an audio signal of the video to be tested, determining an audio depth feature of the audio signal, and detecting whether the audio signal is a pornographic audio signal;

[0054] With the development of internet technology, various video websites and live streaming platforms have emerged, generating a massive amount of video files. This includes unhealthy content, such as pornographic videos, which pollute the online environment and impact viewers' physical and mental health. Therefore, detecting and blocking inappropriate video content online is crucial. Existing methods often suffer from the following issues: First, single-modality detection methods have limited capabilities. Whether based on audio or visual detection, they often fail to address similar scenarios, which can easily lead to false positives. Second, regarding video frame extraction, a high frame extraction frequency requires more frames to detect a video, resulting in low detection efficiency. A low frame extraction frequency results in a longer time interval between frames, inevitably missing content between frames and potentially misidentifying pornographic clips. Third, when pornographic videos are detected, there is no effective method for removing the pornographic content; instead, the entire video is directly blocked. This can result in a long video being blocked entirely due to a small pornographic clip, causing legitimate content to be incorrectly blocked, impacting the user experience.

[0055] Therefore, in this embodiment, the amount of calculation is reduced by using audio clips for classification instead of video frame extraction. Since the audio feature dimension is smaller than the video feature dimension, the speed of locating pornographic clips through audio is faster than the frame extraction speed, thereby reducing the pornographic detection time for the entire video. In addition, by using visual and audio bimodal information to determine whether a video is pornographic, the video features are enriched through the complementary information between different modalities, reducing misjudgments caused by the limitations of single-modal information and reducing erroneous interception. In addition, by using multi-task learning, a time consistency task is added to the video classification task, and the video clips are qualitatively characterized at a finer granularity on the timeline to obtain the starting time information of the pornographic video clips, and the pornographic clips are filtered and deleted.

[0056] Therefore, in this embodiment, when performing pornographic detection on the video to be tested, the audio signal of the video to be tested can be obtained first, and the audio signal can be converted into the corresponding MFCC (Mel-Frequency Cepstral Coefficients) spectrogram, that is, the one-dimensional signal is converted into a two-dimensional image, and the converted MFCC spectrogram can be used as an audio spectrogram, wherein the dimension of the audio spectrogram can be [128,96]. The VGGish network is then delayed to directly extract the depth information of the audio spectrogram to obtain the audio depth feature and its vector Fa. Among them, the method of obtaining the audio signal of the video to be tested can be extracted using an audio extraction tool, and the specific method is not limited here. And the audio signal of a preset length in the video to be tested can be extracted according to a preset time interval. Each audio signal is detected in turn.

[0057] Moreover, when performing audio classification, it is necessary to first train the audio pornography classification network in advance, and then detect whether the audio signal is a pornography audio signal based on the trained audio pornography classification network. Among them, the types of audio signals can include pornography audio signals and normal audio signals. Pornography audio signals can be audio signals with sensitive information such as moaning and gasping. Normal audio signals can be audio signals that are not similar to pornography audio signals in natural scenes, such as audio signals emitted by car horns. Among them, when training the audio pornography classification network, the network backbone can select (not limited to) Resnet (residual network) and other backbone networks, and the input of the network is the MFCC spectrum of the audio signal. The audio signal is intercepted in units of 1s, and the 1s audio signal is converted into an MFCC spectrum as the input of the network. Audio signals of different categories are input into the network to train the softmax classifier.

[0058] After obtaining the audio deep features, the trained audio pornography classification network can be directly used to classify and detect the audio signal to detect whether the audio signal is a pornographic audio signal, and perform different operations based on different detection results.

[0059] Specifically, the step of obtaining the audio signal of the video to be tested includes:

[0060] Extracting an audio signal of a preset duration from the video to be tested according to a preset time interval;

[0061] After the step of detecting whether the audio signal is a pornographic audio signal, the method includes:

[0062] If the audio signal is a normal audio signal, the step of extracting the audio signal of the preset duration from the video to be tested according to the preset time interval is continued.

[0063] Step S20: If yes, determine the target video corresponding to the audio signal in the video to be tested, and determine the visual depth feature of the target video;

[0064] When it is determined that the audio signal is a pornographic audio signal, the video segment corresponding to the same time axis can be located in the video to be tested based on the audio signal, and it can be used as the target video. Specifically, the time sequence of the target video in the video to be tested is the same as the time sequence of the audio signal in the video to be tested. The time sequence is the time period when the segment corresponding to the audio signal or the target video is in the playback time axis of the video to be tested. For example, the time period corresponding to the audio signal in the playback time axis of the video to be tested is from the 100th second to the 200th second, then its time sequence is from the 100th second to the 200th second, and the time sequence corresponding to the target video with the same time sequence is also from the 100th second to the 200th second in the video to be tested.

[0065] After determining and acquiring the target video, the target video can be frame-drawn at Nfps to obtain a sequence of images. The visual depth features initially acquired are obtained through the visual depth network. The target video can be acquired using a video editing tool, and the specific acquisition method is not limited here. In order to further ensure the accuracy of the acquired visual depth features, the weight vector of the visual attention mechanism can be calculated using the audio depth features. The weight vector is then multiplied by the vector of the initially acquired visual depth features to obtain visual depth information based on the audio attention mechanism, i.e., the visual depth features of the target video.

[0066] Step S30, constructing a fusion feature matrix based on the audio depth feature and the visual depth feature;

[0067] Step S40: If it is determined according to the fusion feature matrix that the video to be tested is a video involving pornography, then target pornography-related video content in the video to be tested is determined.

[0068] After the audio depth features and visual depth features are determined, the audio depth features and the visual depth features based on the attention mechanism can be fused to construct a fusion feature matrix. Figure 4 As shown, and They are the output results of audio depth feature Fa and visual depth feature AFv after LSTM (Long Short-Term Memory Network). and They are the audio depth feature Fa and the visual depth feature AFv after the long short-term memory network. The feature fusion network is similar to the residual structure. The visual features and sound features are divided into two paths. One path is extracted through the linear layer + hyperbolic tangent tanh layer + linear layer, and then added with the other modality to form a preliminary modal fusion complementary bimodal feature. The other path retains the features of its own modality, adds them to the preliminary fusion features and then passes through the activation layer tanh. and The features of contain both complementary features between modalities and retain the unique features of its own modality. and Splicing or adding together to obtain the final bimodal fusion feature matrix H. And the bimodal fusion feature matrix H is used as the fusion feature matrix.

[0069] Then, a multi-task learning approach is employed, where the fused feature matrix is ​​subjected to both the video classification task and the temporal consistency task, yielding classification and temporal consistency results. The classification result determines whether the video under test is pornographic. If the video is not pornographic, the temporal consistency result is not considered. If the video is pornographic, the temporal consistency result is examined to determine the fine-grained pornographic time range of the video under test. The video content within this pornographic time range is then targeted for deletion.

[0070] Moreover, when it is determined that the video to be tested is pornographic, the same test can be performed on the videos before and after the video to be tested according to the continuity of the video plot to determine whether the videos before and after are pornographic. If so, the pornographic content in the videos before and after will also be deleted.

[0071] In addition, to assist in understanding the dual-modality detection and interception method of pornographic videos based on vision and sound in this embodiment, an example is given below.

[0072] For example, Figure 5As shown, if there is a video that needs to be tested for pornographic content, the video can be used as the video to be tested, and the audio information of the video, such as the audio signal, can be obtained. The audio segment where the audio signal is located is determined based on the audio information, and the audio depth features in the audio segment are extracted based on the audio network. The audio classification is performed based on the audio depth features. If the audio signal is not pornographic, a new audio segment and a new audio signal are obtained, and the audio classification judgment step is continued. If the audio signal is pornographic, a multimodal fusion network is constructed based on the audio depth features and the visual depth features, that is, a fusion feature matrix. Among them, the video segment corresponding to the same time axis is located according to the audio signal in the video, and the frame is extracted to obtain a picture sequence. The visual depth features obtained for the first time are obtained through the visual depth network, and then the weight vector of the visual attention (attention mechanism) is calculated based on the audio depth features. The weight vector and the vector of the visual depth features obtained for the first time are multiplied to obtain the visual depth information based on the audio attention mechanism, that is, the final visual depth features. Then, a multimodal fusion network is constructed based on the final visual depth features and audio depth features, and time consistency and video classification tasks are performed to determine whether the video is pornographic. If so, the pornographic content in the specific time period of the video is determined and deleted. Specifically, Figure 6 As shown in the figure, the Attention calculation is performed through the audio depth feature Fa to obtain the weight, and the weight is vector multiplied with the initially obtained visual depth feature Fv (i.e., the initial visual depth feature) to obtain the visual depth feature AFv. The LSTM model training is performed on the audio depth feature Fa and the visual depth feature AFv at the same time to complete the feature fusion. The result of the feature fusion is used to determine whether the video to be tested is a pornographic video.

[0073] In this embodiment, the visual depth features of the target video corresponding to the audio signal in the video to be tested are determined only when the audio signal of the video to be tested is a pornographic audio signal, thereby avoiding the waste of computing resources caused by directly extracting frames from the video, and improving the efficiency of detecting pornographic videos. In addition, a fusion feature matrix is ​​constructed based on the audio depth features and visual depth features of the audio signal, and when the video to be tested is determined to be a pornographic video based on the fusion feature matrix, the specific target pornographic video content is determined, so that the pornographic content of the video can be detected and characterized by combining vision and audio, avoiding the misjudgment of situations such as excessively large visually exposed areas or similar sounds such as a baby crying, thereby improving the accuracy and efficiency of detecting pornographic videos.

[0074] Furthermore, based on the first embodiment of the present invention, a second embodiment of the video detection method of the present invention is proposed. In this embodiment, in step S20 of the above embodiment, if it is determined that the video to be tested is a video containing pornographic content according to the fusion feature matrix, the refinement of the step of determining a target pornographic video segment in the video to be tested includes:

[0075] Step a, determining whether the target video contains pornographic features based on the fusion feature matrix;

[0076] In this embodiment, after the fusion feature matrix is ​​determined, the time consistency task and the video classification task can be performed in parallel, or the video classification task can be performed first and then the time consistency task. Moreover, when performing the video classification task, the final fused features in the fusion feature matrix are determined, and the final fused features are input into the video classification module, wherein the video classification module performs an overall classification on a section of video. Since the acquired audio signal can be a part of the audio signal in the video to be tested, it is possible to directly determine whether the target video is pornographic based on the video classification module. That is, the target video is classified and labeled by the video classification module. If it is labeled as 0, it is determined that the target video is not pornographic and there are no pornographic features. If it is labeled as 1, it is determined that the target video is pornographic and there are pornographic features (such as pictures, text and sounds with sensitive information). Among them, the video classification module can classify and judge the target video through the classification loss function loss_c. Among them, loss_c=-(y c log(x c )+(1-y c )log(1-x c )).

[0077] And in the video classification module, for a batch of input, the classification label y c ∈[batch,2]. And the feature matrix x corresponding to the fusion feature matrix c ∈[batch, 2].

[0078] Step b: if the target video contains pornographic features, the video to be tested is determined to be a pornographic video, and the target video is divided into multiple video segments to be tested with equal duration, and the target pornographic video content in the video to be tested is determined based on each of the video segments to be tested.

[0079] When it is determined that there are pornographic features in the target video, it can be determined that the target video as a whole is pornographic, that is, the video to be tested is determined to be a video containing pornographic features, but it is impossible to determine whether the local time interval in the target video is pornographic. Therefore, a time consistency task can be performed, that is, the total duration T of the target video can be determined first, and the target video can be divided into multiple video clip sequences with equal duration according to the total duration, and the video clip corresponding to each video clip sequence can be used as the video clip to be tested. For example, the target video can be divided into n video clip sequences with equal duration t V=[V0,V1,...V n ]. Then, it is determined whether there are pornographic video clips in each of the video clips to be tested, and the content of the pornographic video clips is used as the target pornographic video content.

[0080] In this embodiment, when it is determined that there are pornographic features in the target video based on the fusion feature matrix, the video to be tested is determined to be a video containing pornographic content, thereby achieving dual audio and visual determination, improving the accuracy of detecting that the video to be tested is a video containing pornographic content, and dividing the target video into multiple video segments to be tested with equal length, and then determining the target pornographic video content therefrom, thereby ensuring the accuracy of determining the target pornographic video content.

[0081] Specifically, the step of determining the target pornographic video content in the video to be tested based on each of the video segments to be tested includes:

[0082] Step c, traversing each of the video segments to be tested, and determining whether the pornographic feature exists in the traversed video segments to be tested based on the fusion feature matrix;

[0083] Step d: If the pornographic feature exists in the traversed video segment to be tested, determine the time period corresponding to the traversed video segment to be tested, and use the video content in the video to be tested that matches the time period as the target pornographic video content.

[0084] In this embodiment, each video segment to be tested can be traversed, and then the fusion feature matrix can be used to determine in turn whether there are pornographic features in each traversed video segment to be tested. If there are pornographic features in the traversed video segment to be tested, it is determined that the traversed video segment to be tested at this time is a video segment containing pornographic features. Among them, when both the visual and the sound are pornographic, it is marked as pornographic. That is, the audio and visual corresponding to the traversed video segment to be tested are both pornographic. At this time, it can be regarded as a pornographic video segment, and the time period corresponding to the pornographic video segment can be determined, and the video content in the video to be tested that matches the time period can be directly used as the target pornographic video. Among them, when detecting whether the traversed video segment to be tested is pornographic, it can be determined by calculating the time consistency loss function loss_t. Among them, loss_t=-(y t log(x t)+(1-y t )log(1-x t )).

[0085] Among them, for a batch input, its time consistency label is y t ∈[batch,n,2]. Compared with the overall classification, the temporal consistency branch contains more local information of the video, and its output feature matrix x t ∈[batch,n,2]. n is the number of the video clip to be tested.

[0086] In this embodiment, by traversing each video segment to be tested, and when it is determined that there are pornographic features in the traversed video segment to be tested, the video content in the video to be tested that matches the time period of the traversed video segment to be tested is used as the target pornographic video content, thereby ensuring the accuracy of determining the target pornographic video content.

[0087] Furthermore, after the step of using the video content matching the time period in the video to be tested as the target pornographic video content, the method further includes:

[0088] Step e, determining adjacent videos adjacent to the target video in the video to be tested, and determining whether there is a video clip with the pornographic feature in the adjacent videos;

[0089] In this embodiment, since the video to be tested may not only contain the target video, it is also necessary to detect other videos in the video to be tested in addition to the target video, so that adjacent videos adjacent to the target video in the video to be tested can be determined. The duration of the adjacent videos can be determined based on the total duration of the video to be tested and the duration of the target video. The number of adjacent videos can also be determined based on the time node position of the target video in the video to be tested.

[0090] The same method needs to be used to detect whether there are video clips with pornographic characteristics in adjacent videos. That is, audio and visual pornographic detection is also required to determine whether there are video clips with pornographic characteristics in adjacent videos.

[0091] Step f: If there is a video clip with the pornographic feature in the adjacent video, the video content corresponding to the video clip with the pornographic feature in the video to be tested is used as the target pornographic video content.

[0092] If it is found through judgment that there are no video clips with pornographic features in the adjacent videos, and the video to be tested has been fully tested, the pornographic video content detected in the target video can be directly used as the target pornographic video content of the video to be tested. However, if there are video clips with pornographic features in the adjacent videos, all video clips with pornographic features in the adjacent videos and the target video can be used as the final pornographic video clips, and the video content in the video to be tested that corresponds to the final pornographic video clip (i.e., the video content in the same time sequence as the final pornographic video clip) can be used as the target pornographic video content.

[0093] In this embodiment, by detecting adjacent videos adjacent to the target video, and when it is determined that there are video clips with pornographic features in the adjacent videos, the corresponding video content is used as the target pornographic video content, thereby ensuring the accuracy of determining the target pornographic video content.

[0094] Furthermore, the step of constructing a fusion feature matrix based on the audio depth feature and the visual depth feature includes:

[0095] Step g, performing long short-term memory network training on the audio depth features to obtain audio training features;

[0096] Step h, performing the long short-term memory network training on the visual depth feature to obtain a visual training feature;

[0097] Step i, performing feature extraction and fusion on the audio training features and the visual training features to obtain preliminary fusion features;

[0098] Step j, constructing an audio fusion feature based on the preliminary fusion feature and the audio training feature, and constructing a visual fusion feature based on the preliminary fusion feature and the visual training feature;

[0099] Step k: constructing a fusion feature matrix based on the audio fusion features and the visual training features.

[0100] In this embodiment, when constructing the fusion feature matrix, it is necessary to first convert the audio depth features and the visual depth features, that is, to obtain the sound features corresponding to the audio depth features through long short-term memory network training, that is, the audio training features. Visual features corresponding to visual depth features, i.e. visual training features

[0101] Then the audio training features are extracted through the linear layer, hyperbolic tangent layer and linear layer. At the same time, the visual training features are extracted through the linear layer, hyperbolic tangent layer and linear layer. The features extracted twice are fused to obtain the preliminary fusion features. The preliminary fusion features and the audio training features are then fused and added, and the audio fusion features are obtained after the activation layer tanh. At the same time, the preliminary fusion features and visual training features are fused and added, and then the visual fusion features are obtained by activating the tanh layer. Audio fusion features and visual fusion features Perform concatenation or addition to obtain a fusion feature matrix.

[0102] In this embodiment, the audio training features corresponding to the audio depth features and the visual training features corresponding to the visual depth features are extracted and fused to obtain preliminary fusion features, and then audio fusion features and visual fusion features are constructed based on the preliminary fusion features, and a fusion feature matrix is ​​constructed based on this, thereby ensuring the accuracy and effectiveness of the fusion feature matrix.

[0103] Furthermore, the step of determining the visual depth feature of the target video includes:

[0104] Step x, calculating the visual attention weight of the target video according to the audio depth feature, and extracting the initial visual depth feature of the target video;

[0105] Step y: calculating a visual depth feature based on the visual attention weight and the initial visual depth feature.

[0106] In this embodiment, the target video is extracted to obtain a picture sequence, and the visual depth feature is obtained by the visual depth network, and it is used as the initial visual depth feature. In addition, in order to ensure the accuracy of the final visual depth feature, the visual attention weight of the target video can be calculated based on the audio depth feature technology, and the visual depth feature based on the attention mechanism can be calculated based on the visual attention weight and the initial visual depth feature. And the attention mechanism is essentially a set of weights. For the visual feature v at time t t (i.e., initial visual depth features) and audio features a t (i.e. audio depth features), first pass through a nonlinear function (generally using the ReLU function) U v and U a (v stands for visual, a stands for audio), and the results are then multiplied by the mapping matrix W v and W a , where L Tis a vector of all 1s to ensure the same dimensions after multiplication, σ is the hyperbolic tangent function,

[0107]

[0108]

[0109]

[0110] Among them, AFv is the visual depth feature based on the attention mechanism.

[0111] In this embodiment, the visual attention weight is calculated according to the audio depth feature, and the visual depth feature is calculated according to the visual attention weight and the initial visual depth feature, thereby ensuring the accuracy and effectiveness of determining the visual depth feature.

[0112] Furthermore, the step of determining the audio depth feature of the audio signal includes:

[0113] Step z: convert the audio signal into an audio spectrogram, determine the depth information of the audio spectrogram, and extract audio depth features in the audio spectrogram based on the depth information.

[0114] In this embodiment, the acquired audio signal can be converted into a corresponding MFCC (Mel-Frequency Cepstral Coefficients) spectrogram, i.e., an audio spectrogram. A specific method can be to pre-emphasize the audio signal and then frame and window it, perform Fourier transform on the windowed signal to obtain a logarithmic spectrum, apply Mel filtering to the logarithmic spectrum to obtain a Mel frequency spectrum, take the absolute value or square logarithm of the output of the filter bank to obtain its energy, and perform discrete cosine transform to obtain the MFCC spectrogram.

[0115] The VGGish network is used to directly extract a 128-dimensional deep audio feature vector from the audio spectrogram based on the audio spectrogram depth information, and it is used as the audio depth feature.

[0116] In this embodiment, the audio signal is converted into an audio spectrogram, and the audio depth features in the audio spectrogram are extracted according to the depth information of the audio spectrogram, thereby ensuring the accuracy and effectiveness of the acquired audio depth features.

[0117] In addition, refer to Figure 3 , an embodiment of the present invention further provides a video detection device, comprising:

[0118] An acquisition unit A10 is configured to acquire an audio signal of a video to be tested, determine an audio depth feature of the audio signal, and detect whether the audio signal is a pornographic audio signal;

[0119] a determining unit A20, configured to, if yes, determine a target video corresponding to the audio signal in the video to be tested, and determine a visual depth feature of the target video;

[0120] A construction unit A30, configured to construct a fusion feature matrix based on the audio depth feature and the visual depth feature;

[0121] The detection unit A40 is configured to determine target pornographic video content in the video to be tested if it is determined according to the fusion feature matrix that the video to be tested is a pornographic video.

[0122] Optionally, the detection unit A40 is configured to:

[0123] Determining whether there are pornographic features in the target video according to the fusion feature matrix;

[0124] If the target video contains pornographic features, the video to be tested is determined to be a pornographic video, and the target video is divided into multiple video segments to be tested with equal length, and the target pornographic video content in the video to be tested is determined based on each of the video segments to be tested.

[0125] Optionally, the detection unit A40 is configured to:

[0126] Traversing each of the video segments to be tested, and determining whether the pornographic feature exists in the traversed video segments to be tested based on the fusion feature matrix;

[0127] If the pornographic feature exists in the traversed video segment to be tested, the time period corresponding to the traversed video segment to be tested is determined, and the video content in the video to be tested that matches the time period is used as the target pornographic video content.

[0128] Optionally, the detection unit A40 is configured to:

[0129] Determining adjacent videos adjacent to the target video in the video to be tested, and determining whether there is a video segment with the pornographic feature in the adjacent videos;

[0130] If there is a video segment with the pornographic feature in the adjacent video, the video content corresponding to the video segment with the pornographic feature in the video to be tested is used as the target pornographic video content.

[0131] Optionally, construct unit A30 for:

[0132] Performing long short-term memory network training on the audio depth features to obtain audio training features;

[0133] Performing the long short-term memory network training on the visual depth feature to obtain a visual training feature;

[0134] Performing feature extraction and fusion on the audio training features and the visual training features to obtain preliminary fusion features;

[0135] Constructing an audio fusion feature based on the preliminary fusion feature and the audio training feature, and constructing a visual fusion feature based on the preliminary fusion feature and the visual training feature;

[0136] A fusion feature matrix is ​​constructed according to the audio fusion features and the visual training features.

[0137] Optionally, the determination unit A20 is configured to:

[0138] Calculating the visual attention weight of the target video according to the audio depth feature, and extracting the initial visual depth feature of the target video;

[0139] A visual depth feature is calculated according to the visual attention weight and the initial visual depth feature.

[0140] Optionally, the acquisition unit A10 is configured to:

[0141] The audio signal is converted into an audio spectrogram, depth information of the audio spectrogram is determined, and audio depth features in the audio spectrogram are extracted according to the depth information.

[0142] The steps implemented by the functional units of the video detection device may refer to the various embodiments of the video detection method of the present invention, and will not be described in detail here.

[0143] In addition, the present invention also provides a video detection device, which includes: a memory, a processor, and a video detection program stored in the memory; the processor is used to execute the video detection program to implement the steps of each embodiment of the above-mentioned video detection method.

[0144] The present invention also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can also be executed by one or more processors to implement the steps of each embodiment of the above-mentioned video detection method.

[0145] The specific implementation of the computer-readable storage medium of the present invention is basically the same as the embodiments of the above-mentioned video detection method, and will not be repeated here.

[0146] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0147] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0148] Through the description of the above embodiments, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0149] The above are only preferred embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention description and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A video detection method, characterized in that: The video detection method comprises the following steps: Acquire an audio signal of a video to be tested, determine an audio depth feature of the audio signal, and detect whether the audio signal is a pornographic audio signal; If so, determining a target video corresponding to the audio signal in the video to be tested, and determining a visual depth feature of the target video; Constructing a fusion feature matrix based on the audio depth feature and the visual depth feature; If it is determined according to the fusion feature matrix that the video to be tested is a video containing pornographic content, then determining target pornographic content in the video to be tested; Wherein, if it is determined according to the fusion feature matrix that the video to be tested is a video containing pornographic content, the step of determining the target pornographic content in the video to be tested includes: After determining the fusion feature matrix, multi-task learning is used to simultaneously perform the temporal consistency task and the video classification task; When performing a video classification task, determining whether the target video contains pornographic features based on the fusion feature matrix; When performing the temporal consistency task, the target video is divided into a plurality of video clip sequences of equal length according to the total length of the target video, and each video clip sequence is used as a video clip to be tested, and each of the video clips to be tested is traversed, and it is determined whether the pornographic feature exists in the traversed video clips to be tested based on the fusion feature matrix; If the target video contains pornographic features and the traversed video segments to be tested contain the pornographic features, the video content in the video to be tested that matches the time period of the traversed video segments to be tested is used as the target pornographic video content; wherein, determining adjacent videos adjacent to the target video in the video to be tested, and determining whether there is a video segment with the pornographic feature in the adjacent videos; If there is a video segment with the pornographic feature in the adjacent video, the video content corresponding to the video segment with the pornographic feature in the video to be tested is used as the target pornographic video content.

2. The video detection method according to claim 1, wherein: The step of constructing a fusion feature matrix according to the audio depth feature and the visual depth feature includes: Performing long short-term memory network training on the audio depth features to obtain audio training features; Performing the long short-term memory network training on the visual depth feature to obtain a visual training feature; Performing feature extraction and fusion on the audio training features and the visual training features to obtain preliminary fusion features; Constructing an audio fusion feature based on the preliminary fusion feature and the audio training feature, and constructing a visual fusion feature based on the preliminary fusion feature and the visual training feature; A fusion feature matrix is ​​constructed according to the audio fusion features and the visual training features.

3. The video detection method according to claim 1, wherein: The step of determining the visual depth feature of the target video includes: Calculating the visual attention weight of the target video according to the audio depth feature, and extracting the initial visual depth feature of the target video; A visual depth feature is calculated according to the visual attention weight and the initial visual depth feature.

4. The video detection method according to any one of claims 1 to 3, wherein: The step of determining the audio depth feature of the audio signal comprises: The audio signal is converted into an audio spectrogram, depth information of the audio spectrogram is determined, and audio depth features in the audio spectrogram are extracted according to the depth information.

5. A video detection device, characterized in that: The video detection device comprises: an acquisition unit, configured to acquire an audio signal of a video to be tested, determine an audio depth feature of the audio signal, and detect whether the audio signal is a pornographic audio signal; a determining unit, configured to, if yes, determine a target video corresponding to the audio signal in the video to be tested, and determine a visual depth feature of the target video; A construction unit, configured to construct a fusion feature matrix based on the audio depth feature and the visual depth feature; A detection unit for determining target pornographic video content in the video to be tested if it is determined according to the fusion feature matrix that the video to be tested is a video containing pornographic content, wherein if it is determined according to the fusion feature matrix that the video to be tested is a video containing pornographic content, then determining target pornographic video content in the video to be tested comprises: after determining the fusion feature matrix, simultaneously performing a time consistency task and a video classification task in a multi-task learning manner; when performing the video classification task, determining whether there are pornographic features in the target video according to the fusion feature matrix; when performing the time consistency task, dividing the target video into a plurality of video segment sequences of equal duration according to the total duration of the target video, and taking each video segment sequence as a video segment to be tested, traversing each of the video segments to be tested, and determining whether there are the pornographic features in the traversed video segments to be tested based on the fusion feature matrix; if there are pornographic features in the target video and the traversed video segments to be tested, taking the video content in the video to be tested that matches the time period of the traversed video segments to be tested as the target pornographic video content; Among them, adjacent videos adjacent to the target video in the video to be tested are determined, and whether there is a video clip with the pornographic characteristics in the adjacent videos; if there is a video clip with the pornographic characteristics in the adjacent videos, the video content in the video to be tested corresponding to the video clip with the pornographic characteristics is used as the target pornographic video content.

6. A video detection device, characterized in that: The video detection device includes: a memory, a processor, and a video detection program stored in the memory and executable on the processor. When the video detection program is executed by the processor, the steps of the video detection method according to any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a video detection program, which, when executed by a processor, implements the steps of the video detection method according to any one of claims 1 to 4.

Images