Video auditing method and device, equipment, storage medium and program product

Abstract

The invention relates to a video auditing method and device, equipment, a storage medium and a program product in the technical field of artificial intelligence. The method comprises the following steps: screening and marking obviously illegal videos according to key frames of videos and voice texts of the videos; inputting the key frame and the voice text of the unmarked video into a deep learning model, and outputting visual features of the key frame and semantic features of the voice text; obtaining a fusion feature according to the visual feature of the key frame and the semantic feature of the voice text; and according to the fusion feature, determining a video multi-level auditing result. According to the video auditing method provided by the embodiment of the invention, the auditing accuracy and fineness are improved, and the auditing efficiency is improved while the high reliability of video auditing is ensured.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, storage medium, and program product for video review. Background Technology

[0002] Video content is multimodal information, integrating images, audio, text, and other information formats. These different modalities complement each other, collectively conveying complex and diverse content. Video review refers to the process of manually or automatically checking videos before they are uploaded to online platforms to ensure that the content complies with the platform's community guidelines and laws and regulations. Reviewed content includes, but is not limited to, pornography, violence, hate speech, and copyright infringement. Video review not only needs to control the compliance of various video content but also needs to balance review efficiency and accuracy to meet the demands of the rapid dissemination of massive amounts of video content.

[0003] With the explosive growth of video content, traditional manual video review methods can no longer meet the demands for speed and efficiency. The development of artificial intelligence technology has brought new solutions to video content review. Existing technologies for video review mainly involve multimodal data processing, deep learning-based image recognition, and natural language processing. However, existing video review methods suffer from insufficient comprehensiveness, inadequate feature fusion and semantic mining, and low model computational efficiency, thus failing to guarantee the reliability and efficiency of video review. Summary of the Invention

[0004] To address the aforementioned technical issues, this disclosure provides a method, apparatus, device, storage medium, and program product for video review, which improves review efficiency while ensuring high reliability of video review.

[0005] A first aspect of this disclosure provides a video review method, the method comprising: Based on the keyframes and audio transcripts of the video, screen and mark videos that clearly violate regulations; Input the keyframes and audio text of the unlabeled video into the deep learning model, and output the visual features of the keyframes and the semantic features of the audio text. The fusion features are obtained based on the visual features of the keyframes and the semantic features of the speech text; The multi-level review result of the video is determined based on the fusion features.

[0006] A second aspect of this disclosure provides a video review apparatus, the apparatus comprising: The initial review module is configured to screen and mark obviously illegal videos based on keyframes and audio-visual transcripts of the video. The feature module is configured to take keyframes and speech text from unlabeled videos as input to a deep learning model and output the visual features of the keyframes and the semantic features of the speech text. The fusion module is configured to obtain fusion features based on the visual features of the keyframes and the semantic features of the speech text; The determination module is configured to determine the multi-level review result of the video based on the fusion features.

[0007] A third aspect of this disclosure provides an electronic device, including: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is used to execute the instructions to implement the above-described method.

[0008] A fourth aspect of this disclosure provides a computer-readable storage medium that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the methods described above.

[0009] A fifth aspect of this disclosure provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the video review method described above.

[0010] The at least one technical solution adopted in this disclosure can achieve the following beneficial effects: Firstly, rapid pre-filtering is performed based on keyframes of the video and audio text to mark obviously violating content, thereby significantly improving review efficiency and avoiding content backlog caused by slow manual review. Secondly, fine-grained analysis is performed on the keyframes and audio text of the video that have passed the initial screening, achieving feature fusion of visual features of video keyframes and semantic features of audio text for the goal of multimodal interaction and fusion. Finally, hierarchical review results are output based on the fused features. The video review method provided in this disclosure improves the accuracy and precision of the review, enhancing review efficiency while ensuring high reliability. Attached Figure Description

[0011] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0012] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A flowchart illustrating a video review method provided in this embodiment of the disclosure; Figure 2 A logical schematic diagram of a video review method provided in this embodiment of the disclosure; Figure 3 A flowchart illustrating yet another video review method provided in this disclosure embodiment; Figure 4 A flowchart illustrating a video keyframe extraction method based on information entropy provided in this embodiment of the disclosure; Figure 5 A schematic diagram of the structure of a video review device provided in an embodiment of this disclosure; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure; Figure 7 This is a schematic diagram of the structure of an exemplary computer system provided in an embodiment of the present disclosure. Detailed Implementation

[0014] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0015] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0016] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0017] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0018] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0019] The following is combined Figures 1-7This document describes the video review methods, apparatus, devices, storage media, and program products provided in the embodiments of this disclosure.

[0020] Figure 1 This is a schematic diagram of a video review method provided in an embodiment of the present disclosure, such as... Figure 1 As shown, a video review method includes: S101. Based on the keyframes and audio text of the video, screen and mark videos that clearly violate regulations.

[0021] By inputting keyframes and voice text into a multimodal large model to quickly pre-filter obviously illegal content and return preliminary judgment results, it is equivalent to setting up a fast pre-filtering mechanism at the first layer to replace manual screening, thereby greatly improving the efficiency of review and avoiding content backlog caused by slow manual review.

[0022] Step S101 includes: S1011. Input the key frames of the video and the audio text of the video into the multimodal model according to a preset format. The multimodal model combines cross-modal semantic association analysis with comparison with the violation feature library to generate a structured primary judgment result. S1012. Mark the videos that are deemed to be in violation of the initial assessment.

[0023] Prior to step S101, the video review method also includes: S100: Extract keyframes from the video and extract audio text from the video.

[0024] Keyframes in video are the core frames used to define animation effects, parameter changes, or content transitions in digital video. Keyframe extraction is central to video analysis and processing, aiming to select the most representative frames from a video sequence to efficiently represent the video content.

[0025] For example, extracting keyframes from a video includes: S1001. Obtain the information entropy value of each frame of the video and generate an entropy curve; The amount of information contained in each frame is quantified by the information entropy value. After grayscale processing of all frames, the information entropy value is calculated and concatenated to form an information entropy curve.

[0026] S1002. Identify feature points in the entropy curve that meet preset conditions; The feature points are local maxima of the entropy curve, obtained by screening using a sliding window method combined with threshold conditions of entropy statistical features. The entropy statistical features include the mean and standard deviation of the entropy sequence.

[0027] S1003. Determine the frame corresponding to the feature point as the key frame.

[0028] By accurately identifying local maxima in the entropy curve, and locating and selecting frames with significant information content as keyframes, we can precisely focus on key information in the video, significantly reduce redundant frame interference, effectively reduce data processing volume, and improve the efficiency of video content analysis.

[0029] Unlike existing technologies that extract keyframes based on sampling at fixed time intervals or based on gradient change detection, the embodiments of this disclosure use information entropy values ​​to quantify the amount of information contained in video frames and extract keyframes based on curve maxima. This can effectively filter out redundant information while extracting the key parts of the video.

[0030] For example, extracting audio text from a video includes: S1004. Extract audio data from the video; S1005. Input the audio data into a deep learning model that integrates global semantic understanding to extract the speech text of the video.

[0031] S102. Input the keyframes of the unlabeled video and the speech text into the deep learning model, and output the visual features of the keyframes and the semantic features of the speech text.

[0032] S103. Obtain fused features based on the visual features of the keyframes and the semantic features of the speech text.

[0033] Step S103 includes: S1031. Perform dimensional transformation and dimensional alignment on the visual features of the keyframe and the semantic features of the speech text through linear projection. This disclosure embodiment constructs a linear projection layer to map the visual features of video keyframes and the semantic features of speech text to the same semantic space.

[0034] S1032. Based on the self-attention mechanism, a fusion feature that incorporates global information is obtained.

[0035] Weight allocation based on attention mechanisms captures fine-grained correlation information between two modalities, which can solve the heterogeneity problem of cross-modal data and achieve deep interactive fusion.

[0036] Compared with existing methods such as simple weighted averaging, direct concatenation of feature vectors, and integration of multimodal data through a pre-set violation database, this disclosure provides a feature-level cross-modal fusion method. By combining linear projection and cross-attention mechanisms, it establishes a dynamic association between video image information and text information, effectively achieving deep fusion of visual and semantic features, and providing powerful cross-modal association capabilities for multimodal tasks.

[0037] S104. Determine the multi-level review result of the video based on the fusion features.

[0038] Step S104 includes: S1041. Obtain the probabilities corresponding to various violation types based on the fusion features; S1042. Based on the comparison between the probability corresponding to each type of violation and the threshold, determine the video review result.

[0039] This disclosure embodiment sets up a rapid pre-filtering mechanism in the first layer to replace manual initial screening. Keyframes from the video and audio text are input into a multimodal large-scale model for rapid pre-filtering of obviously violating content and the return of preliminary judgment results, thereby significantly improving review efficiency and avoiding content backlog caused by slow manual review. If the video is not captured or intercepted in the first-layer pre-filtering layer, the keyframes and audio text are further input into the second-layer fine-grained analysis layer for fine-grained analysis. Targeting multimodal interaction and fusion, feature fusion is achieved between the visual features of the video keyframes and the semantic features of the audio text. Finally, a tiered review result is output based on the fused features. The video review method provided by this disclosure embodiment improves the accuracy and precision of the review process.

[0040] Figure 2 This is a logical schematic diagram of a video review method provided in an embodiment of the present disclosure. Figure 3 A flowchart illustrating yet another video review method provided in this disclosure embodiment is shown below. Figure 2 and Figure 3 As shown, in some embodiments, the video review method includes: S301, Extract keyframes from the video and extract audio text from the video.

[0041] This step includes the following two parts: (1) Extract keyframes from the video.

[0042] Keyframes in video are the core frames used to define animation effects, parameter changes, or content transitions in digital video. Keyframe extraction is central to video analysis and processing, aiming to select the most representative frames from a video sequence to efficiently represent the video content.

[0043] This embodiment extracts keyframes from a video based on the maximum value of the information entropy curve.

[0044] Information entropy is a core concept in information theory, used to quantify the uncertainty or amount of information. Information entropy is used to quantify the information content of an image; a higher entropy value indicates richer image content. Unlike conventional keyframe extraction methods based on fixed-interval sampling, this disclosure utilizes information entropy to quantify the information content of video frames, thereby overcoming the limitations of fixed-interval sampling and enabling dynamic capture of changes in video content to extract more representative keyframes. In this disclosure, video frames are arranged chronologically, and after grayscale transformation of each frame, the information entropy of each frame is calculated, forming a two-dimensional entropy curve. Figure 4 This is a flowchart illustrating a video keyframe extraction method based on information entropy, as provided in an embodiment of this disclosure. Figure 4 As shown, a video keyframe extraction method based on information entropy according to an embodiment of this disclosure includes the following steps: S411. Perform grayscale processing on all video frames; Video by Frames, ordered by time The last frame (frame v) image is , No. The image of the frame is Each frame It is The RGB image matrix (H represents image height, W represents image width, 3 represents 3 channels), let yes The grayscale value of the j-th row and k-th column.

[0045] The core of grayscale transformation is mapping RGB pixel values ​​to single-channel grayscale values, which is achieved by binarizing the image. ; in, The average or weighted average of the RGB components, and T is the threshold (ranging from 0 to 255, which can be adjusted according to the overall image).

[0046] S412. Calculate the information entropy value of each video frame; After completing the grayscale transformation of all video frames, the information entropy value of each frame is calculated sequentially, and the result is obtained by the following formula. Frame Image Information entropy value: ; in, express Calculation of the probability of each grayscale value appearing. x i Represents the possible values ​​of a pixel in the image. iA specific grayscale value (for example, for an 8-bit grayscale image, the grayscale value range is [0, 255], then...) ); The gray level is the number of different gray values ​​that actually exist in the image.

[0047] S413. Obtain the information entropy curve; Based on the information entropy value of each video frame obtained in step S412, an information entropy value curve is obtained, which can be represented as: ; S414. Calculate the maximum point of the information entropy curve; In the information entropy curve, the frames corresponding to local maxima usually represent nodes where content changes abruptly or the amount of information increases significantly, such as scene transitions or key actions. Therefore, these frames are selected as keyframes.

[0048] The sliding window extremum method is used to determine whether frame t is a local maximum point. Specifically, a window radius k is defined. If the value is the maximum value within the window, it is determined to be a local maximum point: ; To avoid detecting spurious extrema in low-entropy regions, an additional threshold condition is applied: ; in, The mean of the information entropy value sequence; The standard deviation of the entropy sequence; This represents the adjustment coefficient, used to control sensitivity.

[0049] S415. Take the frame corresponding to the maximum value point as the video keyframe.

[0050] The frame corresponding to the maximum value of the information entropy curve is taken as the video keyframe.

[0051] To address the shortcomings of existing keyframe extraction algorithms in accurately and effectively locating key content, this disclosure employs an algorithm based on frame image information entropy to extract key frames. This algorithm can accurately focus on key information in the video, significantly reduce redundant frame interference, effectively reduce data processing volume, and improve the efficiency of video content analysis.

[0052] (2) Extract the audio text from the video.

[0053] S511. Extract audio data from the video; In some embodiments, video processing can be performed by extracting separate audio data using Python's moviepy library: the video file is loaded using the VideoFileClip class, the audio portion of the video is obtained through the audio attribute, and then the audio data is saved as a file using the write_audiofile method.

[0054] S512. Input the audio data into a deep learning model with global semantic understanding capabilities to extract the audio text of the video.

[0055] For example, the audio data extracted in step S511 is used as input, and the Whisper-Large-v3 model is called to perform speech recognition and analysis on the saved audio file. Whisper-Large-v3 is an encoder-decoder model based on the Transformer architecture developed by OpenAI, mainly used for multilingual automatic speech recognition (ASR) and speech translation.

[0056] The Whisper-Large-v3 model supports multiple languages. By specifying the `language` parameter, the recognition accuracy for specific languages ​​can be improved, taking into account the characteristics of the relevant media materials. This method relies on the Python libraries `torch` and `whisper`. After installing the dependencies, a Whisper model of the specified size (specifying `large-v3`) is loaded based on the `whisper` library, and the device type is specified (CUDA or CPU, depending on the situation). The `transcribe()` method of the `Whisper` class is used to transcribe audio files (setting the audio file address and language). Finally, the output format is set to JSON or SRT, generating timestamped segments (each segment containing start time, end time, and corresponding text content). After audio transcription, the timestamps need to be formatted: if the transcription result format is SRT, the timestamps (floating-point numbers in seconds) in the transcription result are converted to the "HH:MM:SS,mmm" string format required by SRT; if the transcription result format is JSON, floating-point seconds are used directly as the timestamps, and the transcription result dictionary is directly written to the JSON file. This ultimately converts the speech information into text form and outputs it.

[0057] This embodiment combines Whisper-Large-v3 and BERT-TextCNN to extract video speech information. Compared with existing technologies that use VGGish to directly extract audio features, this embodiment first converts speech to text using Whisper-Large-v3, and then uses the hybrid deep learning architecture BERT-TextCNN, which has the advantages of strong robustness and strong deep feature extraction capabilities.

[0058] S302. Based on the keyframes of the video and the audio text of the video, perform pre-filtering based on a multimodal large model.

[0059] A multimodal large model is an artificial intelligence system capable of processing multiple data formats (such as text, images, audio, and video) and achieving cross-modal understanding and completing complex tasks (such as visual question answering and AIGC generation). Multimodal large models are typically based on the Transformer architecture and combine methods such as positional encoding and multimodal alignment to achieve unified modeling of cross-modal information.

[0060] Taking the Qwen2.5-Omni multimodal large model as an example, after the extracted keyframes and the audio text of the video are input into the model, the multimodal encoder inside the model processes the text, image, and video features through the Transformer architecture, and performs semantic association analysis by combining contextual information (based on the Thinker-Talker dual-core architecture). Regarding the illegal content detection task, the model learns pre-trained multimodal representations and compares them with a sensitive word library (such as pornography, violence, and politically sensitive words) and an image feature library (such as facial recognition and object detection).

[0061] While such multimodal large-scale models support full-modal input and output of text, images, audio, and video, the sheer volume of multimedia resources like video can lead to excessive memory usage and computational resource consumption if directly input into the large model. Therefore, processing the video by extracting keyframes and extracting speech-text before inputting it into the multimodal large-scale model can reduce computational load and improve review efficiency. The input consists of keyframes from the video extracted in step S301, image resizing, and conversion to Base64 encoding (inputting image data as text further compresses computational complexity); and the speech-text containing timestamps extracted in step S301 after Whisper-Large-v3 conversion. The multimodal large-scale model analyzes and processes these two types of input and designs prompts (which should include specific review requirements and the format of the detection results). The multimodal large-scale model first analyzes and processes each keyframe and its corresponding timestamped speech-text, then integrates the analysis results from each frame. For each keyframe, the multimodal model outputs information such as whether the frame contains inappropriate content (e.g., nudity, gore), the specific type of violation, and its location. For the corresponding audio / text segment, the multimodal model analyzes for semantic violations, such as keywords or expressions related to violence or pornography. Then, timestamp information is used to correlate the analysis results of each frame according to the video's chronological order. Finally, the multimodal model generates a structured report (e.g., in table or JSON format) based on the integrated analysis results, facilitating quick review and processing by subsequent auditors.

[0062] Existing video review processes do not pre-filter obviously illegal content, resulting in high computational load and low processing efficiency. This disclosure adds a pre-filtering layer based on a pre-trained multimodal large model, which can quickly screen and return preliminary judgment results, effectively improving review efficiency.

[0063] S303, Fine-grained analysis review.

[0064] If no obvious violations are detected after the pre-filtering layer in step S302, the keyframes of the video and the audio / text of the video are input into the fine-grained analysis and review layer for further analysis. The fine-grained analysis and review mainly consists of the following three parts: (1) Video keyframe feature mining In this embodiment of the disclosure, keyframes of a video are input into a convolutional neural network model for extracting visual features to extract the visual features of the keyframes.

[0065] In some embodiments, keyframes of a video are input into a ResNet-152 model to extract video image features. ResNet-152 consists of an input layer, an initial convolutional layer, four residual block groups, a global average pooling layer, and a fully connected layer, for a total of 152 layers (only convolutional and fully connected layers are computed). Through deep residual structures and bottleneck design, it achieves high-precision feature extraction while maintaining computational efficiency.

[0066] The keyframes of the input video are first resized to a fixed size (224×224 pixels for ResNet-152) and normalized. Then, the initial convolutional-pooling layers of the input model perform preliminary feature extraction. This process significantly reduces the spatial dimensionality of the feature maps while preserving key texture information. Four residual block groups form the core structure of ResNet-152. Each group contains multiple residual blocks, each with three convolutional layers. Each residual block directly adds the input to the output of the convolution through a skip connection, mitigating the vanishing gradient problem and reducing the feature map size. Thus, local-to-global feature learning is achieved through the internal convolutional layers, enabling in-depth mining of the visual features of the keyframes.

[0067] The last layer of ResNet-152 is a fully connected layer (used to output classification results). Therefore, to adapt to feature extraction tasks, the network structure needs to be adjusted. The fully connected layer at the end of the original ResNet-152 is removed, and the penultimate layer—the global average pooling layer—is used to spatially average the features of each channel, compressing the feature map into a fixed dimension and outputting the visual features of the current video frame. Where v and m represent the number of rows and columns of the visual feature value vector matrix of the video frame, respectively. (The final convolutional layer Conv5_x of the SS network outputs a feature map of size 7×7×2048. The global average pooling layer compresses the 7×7 feature map of each channel into 1×1, with an output dimension of 2048. Assuming 20 frames, this outputs the visual features of the video frames while keeping other structures of ResNet-152 unchanged.) ).

[0068] Visual features of keyframes are extracted using a pre-trained ResNet-152 model, which preserves spatial information and encodes rich semantic features through high-dimensional channels.

[0069] (2) Video text feature mining This disclosure embodiment uses a hybrid deep learning model to extract semantic features from the audio text of a video to obtain the semantic features of the audio text.

[0070] In some embodiments, the audio-visual text of a video is input into the BERT-TextCNN model to extract semantic features from the audio-visual text. BERT-TextCNN is a hybrid deep learning model that combines the BERT and TextCNN models. It consists of word embedding layers, convolutional layers, pooling layers, and fully connected layers, and is primarily used for text classification tasks. By fusing the deep semantic understanding capabilities of BERT with the local feature extraction capabilities of TextCNN, the BERT-TextCNN model significantly improves the accuracy and robustness of text classification.

[0071] The speech-text recognized by Whisper-Large-v3 is preprocessed, including removing redundant spaces and special characters, and converting English letters to lowercase. Then, the text is segmented and truncated (e.g., using BERT's tokenizer) and special markers are added. The text length is padded and truncated according to BERT's maximum limit, generating BERT input tensors (Token IDs, Segment IDs, and Attention Mask). These tensors are then input into BERT-TextCNN, where they are first converted into word vector matrices by the word embedding layer of the BERT sub-model. These matrices are then further input into the Transformer encoder in BERT for layer-by-layer feature extraction, outputting feature vectors. Next, the feature vectors output by BERT are input into the TextCNN model: TextCNN extracts features through multi-scale convolutional kernels. The convolutional kernels of the TextCNN sub-model are configured, denoted as follows: ,in Represents the height of the convolution kernel. This represents the width of the convolutional kernel, and convolution operations are performed based on convolutional kernels of various sizes. Then, the TextCNN pooling layer performs max pooling, taking the maximum value from each convolutional output (preserving the most salient features), and concatenating the pooling results of all convolutional kernels to obtain the final semantic feature vector of the speech text. Where t and n represent the number of rows and columns of the semantic feature vector matrix of the speech text, respectively. (For example, if the input sentence dimension is (15, max_len), and three different widths are used when configuring the convolution kernel, such as setting...) Used to capture 2-gram, 3-gram, and 4-gram features, The input is set to a single channel, and the output channels are set to 256. After max pooling, the output of each convolutional kernel yields 256-dimensional features. The total dimension after concatenation is 256 × 3 = 768. At this point, the output... ).

[0072] (3) Multimodal fusion S611. Initialize the mapping of the dual-modal features; The visual features extracted from the video frames above Semantic features of speech text The bimodal model is mapped to a unified latent space via linear projection: ; in, , It is a projection matrix, whose function is to map feature vectors of different dimensions into a unified latent space; , These are the mapped visual feature vector and semantic feature vector, respectively.

[0073] S612. Align the visual feature vector with the semantic feature vector in terms of dimensions; If the number of video frames is not equal to the number of text sentences, that is When doing this, zero-padding is needed at the tails of the lower-dimensional feature vectors in both systems. Adding zero vectors makes the dimensions of both systems the same, meaning that after padding, the desired result is achieved. or ,in and They are respectively for and The vector obtained after zero padding.

[0074] S613, Perform matrix masking; Simultaneously, a mask matrix M is generated that satisfies: ; in, , is the dimension value of the mask matrix M.

[0075] S614. Weight allocation based on self-attention mechanism; Next, a self-attention mechanism is used to assign weights. Visual features are selected. As a query (Q); select semantic features As a key (K) and a value (V), the calculation is as follows: ; in, For a learnable projection matrix, As a scaling factor, it is usually set to To maintain dimensional consistency, the attention weights are calculated as follows: .

[0076] S615, Output fusion features; After calculation using the self-attention mechanism, the final output fused feature is: It has the ability to enhance contextual representation.

[0077] This multimodal fusion mechanism addresses the video-text semantic gap by dynamically adjusting modal contributions through differentiable attention weights. For example, the spatiotemporal correlation between gory images and abusive language in violent scenes is enhanced.

[0078] S616, Output the hierarchical recognition results.

[0079] The fused features output from step S615 are input into the fully connected layer, and the classification result is obtained based on the Softmax function: ; Where C is the number of preset violation types (such as violence, pornography, etc.); F is the final fusion feature output by the multimodal fusion mechanism mentioned above; This is the weight matrix, used to map features to the classification space; This is the bias vector, used to adjust the classification output. Assuming a preset number of categories C = 6, the output probability vector is... ,in Let i represent the probability of the i-th type of violation, satisfying .

[0080] The three-level decision-making rules are set up, and the specific execution process is as follows: (1) L1 (High Confidence Violation Check) to prevent the spread of high-risk content. Determine whether any type of violation probability value exceeds the L1 threshold. (If it can be set to) ),Right now If the conditions are met, L1 level automatic interception will be triggered. (2) L2 (Suspected Violation Check): Balancing automation with the risk of false positives. If no L1 is triggered, check if the highest probability condition is met. (If it can be set to) If so, L2 is triggered, and the system enters the manual review queue; (3) L3 (Low-risk approval): Low-risk transactions are automatically approved, reducing the burden of manual review. If neither L1 nor L2 triggers the highest probability, the transaction is automatically approved, provided that the conditions are met. .

[0081] In practical applications, the relevant threshold values ​​need to be adjusted according to business requirements.

[0082] This embodiment sets two threshold values ​​to divide video review into three levels: L1, L2, and L3, constructing a progressive review mechanism. This achieves precise hierarchical control of video review risks and can significantly reduce the burden of manual review.

[0083] In this implementation, addressing the limitation of current keyframe extraction algorithms in accurately and effectively locating key content, an algorithm based on frame image information entropy is designed for keyframe extraction. This method is computationally fast and effectively covers the main content of the video. Simultaneously, video audio is extracted and converted to text using the Whisper-Large-v3 model. This model has the ability to perform text correction based on contextual information, outputting accurate text data. Furthermore, a rapid pre-filtering mechanism is implemented to replace manual initial screening. Keyframes and audio text are input into a multimodal large model for rapid pre-filtering of obviously violating content and returns preliminary judgment results, significantly improving review efficiency and avoiding content backlog caused by slow manual review. If the video is not captured or intercepted in the pre-filtering layer, the keyframes and audio text are further input into a fine-grained analysis layer for fine-grained analysis through multiple automated modules. Subsequently, aiming at multimodal interaction and fusion, feature fusion of visual features of video keyframes and semantic features of audio text is achieved through linear projection and cross-attention mechanisms. Finally, a three-level judgment rule is designed to output graded results, achieving precise graded control of video review risks and significantly reducing the burden of manual review. Existing video review technologies, even those employing multimodal analysis, mostly rely on simple feature concatenation and weighted averaging, lacking in-depth interactive and correlational analysis of different modal information. This results in ineffective dynamic correlation and drawbacks such as poor keyframe focusing and extraction capabilities, insufficient audio feature extraction, and high computational demands. This disclosure proposes a video review method with multimodal feature fusion capabilities based on a deep learning architecture. The review process incorporates a two-layer structure: pre-filtering and fine-grained analysis. After rapid initial screening by the pre-filtering layer, the fine-grained analysis layer uses a deep learning model to simultaneously mine both visual and audio features and achieves deep multimodal feature fusion based on an attention mechanism, thereby improving the accuracy and precision of the review process.

[0084] The above description is only a preferred embodiment of the present invention. It should be noted that, for those skilled in the art, several improvements, optimizations and modifications can be made without departing from the principle of the present invention, and these should also be considered within the scope of protection of the present invention.

[0085] Figure 5 This is a schematic diagram of the structure of a video review device provided in an embodiment of the present disclosure, as shown below. Figure 5 As shown, the device 500 includes: The initial review module 501 is configured to screen and mark obviously illegal videos based on the keyframes of the video and the audio text of the video; Feature module 502 is configured to input keyframes and speech text of unlabeled video into a deep learning model and output visual features of keyframes and semantic features of speech text. The fusion module 503 is configured to obtain fusion features based on the visual features of the keyframe and the semantic features of the speech text; The determination module 504 is configured to determine the multi-level review result of the video based on the fusion features.

[0086] In some embodiments, the device 500 further includes: Extraction module 505 is configured to extract keyframes from the video and extract audio text from the video.

[0087] In some embodiments, the extraction module 505 is further configured to obtain the information entropy value of each frame of the video and generate an entropy curve; further configured to identify feature points in the entropy curve that meet preset conditions; and further configured to determine the frame corresponding to the feature point as a key frame.

[0088] In some embodiments, the feature points are local maxima of the entropy curve, obtained by screening using a sliding window method combined with threshold conditions of entropy statistical features, wherein the entropy statistical features include the mean and standard deviation of the entropy sequence.

[0089] In some embodiments, the extraction module 505 is further configured to separate and extract audio data from the video; and is further configured to input the audio data into a deep learning model with fused global semantic understanding to extract the speech text of the video.

[0090] In some embodiments, the preliminary review module 501 includes: The comparison module is configured to input the keyframes of the video and the audio text of the video into a multimodal model in a preset format. The multimodal model combines cross-modal semantic association analysis with comparison with a violation feature database to generate a structured preliminary judgment result. The tagging module is configured to flag videos that are deemed to be in violation of the initial assessment.

[0091] In some embodiments, the fusion module 503 includes: The alignment module is configured to perform dimensional transformation and dimensional alignment on the visual features of the keyframe and the semantic features of the speech text through linear projection. The global module is configured to perform computation based on a self-attention mechanism to obtain fused features that incorporate global information.

[0092] In some embodiments, the determination module 504 includes: The probability module is configured to obtain probabilities corresponding to various violation types based on the fused features; The review module is configured to determine the video review result based on a comparison between the probability corresponding to each type of violation and a threshold value.

[0093] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0094] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as... Figure 6 As shown, this disclosure also provides an electronic device 600, which includes at least one processor 601 and a memory 602 coupled to the processor 601. The memory 602 is used to store at least one processor 601 executable instructions, wherein the at least one processor 601 is used to execute the instructions to implement the steps of the method described above in this disclosure.

[0095] The processor 601 described above can also be called a Central Processing Unit (CPU), which can be an integrated circuit chip with signal processing capabilities. Each step in the method described in this embodiment can be implemented by the integrated logic circuitry in the processor 601 or by software instructions. The processor 601 can be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method in this embodiment can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in the memory 602, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The processor 601 reads information from the memory 602 and, in conjunction with its hardware, completes the steps of the method described above.

[0096] Figure 7 This is a schematic diagram of an exemplary computer system provided by an embodiment of the present disclosure. Various operations / processes according to embodiments of the present disclosure, implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, for example... Figure 7 The computer system 700 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including those described above.

[0097] Computer system 700 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0098] like Figure 7As shown, the computer system 700 includes a computing unit 701, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the computer system 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0099] Multiple components in the computer system 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 can be any type of device capable of inputting information into the computer system 700. The input unit 706 can receive input numerical or character information and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 707 can be any type of device capable of presenting information and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. The storage unit 708 may include, but is not limited to, a hard disk and an optical disk. The communication unit 709 allows the computer system 700 to exchange information / data with other devices via a network such as the Internet, and may include, but is not limited to, modems, network interface cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth™ devices, Wi-Fi devices, WiMax devices, cellular communication devices, and / or the like.

[0100] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the methods described in the embodiments of this disclosure can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 702 and / or communication unit 709. In some embodiments, the computing unit 701 can be configured to perform the methods described in the embodiments of this disclosure by any other suitable means (e.g., by means of firmware).

[0101] This disclosure provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the methods described in this disclosure.

[0102] Computer-readable storage media can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or devices that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0103] It should be noted that the computer-readable storage medium described in this disclosure can be a computer-readable signal medium, a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), or any suitable combination thereof.

[0104] Embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the video review method described above.

[0105] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on a computer, partially on a computer, as a standalone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0106] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0107] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc.

[0108] It should be noted that, in this document, terms such as "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0109] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for video review, characterized in that, The method includes: Based on the keyframes and audio transcripts of the video, screen and mark videos that clearly violate regulations; Input the keyframes and audio text of the unlabeled video into the deep learning model, and output the visual features of the keyframes and the semantic features of the audio text. The fusion features are obtained based on the visual features of the keyframes and the semantic features of the speech text; The multi-level review result of the video is determined based on the fusion features.

2. The method according to claim 1, characterized in that, Before screening and marking obviously violating videos based on keyframes and audio text of the video, the method further includes: Extract keyframes from the video and extract the audio text from the video.

3. The method according to claim 2, characterized in that, The keyframes extracted from the video include: Obtain the information entropy value of each frame of the video and generate an entropy curve; Identify feature points in the entropy curve that meet preset conditions; The frames corresponding to the feature points are identified as keyframes.

4. The method according to claim 3, characterized in that, The feature points are local maxima of the entropy curve, obtained by screening using a sliding window method combined with threshold conditions of entropy statistical features. The entropy statistical features include the mean and standard deviation of the entropy sequence.

5. The method according to claim 2, characterized in that, The extracted audio-text from the video includes: Extract audio data from the video; The audio data is input into a deep learning model that integrates global semantic understanding to extract the speech text of the video.

6. The method according to claim 1, characterized in that, The process of screening and marking obviously violating videos based on keyframes and audio-visual transcripts includes: The keyframes and audio text of the video are input into a multimodal model in a preset format. The multimodal model combines cross-modal semantic association analysis with a violation feature database to generate a structured preliminary judgment result. Videos that are marked as obviously in violation of the initial assessment are flagged.

7. The method according to claim 1, characterized in that, The process of obtaining fused features based on the visual features of the keyframes and the semantic features of the speech text includes: The visual features of the keyframe and the semantic features of the speech text are transformed and aligned using linear projection. Computation based on the self-attention mechanism yields fused features that incorporate global information.

8. The method according to claim 1, characterized in that, The determination of the multi-level video review result based on the fusion features includes: Based on the fusion features, probabilities corresponding to various violation types are obtained; The video review result is determined by comparing the probability corresponding to each type of violation with the threshold value.

9. A device for video review, characterized in that, The device includes: The initial review module is configured to screen and mark obviously illegal videos based on keyframes and audio-visual transcripts of the video. The feature module is configured to take keyframes and speech text from unlabeled videos as input to a deep learning model and output the visual features of the keyframes and the semantic features of the speech text. The fusion module is configured to obtain fusion features based on the visual features of the keyframes and the semantic features of the speech text; The determination module is configured to determine the multi-level review result of the video based on the fusion features.

10. An electronic device, characterized in that, include: At least one processor; Memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1-8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the video review method as described in any one of claims 1-8.

Images