Video auditing method and device and storage medium

Abstract

The invention discloses a video auditing method and device and a storage medium, and relates to the technical field of video detection.The video auditing method comprises the steps that after an input video is received, visual features of the input video are extracted, and feature vectors are formed; inputting the feature vectors into a vector retrieval engine constructed on the basis of a hierarchical navigable map, and performing neighbor search matching with vectors in an illegal video feature library to determine a similarity matching result; and generating an interception instruction of the input video according to a similarity matching result. The technical effect of improving the generalization ability of the video auditing model is achieved.

Description

Technical Field

[0001] The present application relates to the field of video detection technology, and in particular to video review methods, equipment and storage media. Background Art

[0002] At present, the video review method based on deep learning relies on fully supervised training and requires the collection of a large amount of labeled data in advance to train model parameters. As a result, when dealing with new types of illegal content, it is impossible to directly identify new illegal content due to the lack of labeled data. New sample data needs to be collected and labeled again, and then the model needs to be retrained. As a result, the model's response cycle to new illegal content is long and cannot meet dynamically changing review needs.

[0003] The above content is only used to assist in understanding the technical solution of this application and does not constitute an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a video review method, device and storage medium, aiming to solve the technical problem of how to improve the generalization ability of the video review model.

[0005] To achieve the above objectives, this application proposes a video review method, which includes: After receiving the input video, extracting visual features of the input video to form a feature vector; Input the feature vector into a vector search engine built based on a hierarchical navigable graph, perform a nearest neighbor search and match with the vectors in the illegal video feature library, and determine a similarity matching result; An interception instruction for the input video is generated according to the similarity matching result.

[0006] In one embodiment, after receiving the input video, before the step of extracting visual features of the input video to form a feature vector, the following steps are included: Extracting a semantic understanding representation of the input video through a multimodal video feature extraction model, and performing a first-level illegal content review on the semantic understanding representation; According to the result of the first-level illegal content review, the first-level video interception operation is directly triggered to generate an interception instruction for the input video.

[0007] In one embodiment, the step of extracting a semantic understanding representation of the input video using a multimodal video feature extraction model and performing a first-level illegal content review on the semantic understanding representation includes: receiving the input video, decoding and segmenting the input video, and determining multimodal data of the input video; Parsing the multimodal data through the multimodal video feature extraction model to generate the semantic understanding representation of the video content; A dynamic query instruction is generated based on a preset violation detection prompt template, and a targeted content review is performed on the semantic understanding representation to determine whether there is any illegal content.

[0008] In one embodiment, after receiving the input video, the step of extracting visual features of the input video to form a feature vector includes: Dynamically sampling the input video using an adaptive key frame extraction algorithm to obtain the key frame sequence; The key frame sequence is input into a visual feature extraction network driven by a spatiotemporal state space model, visual features of each key frame in the key frame sequence are extracted, and a feature vector of the key frame sequence is generated.

[0009] In one embodiment, the step of inputting the feature vector into a vector search engine constructed based on a hierarchical navigable graph, performing a nearest neighbor search and matching with vectors in a feature library of illegal videos, and determining a similarity matching result includes: Establishing the hierarchical navigable graph for the vectors in the illegal video feature library, and establishing a connection relationship between nodes in each layer of the hierarchical navigable graph based on vector similarity; Starting from the top layer of the hierarchical navigable graph, searching layer by layer, and screening the candidate set most similar to the feature vector through a dynamic path optimization strategy; A similarity matching result is generated based on the similarity score of the candidate set and a preset dynamic threshold.

[0010] In one embodiment, before the step of inputting the feature vector into a vector search engine constructed based on a hierarchical navigable graph, performing a nearest neighbor search and matching with the vectors in the illegal video feature library, and determining a similarity matching result, the step further includes: Extracting a key frame sequence of each illegal video in the illegal video library by an adaptive key frame extraction algorithm; Processing a key frame sequence of each of the illegal videos using a feature extraction network driven by a spatiotemporal state space model to generate an illegal feature vector for each of the illegal videos; Each of the violation feature vectors is stored in a standardized format as a structured feature file to obtain the violation video feature library.

[0011] In one embodiment, after the step of generating an interception instruction for the input video based on the similarity matching result, the following steps are included: If a new illegal video is detected in the illegal video library, a new feature vector of the new illegal video is generated through a feature extraction network driven by a spatiotemporal state space model; Inserting the newly added feature vector into a vector retrieval engine built based on a hierarchical navigable graph, and dynamically adjusting the node connection relationship through an incremental graph structure optimization algorithm to update the illegal video feature library; Based on the updated illegal video feature library, the input video is reviewed for violations.

[0012] In one embodiment, the step of generating an interception instruction for the input video based on the similarity matching result includes: Determine, based on the similarity matching result, a violation feature vector in a matched violation video feature library, and extract the violation type of the violation feature vector; Based on the timestamp information of the input video, intercepting the key frame sequence of the illegal segment to generate a screenshot of the illegal content; Extracting multimodal data of the violation feature vector in the same time period according to the timestamp information, and generating a violation feature comparison heat map; The interception instruction is constructed based on the violation type, and the screenshot of the violation content and the violation feature comparison heat map are rendered to the review interface.

[0013] In addition, to achieve the above-mentioned purpose, the present application also proposes a video review device, which includes: a memory, a processor, and a computer program stored on the memory and runnable on the processor, and the computer program is configured to implement the steps of the video review method described above.

[0014] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium, and when the computer program is executed by the processor, the steps of the video review method described above are implemented.

[0015] The present application provides a video review method. After receiving the input video, the present application extracts the visual features of the input video to form a feature vector; inputs the feature vector into a vector retrieval engine constructed based on a hierarchical navigable graph, performs a nearest neighbor search and matching with the vectors in the illegal video feature library to determine the similarity matching result; and generates an interception instruction for the input video based on the similarity matching result. The present application first uses key frame extraction technology to filter out key frame sequences that can represent the video content from the video, reducing the amount of data while retaining key visual information. By extracting the visual features of the key frame sequence to form a feature vector, it helps to more accurately identify the content of the video and improve the model's ability to identify different types of illegal content. By using an efficient index structure through the vector retrieval engine, the illegal feature vector that is most similar to the input video feature vector can be quickly found in the high-dimensional feature space, thereby improving retrieval efficiency and accuracy. Based on the similarity matching result, an interception instruction is generated to achieve rapid identification and interception of illegal videos and ensure the accuracy of the review. The present application achieves the technical effect of improving the generalization ability of the video review model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A flowchart of the first embodiment of the video review method of this application is provided; Figure 2 A flowchart of the second embodiment of the video review method of this application is provided; Figure 3 A flowchart of the third embodiment of the video review method of this application is provided; Figure 4 A flowchart of the fourth embodiment of the video review method of this application is provided; Figure 5 A schematic diagram of the overall process provided for the video review method of this application; Figure 6 This is a schematic diagram of the device structure of the hardware operating environment involved in the video review method in the embodiment of the present application.

[0019] The purpose, features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0020] It should be understood that the specific embodiments described herein are merely used to explain the technical solutions of the present application and are not intended to limit the present application.

[0021] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0022] At present, the video review method based on deep learning relies on fully supervised training and requires the collection of a large amount of labeled data in advance to train model parameters. As a result, when dealing with new types of illegal content, it is impossible to directly identify new illegal content due to the lack of labeled data. New sample data needs to be collected and labeled again, and then the model needs to be retrained. As a result, the model's response cycle to new illegal content is long and cannot meet dynamically changing review needs.

[0023] This application uses key frame extraction technology to filter out key frame sequences that can represent the video content from the video, reducing the amount of data while retaining key visual information. By extracting the visual features of the key frame sequence and forming a feature vector, it helps to more accurately identify the content of the video and improve the model's ability to identify different types of illegal content. By using an efficient index structure through a vector retrieval engine, the illegal feature vector that is most similar to the input video feature vector can be quickly found in the high-dimensional feature space, improving retrieval efficiency and accuracy. Based on the similarity matching results, interception instructions are generated to achieve rapid identification and interception of illegal videos, ensuring the accuracy of the review.

[0024] It should be noted that the execution entity of this embodiment can be a video review system, or a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, mobile phone, etc., or a control device of a video review system capable of implementing the above functions, etc. This embodiment does not specifically limit this. The following uses the video review system as an example to illustrate this embodiment and the following embodiments.

[0025] Example 1 Based on this, this application proposes a video review method of the first embodiment, please refer to Figure 1 , the video review method includes: Step S10: After receiving the input video, extract the visual features of the input video to form a feature vector.

[0026] Extracting representative visual features from the input video provides basic data for subsequent video content analysis. By extracting key frame sequences and extracting their visual features, the amount of computation can be significantly reduced while retaining the main visual information in the video.

[0027] It's important to note that a keyframe is a frame in a video that represents its primary content or scene changes. A keyframe sequence is a series of keyframes extracted from a video according to specific rules. Visual features are attributes or information extracted from an image or video frame that describe its visual content, including color, texture, shape, and edges. A feature vector is a vector that represents the extracted visual features in numerical form, with each element representing a specific visual feature.

[0028] As an optional implementation, frames are extracted from the input video at fixed time intervals to form a keyframe sequence. Visual features of the keyframes are extracted using image processing methods. The extracted visual features are expressed as numerical values ​​and arranged in a vector form in a specific order to form a feature vector.

[0029] As another optional implementation of extracting key frames, the difference between adjacent frames is calculated, and when the difference exceeds a certain threshold, the current frame is extracted as the key frame.

[0030] Optionally, step S10 includes: Step S11 , dynamically sampling the input video using an adaptive key frame extraction algorithm to obtain the key frame sequence.

[0031] If no illegal content is found in the first-level illegal content review, the video is dynamically sampled through an adaptive key frame extraction algorithm to extract a key frame sequence that can represent the video content.

[0032] It should be noted that the adaptive key frame extraction algorithm is an algorithm that can dynamically adjust the sampling interval according to the complexity and change frequency of the video content. It aims to extract the most representative key frames to reduce data redundancy and retain key information.

[0033] As an optional implementation, a preliminary analysis of the video content is performed to identify scene change information, and based on the scene change information, the sampling interval is dynamically adjusted to extract key frames. The extracted key frames are arranged in chronological order to form a key frame sequence.

[0034] As an optional implementation of sampling, a shot boundary detection algorithm is used to divide the video scenes, and key frames are sampled at fixed intervals for each scene.

[0035] As another optional implementation of sampling, the difference in pixel values ​​between adjacent frames is calculated, and when the difference exceeds a threshold, the key frame with the current high difference is sampled.

[0036] Step S12: inputting the key frame sequence into a visual feature extraction network driven by a spatiotemporal state space model, extracting visual features of each key frame in the key frame sequence, and generating a feature vector of the key frame sequence.

[0037] A visual feature extraction network driven by a spatiotemporal state space model is used to process key frame sequences and generate feature vectors that can comprehensively reflect the visual characteristics of video content, so as to improve the understanding and analysis capabilities of video content.

[0038] It should be noted that the spatiotemporal state space model is a modeling method that combines time and space to capture the dynamic and static characteristics of the video. The visual feature extraction network is a deep learning-based neural network used to extract visual features from key frames.

[0039] Optionally, a spatiotemporal state space model is constructed to define state transition equations and observation equations to describe the state changes and observation values ​​of objects in the video.

[0040] Exemplarily, the key frame sequence is input into a visual feature extraction network, the visual features of each key frame are extracted, and a feature vector of the key frame sequence is generated.

[0041] Optionally, audio modal features and text modal features of the key frame sequence are extracted, and the visual features are fused with the audio modal features and the text modal features to generate a multimodal feature vector.

[0042] Step S20: input the feature vector into a vector search engine constructed based on a hierarchical navigable graph, perform a nearest neighbor search and match with the vectors in the illegal video feature library, and determine a similarity matching result.

[0043] By matching similarities with the illegal video feature library, we can confirm whether the video contains potential illegal content, improving the accuracy and comprehensiveness of the review. By comparing the historical illegal feature library, we can achieve illegal content discovery through zero-sample learning.

[0044] In this embodiment, the Hierarchical Navigable Small World (HNSW) is a graph-based approximate nearest neighbor search algorithm that accelerates the search through a hierarchical structure and local shortcuts. The vector retrieval engine is a system for storing and retrieving high-dimensional vectors, capable of quickly searching for illegal feature vectors similar to multimodal feature vectors in the illegal video feature library. The illegal video feature library is a pre-established database containing known illegal video features, used for similarity matching with the input video. Similarity matching is performed by calculating the similarity between the input video feature vector and the feature vectors in the illegal video feature library to determine whether the input video is similar to the known illegal video. If there is an illegal video whose similarity exceeds a preset threshold, the similarity matching result is determined to be the presence of illegal video features.

[0045] Exemplarily, a hierarchical navigable graph algorithm is used to construct the graph structure of the illegal video feature library, selecting connecting nodes based on a hierarchical strategy. Each modal vector of the multimodal feature vector is searched separately, starting from the high-level graph and gradually refining the path toward the lower layers. The cosine similarity between each modal vector and the vectors in the feature library is calculated. If the similarity exceeds a preset threshold, a match is determined. The matching results for each modality are aggregated, and the final match result is determined through weighted voting.

[0046] Optionally, step S20 includes: Step S21 : establishing the hierarchical navigable graph for the vectors in the illegal video feature library, and establishing a connection relationship between the nodes in each layer of the hierarchical navigable graph based on vector similarity.

[0047] High-dimensional feature vectors are organized into a hierarchical graph structure through a hierarchical navigable graph to accelerate approximate nearest neighbor search.

[0048] Exemplarily, the parameters of the hierarchical navigable graph are initialized, the maximum number of layers, the number of connections per layer, and the number of insertion candidates are set, and for each vector in the illegal video feature library, the nearest node is searched from the top layer according to the number of insertion candidates, the cosine similarity is calculated for all node pairs, and the number of nodes corresponding to the number of connections per layer is selected based on the similarity, and connections are established and recursively inserted into the next layer to construct a hierarchical navigable graph.

[0049] Step S22 , searching layer by layer starting from the top layer of the hierarchical navigable graph, and screening the candidate set most similar to the feature vector by a dynamic path optimization strategy.

[0050] It should be noted that the dynamic path optimization strategy is a strategy that dynamically adjusts the search path according to the similarity of the current node and the distribution of neighbors during the search process, giving priority to visiting nodes with high similarity.

[0051] For example, the feature vector is used as the query vector, and the search begins at the top level of the hierarchical navigable graph. Based on a dynamic path optimization strategy, at each level, the most similar neighboring nodes to the current node are prioritized. Based on a set number of search candidates ef, the ef neighboring nodes most similar to the query vector are found, and the search continues at the next level. At the bottom level, all candidate nodes are collected and sorted by similarity, and a preset number of nodes are selected as the candidate set.

[0052] Step S23: Generate a similarity matching result based on the similarity score of the candidate set and a preset dynamic threshold.

[0053] Based on the similarity scores of the candidate set and the preset dynamic threshold, a similarity matching result is generated to determine whether the input video is similar to the videos in the illegal video feature library.

[0054] It should be noted that the preset dynamic threshold is a similarity threshold that is dynamically adjusted according to review requirements and is used to determine whether the input video violates the regulations.

[0055] Exemplarily, the similarity score of each vector in the candidate set is compared with a preset dynamic threshold in the order of the candidate set. If there is a vector with a similarity score greater than the preset dynamic threshold, the similarity matching result is considered to be an illegal video feature.

[0056] As an example of this embodiment, an Adaptive algorithm is used to perform a holistic analysis of the input video. The interval for extracting key frames is automatically adjusted based on the dynamics of the input video, and video frames are extracted at the determined intervals as key frames. The extracted key frames are input into the MambaVision model, which extracts visual features such as edges, texture, color distribution, and object shape from the key frame images to form feature vectors. The input video feature vectors extracted by MambaVision are input into Faiss, which quickly searches for the most similar feature vectors in a stored library of illegal video features. Based on the matched feature vectors, matching results such as a similarity score and corresponding illegal video information are output. The Adaptive algorithm dynamically adjusts parameters or behaviors based on the characteristics and changes of the input data and is used to extract the most representative key frames from the video. The MambaVision model is a visual feature extraction model used to extract key visual features from images or video frames. Faiss is a similarity search library developed by Facebook AI that is suitable for processing large-scale vector data and is used to quickly find similar videos.

[0057] Step S30: generating an interception instruction for the input video according to the similarity matching result.

[0058] After confirming that the input video contains illegal features based on the similarity matching results, interception instructions are automatically generated to improve review efficiency. By generating interception instructions in a timely manner, the further spread of illegal videos on the platform is prevented.

[0059] In this embodiment, the interception instruction is a control instruction generated to prevent videos containing illegal content from spreading on the platform, including operations such as deletion, removal from shelves or restricting access, which contains information such as the reason for interception, violation type and timestamp.

[0060] Optionally, the similarity value in the similarity matching result is compared with a set similarity threshold to determine whether the similarity between the input video and the feature vector in the illegal video feature library exceeds the threshold. When the similarity matching result exceeds the set threshold, the interception instruction generation mechanism is automatically triggered.

[0061] As an optional implementation, a database query interface is used to obtain a unique identifier for the input video based on its file name. A description of the input video's violation type is obtained based on the most similar violation video sample retrieved from the violation video feature library. The corresponding interception operation type is then obtained based on the audit rule configuration file. When a similarity match indicates the presence of a violation video feature, the collected information, including the input video's unique identifier, violation type description, and corresponding interception operation type, is integrated to generate a structured interception instruction.

[0062] Optionally, the generated interception instruction is transmitted to the video management system, which receives and parses the interception instruction, performs corresponding operations according to the operation type of the interception instruction, and records detailed information of the interception operation, including the operation time, operation type and unique identifier of the video.

[0063] Optionally, step S30 includes: Step S31 : determining the violation feature vector in the matched violation video feature library according to the similarity matching result, and extracting the violation type of the violation feature vector.

[0064] Through the similarity matching results, the matching status of the input video and the specific illegal feature vector in the illegal video feature library is determined, so as to accurately locate the type of illegal content in the input video.

[0065] It should be noted that the similarity matching result is calculated by calculating the similarity between the multimodal feature vector of the input video and the feature vectors in the illegal video feature library, indicating whether the input video contains illegal video features. The illegal feature vector is a multimodal feature vector extracted from known illegal videos and stored in the illegal video feature library. It is used for similarity matching with the input video to identify potential illegal content. The violation type is the illegal content category corresponding to the violation feature vector.

[0066] As an optional implementation, the violation type information corresponding to the violation feature vector is stored in the violation video feature library, and the corresponding violation type in the violation video feature library is queried through the unique identifier of the matched feature vector.

[0067] As another optional implementation, feature vectors with labeled violation types are used as training data, and a suitable classification algorithm is selected to train a classification model. The matched feature vectors are input into the trained classification model to predict the violation type.

[0068] Step S32: based on the timestamp information of the input video, intercept the key frame sequence of the illegal segment to generate a screenshot of the illegal content.

[0069] By capturing the keyframe sequence of the offending clip and generating a screenshot, the content of the offending clip can be intuitively displayed. By combining it with timestamp information, the start and end times of the offending clip can be accurately marked.

[0070] It should be noted that the timestamp information is the start and end time of the illegal segment in the input video. The key frame sequence is a series of key frames extracted from the video that can represent the content of the illegal segment.

[0071] As an optional implementation, the timestamp information of all key frames in the input video and the similarity matching results are used to determine the timestamp information of the key frames that match the illegal feature vector. Based on the timestamp information, the key frame sequence in which the illegal content appears in the input video is captured, and the key frame sequence is saved as an image file to generate a screenshot of the illegal content.

[0072] Step S33: extracting multimodal data of the violation feature vector in the same time period according to the timestamp information, and generating a violation feature comparison heat map.

[0073] By extracting multimodal data of the same time period of the illegal feature vector, a comparative heat map is generated to intuitively display the characteristic distribution of illegal content in different modes.

[0074] It should be noted that the violation feature comparison heat map is a heat map generated by visualization technology, which maps the similarity between the input video and the violation feature vector into a color matrix to show the distribution and intensity of the violation features in different modes.

[0075] For example, based on the timestamp information corresponding to the violation time period of the input video, image key frames are extracted at the same time interval as the input video. A pre-trained visual feature extraction model is used to process each image frame to extract a visual feature vector. Audio data is extracted based on the timestamp information, and the audio is processed using an audio feature extraction model to extract an audio feature vector. Based on the timestamp information, optical character recognition technology is used to extract text information such as subtitles and titles from the image frames. The text is processed using a text feature extraction model to extract a text feature vector. The extracted visual, audio, and text features are fused to generate a comprehensive feature matrix, i.e., multimodal data of the violation feature vectors for the same time period. A visualization tool is used to convert the comprehensive feature matrix into a comparative heat map.

[0076] Step S34: construct the interception instruction based on the violation type, and render the screenshot of the violation content and the violation feature comparison heat map to the review interface.

[0077] Exemplarily, the corresponding interception operation type is determined according to the violation type, and an interception instruction is constructed based on the unique identifier, violation type and interception operation type of the input video. According to the layout of the review interface, the unique identifier, violation type and interception operation type information of the video are rendered to the video information area of ​​the review interface, the generated screenshot of the violation content is rendered to the screenshot display area of ​​the review interface, and the violation feature comparison heat map is rendered to the heat map display area of ​​the review interface for review by the review manager.

[0078] Optionally, the audit interface may further include an audit operation area, providing operation buttons such as audit pass, intercept, and re-audit for manual review by auditors.

[0079] This embodiment provides a video review method. This embodiment first uses key frame extraction technology to filter out key frame sequences that can represent the video content from the video, reducing the amount of data while retaining key visual information. By extracting the visual features of the key frame sequence to form a feature vector, it helps to more accurately identify the content of the video and improve the model's ability to identify different types of illegal content. Through the vector retrieval engine using an efficient index structure, the illegal feature vector that is most similar to the input video feature vector is quickly found in the high-dimensional feature space, thereby improving retrieval efficiency and accuracy. Based on the similarity matching results, an interception instruction is generated to achieve rapid identification and interception of illegal videos, ensuring the accuracy of the review.

[0080] Based on Example 1, Example 2 of this application proposes a video review method, referring to Figure 2 , before step S10, including: In step S40 , a semantic understanding representation of the input video is extracted through a multimodal video feature extraction model, and a first-level illegal content review is performed on the semantic understanding representation.

[0081] Perform efficient preliminary violation screening on input videos, utilize collaborative analysis of multimodal information, intercept explicit illegal content while ensuring real-time performance, and reduce subsequent consumption of computing resources.

[0082] In this embodiment, the multimodal video feature extraction model is a deep learning model that integrates multimodal information such as vision, hearing and text, and can extract spatiotemporal joint features from the video. The multimodal video feature extraction model analyzes and judges the video content through a comprehensive understanding of multimodal information, and the VideoLLaMA2 model can be used. The input video is a marketing video to be reviewed that is input into the multimodal video feature extraction model. The first-level illegal content review is the preliminary review stage in the entire video review process. It is guided by the multimodal video feature extraction model and prompt words to automatically analyze the multimodal features of the video, make a preliminary illegal judgment on the video content, and judge whether the video has obvious illegal content. Illegal content refers to videos that violate laws and regulations, social ethics, platform regulations, etc., such as scenes containing violent terrorist acts, pictures involving pornography and vulgarity, voice or text expressions with false advertising, etc.

[0083] Optionally, based on the rules and key points of marketing video review and the violation detection prompt template, targeted prompt words (prompt) are designed as dynamic query instructions.

[0084] As an optional implementation method, the marketing video to be reviewed is input into the VideoLLaMA2 model, and the model synchronously processes the visual, audio, and text information of the video to extract the key features of each modality. The multimodal features of the video are converted into a unified vector representation through a fusion mechanism to reflect the semantic connotation of the video. The designed prompt words are input into the VideoLLaMA2 model, guiding the model to analyze and judge the vector representation of the marketing video according to the requirements of the prompt words, and preliminarily screen out marketing videos that may violate regulations. Among them, the prompt words are a guiding instruction or prompt information for the VideoLLaMA2 model, enabling the model to judge whether the video contains illegal content according to the set rules.

[0085] For example, the VideoLLaMA2 model extracts video frames from a marketing video at a frequency of 1 frame per second. The audio is separated from the video and converted into text. Textual information such as subtitles and titles is extracted from the video frames. The VideoLLaMA2 model then inputs the pre-designed prompt "Please determine whether the following video contains gambling-related information." The model then identifies objects in the video frame based on the prompt. If gambling-related items such as playing cards, chips, and dice are detected, or if the extracted audio or text contains gambling-related dialogue such as "bet," "open the cards," or "wager," the video is judged to contain gambling content, i.e., illegal content.

[0086] Optionally, step S40 includes: Step S41: Receive the input video, decode and segment the input video, and determine multimodal data of the input video.

[0087] Convert the input video into a format suitable for model processing and extract multimodal data from the video, laying the foundation for subsequent feature extraction and content understanding. Through segmentation processing, the video is broken down into smaller units, facilitating parallel processing and analysis, improving overall processing efficiency.

[0088] It should be noted that the input video is the marketing video file that needs to be reviewed. Decoding converts the video file from a compressed encoding format into raw image and audio data for further processing. Segmentation divides the video into multiple segments at regular intervals, with each segment containing a period of video content. Multimodal data refers to the different types of data in a video, including visual data, audio data, and text data.

[0089] As an optional implementation, the input marketing video file is read and decoded into original image frames and audio streams using a decoding tool. The decoded image frames and audio streams are segmented at fixed time intervals to obtain multimodal data.

[0090] Step S42: parsing the multimodal data through the multimodal video feature extraction model to generate the semantic understanding representation of the video content.

[0091] It should be noted that semantic understanding representation is a high-level semantic representation of video content, which is used by the model to understand and judge the content.

[0092] Exemplarily, visual feature vectors, audio feature vectors, and text feature vectors are extracted separately, and the three feature vectors are integrated using multimodal fusion technology to fuse them into a unified feature vector to generate a semantic understanding representation of the video content.

[0093] Step S43: Generate a dynamic query instruction based on a preset violation detection prompt template, and perform a targeted content review on the semantic understanding representation to determine whether there is any violation content.

[0094] Through dynamic query instructions, the model is guided to focus on specific violations, improving the accuracy and efficiency of violation detection.

[0095] It's important to note that the violation detection prompt template is a predefined prompt word template used to generate dynamic query instructions to guide the model in detecting corresponding types of illegal content. Dynamic query instructions are specific instructions generated from the prompt template, also known as the model's prompt words, used to guide the model in targeted review of video content.

[0096] Exemplarily, based on the violation detection prompt template, a specific dynamic query instruction is generated, and the generated dynamic query instruction and the semantic understanding representation of the video are input into the multimodal video feature extraction model. The model then conducts a targeted review of the semantic understanding representation of the video content based on the dynamic query instruction.

[0097] Step S50: directly triggering the first-level video interception operation based on the result of the first-level illegal content review, and generating an interception instruction for the input video.

[0098] Through the first-level illegal content review, if it is detected that there is obvious illegal information in the video content of the input video, the first-level video interception operation is directly triggered to generate an interception instruction for the input video.

[0099] It should be noted that obvious illegal information refers to information in the video content that clearly violates platform rules or laws and regulations.

[0100] For example, a pre-set prompt word guides the model to analyze and determine whether gambling-related information is present in the video. If the model determines that gambling-related information is present in the video, it immediately triggers the first-level interception operation. Based on information such as the video's unique identifier, the violation reason, and the interception operation type, the corresponding interception instruction is generated.

[0101] Optionally, while executing the blocking instruction, the user who uploaded the video is retrieved, the unique identifier of the offending video and the reason for the violation are written into a violation notification template, and a corresponding violation notification is generated and sent to the user. The violation notification may also include a link to an appeal entry. The violation notification template reads, "Hello user, your video (unique identifier) ​​has been blocked due to (reason for violation). The appeal entry point is (appeal entry link)."

[0102] Optionally, the intercepted video information is reported to the review management platform for further review by manual reviewers.

[0103] This embodiment provides a video review method that first performs a preliminary review of input videos using a multimodal video feature extraction model. Upon detecting obvious violations, an interception operation is immediately triggered, automatically generating interception instructions to directly intercept the obviously violating video without further feature library comparison, thereby improving review efficiency. This preliminary review can quickly filter out a large number of videos that clearly violate regulations, reducing the number of videos that need to be compared with the violation video feature library, thereby reducing the computational burden and resource consumption of subsequent processing.

[0104] Based on Example 1, Example 3 of this application proposes a video review method, referring to Figure 3 , before step S20, further comprising: Step S60: extracting a key frame sequence of each illegal video in the illegal video library by using an adaptive key frame extraction algorithm.

[0105] The key frame sequence of each illegal video is extracted from the illegal video library to reduce the data volume and retain the core features, providing a basis for subsequent feature extraction and matching.

[0106] For example, an adaptive keyframe extraction algorithm is used to analyze the scene change frequency of the illegal videos in the illegal video library, dynamically adjust the sampling interval, and increase the sampling frequency in areas with high scene change frequency. Keyframes are extracted based on the dynamically adjusted sampling interval to form a keyframe sequence.

[0107] Optionally, a public illegal video dataset is obtained as an illegal video library. New illegal video samples are collected regularly to update the illegal video library.

[0108] Step S70 : Processing the key frame sequence of each of the illegal videos using a feature extraction network driven by a spatiotemporal state space model to generate an illegal feature vector for each of the illegal videos.

[0109] Process key frame sequences, generate violation feature vectors, fuse spatiotemporal information, and improve understanding of video content.

[0110] For example, a spatiotemporal state space model is constructed to define state transition and observation equations. The key frame sequence of each illegal video is input into a feature extraction network to extract visual features and generate an illegal feature vector for the illegal video.

[0111] Step S80: storing each of the violation feature vectors in a standardized format as a structured feature file to obtain the violation video feature library.

[0112] The violation feature vectors are stored as structured feature files for easy management and use.

[0113] Exemplarily, the violation feature vector is converted into a standardized format, and a structured feature file is created for each violation video, including video information and feature vectors. The feature file is stored in a database as a violation video feature library.

[0114] This embodiment provides a video review method. This embodiment first uses an adaptive algorithm to dynamically adjust the sampling interval according to changes in video content, ensuring that key information is not lost while reducing the number of sampling frames and improving processing efficiency. The spatiotemporal state space model can capture the dynamic changes of objects in the video in the spatiotemporal dimension, driving the feature extraction network to more accurately extract visual features in key frames, which helps to gain a deeper understanding of the video content. Storing the violation feature vector in a standardized format ensures the consistency and compatibility of the data in the violation video feature library, facilitating subsequent retrieval and use.

[0115] Based on Example 1, Example 4 of this application proposes a video review method, referring to Figure 4 , after step S30, including: Step S90: If it is detected that a new illegal video appears in the illegal video library, a new feature vector of the new illegal video is generated through a feature extraction network driven by a spatiotemporal state space model.

[0116] When new illegal videos are added to the illegal video library, their feature vectors are generated promptly to ensure the timeliness and accuracy of the feature library, allowing for the effective detection of new illegal content in the future. This provides a foundation for incremental updates to the feature library, avoiding audit blind spots caused by missing new data.

[0117] It should be noted that the newly added illegal videos are newly detected illegal video samples that are not included in the illegal video library.

[0118] For example, the newly added illegal video is decoded into a frame sequence, the key frame size is adjusted and the pixel values ​​are normalized, and an adaptive sampling algorithm is used to extract the key frame sequence. The key frame sequence is input into a pre-trained spatiotemporal feature extraction network to generate a spatiotemporal joint feature vector for the newly added illegal video.

[0119] Step S100: inserting the newly added feature vector into a vector search engine constructed based on a hierarchical navigable graph, and dynamically adjusting the node connection relationship through an incremental graph structure optimization algorithm to update the illegal video feature library.

[0120] The newly added feature vectors are inserted into the vector retrieval engine built on the hierarchical navigable graph, and the node connection relationship is dynamically adjusted to maintain the efficiency of the graph structure.

[0121] Optionally, if the illegal video feature library is empty, the hierarchical navigable graph index is initialized; otherwise, an existing index is loaded.

[0122] For example, the newly added feature vectors are used as new nodes, and the new nodes are inserted into the appropriate position of the hierarchical navigable graph through a greedy algorithm. The connection relationship between the new node and the existing nodes is dynamically adjusted based on the vector similarity using an incremental graph structure optimization algorithm.

[0123] Step S110: Based on the updated illegal video feature library, the input video is reviewed for illegal activities.

[0124] Utilize the updated illegal video feature library to conduct a comprehensive violation review of input videos to ensure that all illegal content can be accurately identified and blocked.

[0125] For example, feature extraction is performed on the input video to generate a multimodal feature vector. The feature vector of the input video is then similarly matched with feature vectors in the updated illegal video feature library. Based on the similarity matching results, a determination is made as to whether the input video contains illegal content, and appropriate review operations are performed.

[0126] This embodiment provides a video review method. By generating feature vectors for newly added illegal videos, this method ensures that the illegal video feature library is updated promptly and contains the latest illegal features. This helps to quickly identify new illegal content and more accurately detect illegal content. When new illegal content appears, a rapid response is achieved, requiring only an update of the video vector library, without the need to retrain the model.

[0127] For example, in order to help understand the implementation process of the video review method obtained by combining this embodiment with the above embodiment 1, please refer to Figure 5 , Figure 5 A brief flowchart of a video review method is provided, specifically: First, the system receives the input marketing video for review, decodes and segments it, and determines its multimodal data. Using a multimodal video feature extraction model, it parses this multimodal data and generates a semantically understood representation of the video content. Based on a preset violation detection prompt template, it generates dynamic query commands (prompt words) and performs targeted content review on the semantically understood representation produced by the multimodal video feature extraction model (VideoLLaMA2 model). The system then returns a judgment result, determining whether any content violates the law.

[0128] If there is obvious illegal content, the video blocking operation will be directly triggered and the blocking instruction of the input video will be generated.

[0129] If there is no obvious illegal content, the input video is dynamically sampled using a video keyframe extraction algorithm (adaptive algorithm) to generate a keyframe sequence. A visual feature extraction network (MambaVision) driven by a spatiotemporal state space model is used to process the keyframe sequence, extract features, and generate a multimodal feature vector. Feature matching is then performed based on Faiss and illegal video features in the marketing video feature library (illegal video feature library).

[0130] Optionally, a video keyframe extraction algorithm (adaptive algorithm) is used to extract the keyframe sequence of each illegal marketing video in the marketing video library (illegal video library). A feature extraction network (MambaVision) driven by a spatiotemporal state-space model is used to process the keyframe sequences of the illegal videos, extract features, and generate a spatiotemporal joint feature vector for the illegal videos. This spatiotemporal joint feature vector is stored as a structured feature file in a standardized format (h5) as the marketing video feature library (illegal video feature library).

[0131] Specifically, a hierarchical navigable graph is constructed for the vectors in the marketing video feature library (the illegal video feature library). Connections are established between nodes in each layer of the graph based on vector similarity. Starting from the top layer of the hierarchical navigable graph, a layer-by-layer search is performed, using a dynamic path optimization strategy to select candidate sets that are most similar to the multimodal feature vectors. Similarity matching results are generated based on the similarity scores of the candidate sets and a preset dynamic threshold.

[0132] Furthermore, based on the similarity matching result, it is determined whether there are any illegal video features whose similarity scores exceed a threshold value, so as to determine whether the input marketing video to be reviewed matches the illegal video features.

[0133] If there are matching illegal video features, the marketing video to be reviewed is determined to be illegal, and the video blocking operation is directly triggered, generating an interception instruction for the input video.

[0134] If there are no matching illegal video features, the marketing video to be reviewed is determined not to be illegal and the next step can be carried out normally.

[0135] It should be noted that the above examples are only used to understand this application and do not constitute a limitation on the video review method of this application. More simple transformations based on this technical concept are all within the scope of protection of this application.

[0136] The present application provides a video review device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video review method in the above-mentioned embodiment one.

[0137] Reference below Figure 6 , which shows a schematic diagram of the structure of a video audit device suitable for implementing the embodiments of the present application. The video audit device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, personal digital assistants (PDAs), tablet computers (PADs), etc., as well as fixed terminals such as desktop computers. Figure 6 The video review device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0138] like Figure 6As shown, the video review device may include a processing device 1001 (e.g., a central processing unit, graphics processing unit, etc.), which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. RAM 1004 also stores various programs and data required for the operation of the video review device. Processing device 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to I / O interface 1006: input devices 1007, such as a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008, such as a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 1003, such as a magnetic tape or hard disk; and communication devices 1009. Communication device 1009 can allow the video audit device to communicate with other devices wirelessly or wired to exchange data. Although the figure shows a video audit device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have instead.

[0139] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device 1003, or installed from a read-only memory 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are performed.

[0140] The video review device provided in this application, using the video review method in the above-mentioned embodiment, can solve the technical problem of a high rate of missed detection of illegal content. Compared with the prior art, the beneficial effects of the video review device provided in this application are the same as the beneficial effects of the video review method provided in the above-mentioned embodiment, and the other technical features of the video review device are the same as those disclosed in the method of the previous embodiment, which will not be repeated here.

[0141] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0142] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0143] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, wherein the computer-readable program instructions are used to execute the video review method in the above-mentioned embodiment.

[0144] The computer-readable storage medium provided herein may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including, but not limited to, wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0145] The computer-readable storage medium may be included in the video review device; or it may exist independently without being assembled into the video review device.

[0146] The above-mentioned computer-readable storage medium carries one or more programs, and when the above-mentioned one or more programs are executed by the video audit device, the video audit device can write computer program codes for performing the operations of the present application in one or more programming languages ​​or a combination thereof, and the above-mentioned programming languages ​​include object-oriented programming languages ​​- such as Java, Smalltalk, C++, and also include conventional procedural programming languages ​​- such as "C" language or similar programming languages. The program code can be executed entirely on the user computer, partially on the user computer, as an independent software package, partially on the user computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user computer through any type of network - including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0147] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0148] The modules described in the embodiments of the present application may be implemented in software or hardware, wherein the name of a module does not necessarily limit the unit itself.

[0149] The computer-readable storage medium provided in this application is a computer-readable storage medium storing computer-readable program instructions (i.e., a computer program) for executing the aforementioned video review method, thereby resolving the technical issue of a high rate of missed detection of illegal content. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the video review method provided in the aforementioned embodiments, and are not further elaborated here.

[0150] The above description is only part of the embodiments of the present application and does not limit the patent scope of the present application. All equivalent structural transformations made by using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect application in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A video review method, characterized in that: The video review method includes: After receiving the input video, extracting visual features of the input video to form a feature vector; Input the feature vector into a vector search engine built based on a hierarchical navigable graph, perform a nearest neighbor search and match with the vectors in the illegal video feature library, and determine a similarity matching result; An interception instruction for the input video is generated according to the similarity matching result.

2. The video review method according to claim 1, wherein: After receiving the input video, before extracting the visual features of the input video to form a feature vector, the method includes: Extracting a semantic understanding representation of the input video through a multimodal video feature extraction model, and performing a first-level illegal content review on the semantic understanding representation; According to the result of the first-level illegal content review, the first-level video interception operation is directly triggered to generate an interception instruction for the input video.

3. The video review method according to claim 2, wherein: The steps of extracting the semantic understanding representation of the input video through the multimodal video feature extraction model and performing a first-level illegal content review on the semantic understanding representation include: receiving the input video, decoding and segmenting the input video, and determining multimodal data of the input video; Parsing the multimodal data through the multimodal video feature extraction model to generate the semantic understanding representation of the video content; A dynamic query instruction is generated based on a preset violation detection prompt template, and a targeted content review is performed on the semantic understanding representation to determine whether there is any illegal content.

4. The video review method according to claim 1, wherein: After receiving the input video, the step of extracting visual features of the input video to form a feature vector includes: Dynamically sampling the input video using an adaptive key frame extraction algorithm to obtain the key frame sequence; The key frame sequence is input into a visual feature extraction network driven by a spatiotemporal state space model, visual features of each key frame in the key frame sequence are extracted, and a feature vector of the key frame sequence is generated.

5. The video review method according to claim 1, wherein: The step of inputting the feature vector into a vector search engine constructed based on a hierarchical navigable graph, performing a nearest neighbor search and matching with the vectors in the illegal video feature library, and determining a similarity matching result comprises: Establishing the hierarchical navigable graph for the vectors in the illegal video feature library, and establishing a connection relationship between nodes in each layer of the hierarchical navigable graph based on vector similarity; Starting from the top layer of the hierarchical navigable graph, searching layer by layer, and screening the candidate set most similar to the feature vector through a dynamic path optimization strategy; A similarity matching result is generated based on the similarity score of the candidate set and a preset dynamic threshold.

6. The video review method according to claim 1, wherein: Before the step of inputting the feature vector into a vector search engine constructed based on a hierarchical navigable graph and performing a neighbor search and matching with the vectors in the illegal video feature library to determine the similarity matching result, the method further includes: Extracting a key frame sequence of each illegal video in the illegal video library by an adaptive key frame extraction algorithm; Processing a key frame sequence of each of the illegal videos using a feature extraction network driven by a spatiotemporal state space model to generate an illegal feature vector for each of the illegal videos; Each of the violation feature vectors is stored in a standardized format as a structured feature file to obtain the violation video feature library.

7. The video review method according to claim 1, wherein: After the step of generating an interception instruction for the input video according to the similarity matching result, the method includes: If a new illegal video is detected in the illegal video library, a new feature vector of the new illegal video is generated through a feature extraction network driven by a spatiotemporal state space model; Inserting the newly added feature vector into a vector retrieval engine built based on a hierarchical navigable graph, and dynamically adjusting the node connection relationship through an incremental graph structure optimization algorithm to update the illegal video feature library; Based on the updated illegal video feature library, the input video is reviewed for violations.

8. The video review method according to claim 1, wherein: The step of generating an interception instruction for the input video according to the similarity matching result includes: Determine, based on the similarity matching result, a violation feature vector in a matched violation video feature library, and extract the violation type of the violation feature vector; Based on the timestamp information of the input video, intercepting the key frame sequence of the illegal segment to generate a screenshot of the illegal content; Extracting multimodal data of the violation feature vector in the same time period according to the timestamp information, and generating a violation feature comparison heat map; The interception instruction is constructed based on the violation type, and the screenshot of the violation content and the violation feature comparison heat map are rendered to the review interface.

9. A video review device, characterized in that: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the video review method according to any one of claims 1 to 8.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the video review method according to any one of claims 1 to 8 are implemented.

Images