Video content auditing method and device, equipment, storage medium and program product

Abstract

The invention relates to a video content auditing method and device, equipment, a storage medium and a program product. The method comprises the following steps: acquiring a to-be-audited video input by a user; performing frame interception on the to-be-audited video, and determining continuous multi-frame video image frames; respectively inputting the multiple video image frames into a multi-modal content auditing model, and obtaining a content auditing result of the to-be-audited video output by the multi-modal content auditing model; wherein the content auditing result is used for indicating whether the to-be-audited video contains content risk data and / or the risk type of the content risk data, and the multi-modal content auditing model is used for carrying out feature fusion on image features of the multiple video image frames and carrying out content risk identification on the fused features by using a text decoder. By adopting the method, the video content auditing duration can be shortened.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to a video content review method, device, equipment, storage medium and program product. Background Art

[0002] With the rapid development of Internet technology and the widespread popularity of social media platforms, the amount of user-generated content has exploded. As an intuitive and vivid information carrier, images occupy an important position in social media, advertising, news and other fields. The diversity of image content also brings challenges to content review. How to accurately and efficiently identify bad image content has become an urgent problem to be solved.

[0003] In related technologies, for content review in the form of videos, the frame extraction method is usually used to use the model to infer the review results for each frame in the video, and then the review results of all frames are summarized by taking the average, maximum value, etc. However, when performing content review on videos by using the frame extraction method, it usually takes a long time to infer the review results, which cannot meet the needs of large data volume or high concurrency. Summary of the invention

[0004] Based on this, it is necessary to provide a video content review method, device, equipment, storage medium and program product that can reduce the time of video content review in response to the above technical problems.

[0005] In a first aspect, the present application provides a video content review method, comprising:

[0006] Get the video to be reviewed input by the user;

[0007] Cutting frames of the video to be reviewed to determine a plurality of continuous video image frames;

[0008] Input the multiple video image frames into the multimodal content review model respectively, and obtain the content review result of the video to be reviewed output by the multimodal content review model;

[0009] Among them, the content review result is used to indicate whether the video to be reviewed contains content risk data and / or the risk type of the content risk data, and the multimodal content review model is used to perform feature fusion on the image features of the multiple video image frames and use a text decoder to identify content risks on the fused features.

[0010] In one of the embodiments, the multimodal content audit model includes an image feature encoding module, a temporal feature fusion module, and a text decoding module;

[0011] The image feature encoding module is used to extract features from the multiple video image frames respectively through a visual encoder to obtain image features of the multiple video image frames;

[0012] The temporal feature fusion module is used to use an attention mechanism to perform feature fusion on the image features of the multiple video image frames to obtain the fused features;

[0013] The text decoding module is used to use the text decoder to perform text decoding on the fused features to obtain the content review result of the video to be reviewed.

[0014] In one of the embodiments, the image feature encoding module is further used to determine the position encoding information corresponding to the multiple video image frames respectively according to the timing information of the multiple video image frames, and add the position encoding information to the image features of the multiple video image frames.

[0015] In one of the embodiments, the temporal feature fusion module is also used to calculate the similarity between adjacent image features if the number of image features of the multiple video image frames is greater than the fixed feature number of the temporal feature fusion module, and compress the image features of the multiple video image frames based on the similarity between the adjacent image features.

[0016] In one of the embodiments, the temporal feature fusion module is further used to use zero features to pad the image feature numbers of the multiple video image frames to the fixed feature number if the image feature number of the multiple video image frames is less than the fixed feature number of the temporal feature fusion module.

[0017] In one embodiment, the method further comprises:

[0018] Acquire a training sample set, wherein the training sample set includes a plurality of video image frames corresponding to a sample video and a standard content review text corresponding to the sample video;

[0019] Using the training sample set to train the multimodal content review model;

[0020] Calculating loss data between actual content review results output by the multimodal content review model during training and standard content review texts through a cross entropy loss function;

[0021] Based on the loss data, the model parameters of the multimodal content review model are optimized.

[0022] In a second aspect, the present application provides a video content review device, comprising:

[0023] The acquisition module is used to obtain the video to be reviewed input by the user;

[0024] A frame capture module, used to capture frames of the video to be reviewed and determine a plurality of continuous video image frames;

[0025] An identification module, used to input the multiple video image frames into a multimodal content review model respectively, and obtain a content review result of the video to be reviewed output by the multimodal content review model;

[0026] Among them, the content review result is used to indicate whether the video to be reviewed contains content risk data and / or the risk type of the content risk data, and the multimodal content review model is used to perform feature fusion on the image features of the multiple video image frames and use a text decoder to identify content risks on the fused features.

[0027] In one of the embodiments, the multimodal content audit model includes an image feature encoding module, a temporal feature fusion module, and a text decoding module;

[0028] The image feature encoding module is used to extract features from the multiple video image frames respectively through a visual encoder to obtain image features of the multiple video image frames;

[0029] The temporal feature fusion module is used to use an attention mechanism to perform feature fusion on the image features of the multiple video image frames to obtain the fused features;

[0030] The text decoding module is used to use the text decoder to perform text decoding on the fused features to obtain the content review result of the video to be reviewed.

[0031] In one of the embodiments, the image feature encoding module is further used to determine the position encoding information corresponding to the multiple video image frames respectively according to the timing information of the multiple video image frames, and add the position encoding information to the image features of the multiple video image frames.

[0032] In one of the embodiments, the temporal feature fusion module is also used to calculate the similarity between adjacent image features if the number of image features of the multiple video image frames is greater than the fixed feature number of the temporal feature fusion module, and compress the image features of the multiple video image frames based on the similarity between the adjacent image features.

[0033] In one of the embodiments, the temporal feature fusion module is further used to use zero features to pad the image feature numbers of the multiple video image frames to the fixed feature number if the image feature number of the multiple video image frames is less than the fixed feature number of the temporal feature fusion module.

[0034] In one embodiment, the video content review device further includes:

[0035] A training module is used to obtain a training sample set, which includes multiple video image frames corresponding to a sample video and a standard content review text corresponding to the sample video; use the training sample set to train the multimodal content review model; calculate the loss data between the actual content review results output by the multimodal content review model during training and the standard content review text through a cross-entropy loss function; and optimize the model parameters of the multimodal content review model based on the loss data.

[0036] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned video content review method when executing the computer program.

[0037] In a fourth aspect, the present application also provides a computer-readable storage medium having a computer program stored thereon, and the computer program implements the above-mentioned video content review method when executed by a processor.

[0038] In a fifth aspect, the present application also provides a computer program product, including a computer program, which implements the above-mentioned video content review method when executed by a processor.

[0039] The above-mentioned video content review method, device, equipment, storage medium and program product first obtain the video to be reviewed input by the user. Then, the video to be reviewed is framed to determine a continuous multi-frame video image frame. Finally, the multiple video image frames are respectively input into the multimodal content review model, and the content review result of the video to be reviewed output by the multimodal content review model is obtained. Among them, the content review result is used to indicate whether the video to be reviewed contains content risk data and / or the risk type of content risk data, and the multimodal content review model is used to perform feature fusion on the image features of the multiple video image frames and use the text decoder to identify the content risk of the fused features. Because the multimodal content review model first performs feature fusion on the image features of the multiple video image frames in the video to be reviewed, and then performs content risk identification on the fused features, there is no need to identify the video to be reviewed frame by frame, thereby reducing the video content review time. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0041] Figure 1 An application environment diagram of a video content review method provided in an embodiment of the present application;

[0042] Figure 2 A flowchart of a video content review method provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of the structure of a multimodal content review model provided in an embodiment of the present application;

[0044] Figure 4 A flowchart of another video content review method provided in an embodiment of the present application;

[0045] Figure 5 A structural block diagram of a video content review device provided in an embodiment of the present application;

[0046] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] The video content review method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0049] When video content review is required, the terminal 102 can send a video review request to the server 104, and the video review request includes the video to be reviewed input by the user. After the server 104 obtains the video to be reviewed input by the user, it can cut frames of the video to be reviewed and determine a continuous multi-frame video image frame. Subsequently, the server 104 inputs the multiple video image frames into the multimodal content review model respectively, and obtains the content review result of the video to be reviewed output by the multimodal content review model. Among them, the content review result is used to indicate whether the video to be reviewed contains content risk data and / or the risk type of content risk data, and the multimodal content review model is used to perform feature fusion on the image features of multiple video image frames and use a text decoder to perform content risk identification on the fused features.

[0050] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0051] In an exemplary embodiment, Figure 2 As shown, a video content review method is provided, which is applied to Figure 1 The following is an example of the server in Figure 1, including S201-S203:

[0052] S201, obtaining a video to be reviewed input by a user.

[0053] In the present application, when it is necessary to review the video content, the terminal can send the video to be reviewed input by the user to the server, so that the server obtains the video to be reviewed input by the user.

[0054] It should be understood that the embodiments of the present application do not limit the videos to be reviewed. By way of example, the videos to be reviewed may include video files, surveillance video streams, and the like.

[0055] S202: Cut frames of the video to be reviewed to determine a plurality of continuous video image frames.

[0056] In this step, after the server obtains the video to be reviewed input by the user, it can cut frames of the video to be reviewed to determine a plurality of continuous video image frames.

[0057] It should be understood that since a video is composed of a series of continuous image frames, each frame can be regarded as a static picture. Therefore, by cutting frames, a single frame at each moment can be extracted from the video to be verified and saved as a separate video image frame.

[0058] It should be understood that the embodiments of the present application do not limit how to capture frames of the video to be reviewed. In some embodiments, the configuration parameters of the capture tool can be preset, and then the capture tool is used to capture frames of the video to be reviewed to determine a continuous multi-frame video image frame. The configuration parameters of the capture tool can include the size of the captured video image frame, the position in the video, etc.

[0059] S203. Input multiple video image frames into the multimodal content review model respectively, and obtain the content review result of the video to be reviewed output by the multimodal content review model.

[0060] In this step, after cutting frames of the video to be reviewed and determining multiple continuous video image frames, the multiple video image frames can be input into the multimodal content review model respectively, and the content review result of the video to be reviewed output by the multimodal content review model can be obtained.

[0061] The above-mentioned content review result is used to indicate whether the video to be reviewed contains content risk data and / or the risk type of the content risk data.

[0062] It should be understood that the embodiments of the present application do not limit the risk type of content risk data, which may include prohibited, violent, terrorist, sensitive and other content risk types.

[0063] Exemplarily, the content review result may indicate that the video to be reviewed does not contain content risk data, or the content review result may also indicate that the video to be reviewed does not contain content risk data and indicate a specific content risk type.

[0064] It should be understood that the above-mentioned multimodal content review model is a multimodal model. The multimodal model can usually convert data of other modalities into the input form of a large language model through a corresponding encoder, and can be spliced ​​with the text input and then use the large language model to infer the output.

[0065] The multimodal content review model is explained below.

[0066] Figure 3 A schematic diagram of the structure of a multimodal content review model provided in an embodiment of the present application, such as Figure 3 As shown, the multimodal content review model includes an image feature encoding module, a temporal feature fusion module and a text decoding module.

[0067] In some embodiments, the image feature encoding module is used to extract features from multiple video image frames respectively through a visual encoder to obtain image features of the multiple video image frames.

[0068] Among them, the above-mentioned visual encoder is a neural network structure used to convert image or video data into potential represented image features for subsequent analysis and processing.

[0069] Exemplarily, the visual encoder may be a pre-trained deep learning model, which may include a convolutional neural network model or a vision transformer network model, etc.

[0070] Exemplarily, continuous multiple video image frames can be input into the multimodal content review model frame by frame, and the image feature encoding module in the multimodal content review model extracts image features frame by frame to obtain image features of the multiple video image frames.

[0071] In other embodiments, the image feature encoding module is further used to determine the position encoding information corresponding to the multiple video image frames according to the timing information of the multiple video image frames, and add the position encoding information to the image features of the multiple video image frames.

[0072] It should be understood that since it is a continuous multi-frame video image frame, each video image frame contains timing information. Therefore, the image feature encoding module can determine the position encoding information corresponding to the multiple video image frames according to the timing information of the multiple video image frames, and then add the position encoding information to the image features of the multiple video image frames.

[0073] For example, for the video image frame of the Tth frame, the image features are extracted by the visual encoder After that, the image features of the video image frame can be obtained from the image features of the video image frame Add position encoding information in , and accordingly, the output of the image feature encoding module can be , where PE is the position encoder.

[0074] In some embodiments, the temporal feature fusion module is used to use an attention mechanism to fuse image features of multiple video image frames to obtain fused features.

[0075] Among them, the Attention Mechanism is a deep learning technology. When processing sequence data or image data, the Attention Mechanism simulates the attention allocation of humans in the information processing process, enabling the temporal feature fusion module to focus on the important parts of the input data while ignoring the unimportant parts.

[0076] Exemplarily, the image features of multiple video image frames output by the image feature encoding module can be input to the temporal feature fusion module for feature fusion. The temporal feature fusion module includes a feature group, through which the image features of multiple video image frames can be saved in time sequence. Subsequently, the image features of multiple video image frames in the feature group saved in time sequence can be spliced. After the splicing is completed, the learnable query (Query) sub-model with the same length as the feature group in the temporal feature fusion module is used to perform attention calculation with the spliced ​​image features. After the attention calculation is completed, the fused features can be obtained.

[0077] For example, the length of the feature group of the temporal feature fusion module is K, and the feature group can store the most recent K image features. When the temporal feature fusion module is inferring, after splicing the features corresponding to multiple frames of images in the feature group, the spliced ​​image features obtained can be shown as shown in formula (1).

[0078] (1)

[0079] in, is the image feature of the t-th video image frame, To stitch the completed image features, , C is the dimension of the token vector, and P is the number of tokens corresponding to a single video image frame.

[0080] For example, the attention calculation using the query sub-model and the concatenated image features can be performed as shown in formulas (2) and (3).

[0081] (2)

[0082] (3)

[0083] in, , , are the weights of the corresponding three paths in the sub-attention module, Q, K, V are the outputs of the three paths, O is the result of the sub-attention module fusing QKV, and c in the equation is the dimension of each vector.

[0084] It should be noted that the fused features output by the above-mentioned temporal feature fusion module include the characteristic features of continuous multi-frame video image frames, but the output size of the fused features is independent of the sequence length of the multi-frame video image frames input by the multimodal content review model.

[0085] In other embodiments, the temporal feature fusion module is also used to calculate the similarity between adjacent image features if the number of image features of multiple video image frames is greater than the fixed feature number of the temporal feature fusion module, and compress the image features of multiple video image frames based on the similarity between adjacent image features.

[0086] The number of fixed features of the temporal feature fusion module may be equal to the length of the feature group in the temporal feature fusion module, which is not limited in the present embodiment. For example, the number of fixed features of the temporal feature fusion module may be 64, 32, 16, etc.

[0087] For example, for a newly input video image frame, after the image feature is extracted by the image feature encoding module and the position code is added, the image feature can be added to the feature group in the temporal feature fusion module. If the length of the feature group is K (i.e., the fixed number of features is K), after adding the new image feature, the length of the feature group is K+1. At this time, the similarity between all adjacent features can be calculated, and the two adjacent features corresponding to the maximum similarity can be found, and the two adjacent features can be fused to maintain the number of image features in the feature group unchanged.

[0088] Exemplarily, when the number of image features of a multi-frame video image frame is 32 and the sequence length of continuous multi-frame video image frames is greater than 32, when the input video image frames are greater than 32 frames, the temporal feature fusion module can perform feature compression to fuse the features of adjacent video image frames with the greatest similarity, and maintain the feature group length of the temporal feature fusion module at 32.

[0089] In other embodiments, the temporal feature fusion module is also used to use zero features to pad the image feature numbers of multiple video image frames to the fixed feature number if the image feature number of the multiple video image frames is less than the fixed feature number of the temporal feature fusion module.

[0090] Exemplarily, the number of image features of the multi-frame video image frame is 32 and when the sequence length of the multi-frame video image frame is less than 32, the feature group after all the video image frames are sent to the temporal feature fusion module is not full, and 0 features are added to the unfull feature group.

[0091] It should be understood that when the temporal feature fusion module in the present application performs feature fusion, feature splicing can be performed in the form of feature groups, and the length of the feature group is fixed. When the length of the feature group (for example, K) is much smaller than the length of the video to be reviewed, if the image features before the length of the feature group (K frames) are discarded in a first-in-first-out manner, this will cause the loss of temporal information and make it impossible to model long videos. In the present application, the image features of multiple video frames are compressed by the temporal feature fusion module, so that the video memory occupancy can be kept unchanged. At the same time, adjacent similar features are compressed as much as possible during the compression process, which can also minimize information loss, ensure the accuracy of model recognition, and improve the concurrency capability of the model.

[0092] In some embodiments, the text decoding module is used to use a text decoder to perform text decoding on the fused features to obtain the content review results of the video to be reviewed.

[0093] The text decoder may be a text decoder in a pre-trained large language model. The input of the text decoder may be image features, and the output may be text data, which is the content review result of the video to be reviewed.

[0094] It should be understood that the above-mentioned large language model can be a deep learning model with a large number of parameters and complex structure. The model is learned through a large-scale training data set and aims to complete complex tasks such as text recognition and text generation.

[0095] The following explains how to train a multimodal content review model.

[0096] In some embodiments, the server may first obtain a training sample set. Secondly, the server may use the training sample set to train the multimodal content audit model. Thirdly, the server may calculate the loss data between the actual content audit results output by the multimodal content audit model during training and the standard content audit text through a cross entropy loss function. Finally, the server may optimize the model parameters of the multimodal content audit model based on the loss data.

[0097] The training sample set includes multiple video image frames corresponding to the sample video and standard content review text corresponding to the sample video.

[0098] It should be understood that the embodiment of the present application does not limit the above training sample set, and different sample videos can be collected through multiple data sources. Subsequently, the collected different sample videos can be framed to determine the multiple video image frames corresponding to the different sample videos. Subsequently, the annotated text of each sample video can be received and used as the standard content review text corresponding to the sample video.

[0099] The above loss data may be the cross entropy of a cross entropy loss function.

[0100] It should be understood that the cross-entropy loss function is a commonly used loss function in classification problems. The cross-entropy loss function measures the difference between the probability distribution predicted by the model and the true probability distribution, and is used to evaluate the accuracy of the model's prediction. The smaller the cross entropy calculated by the cross-entropy loss function, the closer the probability distribution predicted by the model is to the true distribution. Conversely, the larger the cross entropy, the greater the deviation between the two distributions.

[0101] The video content review method provided in the embodiment of the present application first obtains the video to be reviewed input by the user. Subsequently, the video to be reviewed is framed to determine a continuous multi-frame video image frame. Finally, the multi-frame video image frames are respectively input into the multimodal content review model, and the content review result of the video to be reviewed output by the multimodal content review model is obtained. Among them, the content review result is used to indicate whether the video to be reviewed contains content risk data and / or the risk type of content risk data, and the multimodal content review model is used to perform feature fusion on the image features of the multi-frame video image frames and use the text decoder to identify the content risk of the fused features. Since the multimodal content review model first performs feature fusion on the image features of the multi-frame video image frames in the video to be reviewed, and then identifies the content risk of the fused features, there is no need to identify the video to be reviewed frame by frame, thereby reducing the video content review time. In addition, through this video content review method, the problem of linear calculation amount and time-consuming delay caused by frame-by-frame processing can be avoided.

[0102] The following explains how to train a multimodal content review model. Figure 4 A flowchart of another video content review method provided in an embodiment of the present application is shown as follows: Figure 4 As shown, the video content review method includes S301-S307:

[0103] S301. Obtain a training sample set, where the training sample set includes multiple video image frames corresponding to a sample video and a standard content review text corresponding to the sample video.

[0104] S302. Use the training sample set to train the multimodal content review model.

[0105] S303. Calculate the loss data between the actual content review results output by the multimodal content review model during training and the standard content review text through the cross entropy loss function.

[0106] S304. Optimize model parameters of the multimodal content review model based on the loss data.

[0107] S305: Obtain the video to be reviewed input by the user.

[0108] S306: Cut frames of the video to be reviewed to determine a plurality of continuous video image frames.

[0109] S307. Input multiple video image frames into the multimodal content review model respectively, and obtain the content review result of the video to be reviewed output by the multimodal content review model.

[0110] Among them, the content review result is used to indicate whether the video to be reviewed contains content risk data and / or the risk type of content risk data. The multimodal content review model is used to fuse the image features of multiple video image frames and use a text decoder to identify content risks on the fused features.

[0111] The video content review method provided in the embodiment of the present application first obtains the video to be reviewed input by the user. Subsequently, the video to be reviewed is framed to determine a continuous multi-frame video image frame. Finally, the multi-frame video image frames are respectively input into the multimodal content review model, and the content review result of the video to be reviewed output by the multimodal content review model is obtained. Among them, the content review result is used to indicate whether the video to be reviewed contains content risk data and / or the risk type of content risk data, and the multimodal content review model is used to perform feature fusion on the image features of the multi-frame video image frames and use the text decoder to identify the content risk of the fused features. Because the multimodal content review model first performs feature fusion on the image features of the multi-frame video image frames in the video to be reviewed, and then performs content risk identification on the fused features, there is no need to identify the video to be reviewed frame by frame, thereby reducing the video content review time.

[0112] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0113] Based on the same inventive concept, the embodiment of the present application also provides a video content review device for implementing the video content review method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more video content review device embodiments provided below can refer to the limitations of the video content review method above, and will not be repeated here.

[0114] In an exemplary embodiment, Figure 5 As shown, a video content review device 400 is provided, comprising: an acquisition module 401, a frame capture module 402 and an identification module 403, wherein:

[0115] The acquisition module 401 is used to acquire the video to be reviewed input by the user.

[0116] The frame capture module 402 is used to capture frames of the video to be reviewed and determine a plurality of continuous video image frames.

[0117] The identification module 403 is used to input multiple video image frames into the multimodal content review model respectively, and obtain the content review result of the video to be reviewed output by the multimodal content review model;

[0118] Among them, the content review result is used to indicate whether the video to be reviewed contains content risk data and / or the risk type of content risk data. The multimodal content review model is used to fuse the image features of multiple video image frames and use a text decoder to identify content risks on the fused features.

[0119] In one of the embodiments, the multimodal content review model includes an image feature encoding module, a temporal feature fusion module, and a text decoding module.

[0120] The image feature encoding module is used to extract features of multiple video image frames respectively through a visual encoder to obtain image features of the multiple video image frames.

[0121] The temporal feature fusion module is used to use the attention mechanism to fuse the image features of multiple video image frames to obtain fused features.

[0122] The text decoding module is used to use a text decoder to perform text decoding on the fused features to obtain the content review results of the video to be reviewed.

[0123] In one of the embodiments, the image feature encoding module is further used to determine the position encoding information corresponding to the multiple video image frames according to the timing information of the multiple video image frames, and add the position encoding information to the image features of the multiple video image frames.

[0124] In one of the embodiments, the temporal feature fusion module is also used to calculate the similarity between adjacent image features if the number of image features of multiple video image frames is greater than the fixed feature number of the temporal feature fusion module, and compress the image features of multiple video image frames based on the similarity between adjacent image features.

[0125] In one of the embodiments, the temporal feature fusion module is further used to use zero features to pad the image feature numbers of the multiple video image frames to the fixed feature number if the image feature number of the multiple video image frames is less than the fixed feature number of the temporal feature fusion module.

[0126] In one embodiment, the video content review device further includes:

[0127] A training module is used to obtain a training sample set, which includes multiple video image frames corresponding to a sample video and a standard content review text corresponding to the sample video; use the training sample set to train a multimodal content review model; calculate the loss data between the actual content review results output by the multimodal content review model during training and the standard content review text through a cross-entropy loss function; and optimize the model parameters of the multimodal content review model based on the loss data.

[0128] Each module in the above-mentioned video content review device can be implemented in whole or in part by software, hardware and their combination. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each of the above modules.

[0129] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a video content review method is implemented.

[0130] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0131] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned video content review method when executing the computer program.

[0132] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned video content review method is implemented.

[0133] In one embodiment, a computer program product is provided, comprising a computer program, which implements the above-mentioned video content review method when executed by a processor.

[0134] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0135] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0136] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A video content review method, characterized in that: The method comprises: Get the video to be reviewed input by the user; Cutting frames of the video to be reviewed to determine a plurality of continuous video image frames; Input the multiple video image frames into the multimodal content review model respectively, and obtain the content review result of the video to be reviewed output by the multimodal content review model; Among them, the content review result is used to indicate whether the video to be reviewed contains content risk data and / or the risk type of the content risk data, and the multimodal content review model is used to perform feature fusion on the image features of the multiple video image frames and use a text decoder to identify content risks on the fused features.

2. The method according to claim 1, characterized in that: The multimodal content audit model includes an image feature encoding module, a temporal feature fusion module and a text decoding module; The image feature encoding module is used to extract features from the multiple video image frames respectively through a visual encoder to obtain image features of the multiple video image frames; The temporal feature fusion module is used to use an attention mechanism to perform feature fusion on the image features of the multiple video image frames to obtain the fused features; The text decoding module is used to use the text decoder to perform text decoding on the fused features to obtain the content review result of the video to be reviewed.

3. The method according to claim 2, characterized in that The image feature encoding module is further used to determine the position encoding information corresponding to the multiple video image frames respectively according to the timing information of the multiple video image frames, and add the position encoding information to the image features of the multiple video image frames.

4. The method according to claim 3, characterized in that The temporal feature fusion module is also used to calculate the similarity between adjacent image features if the number of image features of the multiple video image frames is greater than the fixed feature number of the temporal feature fusion module, and compress the image features of the multiple video image frames according to the similarity between the adjacent image features.

5. The method according to claim 3, characterized in that: The temporal feature fusion module is also used to use zero features to pad the image feature numbers of the multiple video image frames to the fixed feature number if the image feature number of the multiple video image frames is less than the fixed feature number of the temporal feature fusion module.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Obtaining a training sample set, wherein the training sample set includes a plurality of video image frames corresponding to a sample video and a standard content review text corresponding to the sample video; Using the training sample set to train the multimodal content review model; Calculating loss data between actual content review results output by the multimodal content review model during training and standard content review texts through a cross entropy loss function; Based on the loss data, the model parameters of the multimodal content review model are optimized.

7. A video content review device, characterized in that: The device comprises: The acquisition module is used to obtain the video to be reviewed input by the user; A frame capture module, used to capture frames of the video to be reviewed and determine a plurality of continuous video image frames; An identification module, used to input the multiple video image frames into a multimodal content review model respectively, and obtain a content review result of the video to be reviewed output by the multimodal content review model; Among them, the content review result is used to indicate whether the video to be reviewed contains content risk data and / or the risk type of the content risk data, and the multimodal content review model is used to perform feature fusion on the image features of the multiple video image frames and use a text decoder to identify content risks on the fused features.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images