Video review method, device, equipment and readable storage medium

Abstract

The present application discloses a video review method, device, equipment and readable storage medium. The electronic device obtains multiple frames of images from the video to be reviewed, and sequentially inputs the multiple frames of images into the video review model to obtain the recognition result of each frame of image and the confidence level of the recognition result, and determines whether the video to be reviewed is a violation video according to the recognition results and confidence levels of each frame of image. Among them, the recognition result of each frame of image is determined according to the output results of multiple branch networks included in the multi-object recognition network in the video review model. With this solution, the multi-object recognition network of the video review model includes multiple branch networks. Different branch networks are used to recognize different targets. Each branch network only focuses on the targets of its own branch. By refining the task targets, the learning is more targeted, with high accuracy and fast speed.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and particularly to a video review method, device, equipment and readable storage medium. Background Art

[0002] With the prosperous development of Internet technology, there are numerous video resources on the Internet. Video review is an important means to filter out bad or illegal content in the network video resource pool.

[0003] In the traditional technology, the reviewer plays the video to be reviewed (the video uploaded by the user) from beginning to end completely, and during the process of playing the video to be reviewed, the reviewer reviews it.

[0004] However, the method of manually reviewing videos is time-consuming, laborious, limited in efficiency, costly and unable to achieve large-scale video reviews. Summary of the Invention

[0005] This application provides a video review method, device, equipment and readable storage medium, which uses a video review model capable of identifying multiple different targets to review videos, with high accuracy and fast speed.

[0006] In a first aspect, an embodiment of this application provides a video review method, including:

[0007] Obtain multiple frames of images from the video to be reviewed;

[0008] Input the multiple frames of images into a video review model to obtain the recognition result and the confidence level of the recognition result for each frame of the multiple frames of images. The video review model includes a feature extraction network and a multi-target recognition network. The multi-target recognition network includes multiple branch networks, and different branch networks in the multiple branch networks are used to identify different targets. The recognition result of each frame of the multiple frames of images is determined according to the output result of each branch network in the multiple branch networks;

[0009] Determine whether the video to be reviewed is a violation video according to the recognition result and the confidence level of the recognition result for each frame of the multiple frames of images.

[0010] In a second aspect, an embodiment of this application provides a video review device, including:

[0011] An acquisition module, configured to obtain multiple frames of images from the video to be reviewed;

[0012] A processing module, configured to input the multiple frames of images into a video review model to obtain the recognition result of each frame of image in the multiple frames of images and the confidence of the recognition result. The video review model includes a feature extraction network and a multi-object recognition network. The multi-object recognition network includes multiple branch networks, and different branch networks in the multiple branch networks are used to recognize different objects. The recognition result of each frame of image in the multiple frames of images is determined according to the output result of each branch network in the multiple branch networks;

[0013] A determination module, configured to determine whether the video to be reviewed is a violation video according to the recognition result of each frame of image in the multiple frames of images and the confidence of the recognition result.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, including: a processor, a memory, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the electronic device implements the method described in the first aspect or various possible implementation manners of the first aspect above.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed by a processor, they are used to implement the method described in the first aspect or various possible implementation manners of the first aspect above.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product including a computing program. When the computer program is executed by a processor, it implements the method described in the first aspect or various possible implementation manners of the first aspect above.

[0017] For the video review method, device, equipment, and readable storage medium provided by the embodiments of the present application, the electronic device obtains multiple frames of images from the video to be reviewed, sequentially inputs the multiple frames of images into the video review model to obtain the recognition result of each frame of image and the confidence of the recognition result, and determines whether the video to be reviewed is a violation video according to the recognition result and confidence of each frame of image. Among them, the recognition result of each frame of image is determined according to the output results of multiple branch networks included in the multi-object recognition network of the video review model. With this solution, the multi-object recognition network of the video review model includes multiple branch networks, and different branch networks are used to recognize different objects. Each branch network only focuses on the object of its own branch. By refining the task objectives, the learning is more targeted, with high accuracy and fast speed. Description of the Drawings

[0018] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0019] Figure 1A It is a schematic diagram of the implementation environment of the video review method provided by the embodiments of the present application;

[0020] Figure 1B It is a network architecture diagram for generating a video review model in the video review method provided by the embodiments of the present application;

[0021] Figure 2 It is a flowchart of the video review method provided by the embodiments of the present application;

[0022] Figure 3 It is a schematic diagram of the process of training a video review model in the video review method provided by the embodiments of the present application;

[0023] Figure 4 It is a schematic diagram of the structure of video review in the video review method provided by the embodiments of the present application;

[0024] Figure 5 It is a schematic diagram of the structure of the feature extraction network in the video review model provided by the embodiments of the present application;

[0025] Figure 6 It is a schematic diagram of the structure of the fully connected classification layer of the multi-object recognition network in the video review model provided by the embodiments of the present application;

[0026] Figure 7 It is a schematic diagram of the structure of the attention module in the video review method provided by the embodiments of the present application;

[0027] Figure 8 It is a schematic diagram of the basic attention module in the video review method provided by the embodiments of the present application;

[0028] [[ID=3⑨]] Figure 9 It is a schematic diagram of the multi-feature fusion attention module in the video review method provided by the embodiments of the present application;

[0029] Figure 10 It is another schematic diagram of the multi-feature fusion attention module in the video review method provided by the embodiments of the present application;

[0030] Figure 11A It is a kind of heat map in the video review method provided by the embodiments of the present application;

[0031] Figure 11BIt is another heat map in the video review method provided by the embodiments of the present application;

[0032] Figure 12A It is yet another heat map in the video review method provided by the embodiments of the present application;

[0033] Figure 12B It is yet another heat map in the video review method provided by the embodiments of the present application;

[0034] Figure 13 It is another flowchart of the video review method provided by the embodiments of the present application;

[0035] Figure 14 It is a schematic diagram of a video review device provided by the embodiments of the present application;

[0036] Figure 15 It is a schematic diagram of another video review device provided by the embodiments of the present application;

[0037] Figure 16 It is a schematic diagram of the structure of an electronic device provided by the embodiments of the present application. Detailed implementation manners

[0038] To make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.

[0039] Currently, the number of videos on the Internet is increasing day by day, and most of them appear in various forms such as advertisements, video blogs (Vlogs), and short videos. With the rapid development of the fifth-generation (5G) and sixth-generation (6G) communication technologies and the Internet of Things, the number of videos has increased exponentially. Now many websites support and allow users to upload videos by themselves and display them to the public. The videos uploaded by users themselves may have problems, and these videos are usually called bad videos, illegal videos, etc. In order to filter out illegal videos, it is necessary to review the content of the videos (referred to as video review).

[0040] During the manual review process, the reviewer plays the video to be reviewed from beginning to end, so as to review the video to be reviewed. This method is inefficient and costly.

[0041] With the development of artificial intelligence (AI) technology, the method for identifying illegal content based on deep learning has gradually matured. However, on the one hand, the video review method based on deep learning relies on a large number of training samples, and the data of illegal content is relatively limited, resulting in a high cost of accumulating training samples. On the other hand, since different models have different definitions of illegal content, to a certain extent, it increases the difficulty of training.

[0042] Therefore, how to combine effective deep learning methods to construct an efficient video review method and minimize manual intervention as much as possible is an urgent problem to be solved.

[0043] Based on this, the embodiments of the present application provide a video review method, device, equipment and readable storage medium. By refining the task objectives, the learning becomes more targeted. A video review model capable of identifying multiple different targets is used to review videos, with high accuracy and speed. At the same time, an attention mechanism is introduced into the video review model. By combining attention mechanisms of different structures, the learning of each branch network of the multi-target recognition network in the video review model becomes more focused.

[0044] The embodiments of the present application include a video review method and a training method for a video review model, belonging to the categories of machine learning and deep learning in artificial intelligence technology. These two methods can be executed by the same or different electronic devices.

[0045] Figure 1A It is a schematic diagram of the implementation environment of the video review method provided by the embodiments of the present application. This implementation environment includes: a terminal device 11 and a server 12, with a network connection established between the terminal device 11 and the server 12. The user can send the video to be reviewed to the server 12 through the terminal device 11. The terminal device 11 can specifically be a mobile phone, a tablet computer, a notebook, a desktop computer, etc. The server 12 can be implemented by an independent server or a server cluster composed of multiple servers.

[0046] The video review model is deployed on the server 12. This video review model can be obtained by the server 12 training the initial model using a sample set by itself, or can be obtained by the server 12 from other electronic devices. The video review model is a model for a certain violation recognition service. The video review model can be a model that the server 12 trains the initial model using a large number of image samples and has not been accelerated, or a model accelerated by the server 12 using the Tensor RT model acceleration technology. The server 12 can construct the accelerated model into a service or a software development kit (SDK).

[0047] After receiving the video to be reviewed, the server 12 performs frame splitting on the video to be reviewed to obtain multiple frames of images, and inputs some or all of the images of the video to be reviewed into the video review model, so that the video review model outputs the recognition results and confidence levels of each frame of image. Then, the server 12 determines whether the video to be reviewed is a violation video based on the recognition results and confidence levels of each frame of image.

[0048] In the embodiments of the present application, the video review method is taken as an example applied to server 12, and is implemented through the interaction between the terminal device and the server. It can be understood that the video review method can also be applied to terminal devices, etc., and the embodiments of the present application do not limit this.

[0049] Figure 1B It is the network architecture diagram for generating a video review model in the video review method provided by the embodiments of the present application. Please refer to Figure 1B In one embodiment, the model generation system can be fully deployed in the cloud environment. The cloud environment is an entity that uses basic resources to provide cloud services to users under the cloud computing model. The cloud environment includes a cloud data center and a cloud service platform. Among them, the cloud data center includes a large number of basic resources owned by the cloud service provider, including computing resources, storage resources, or network resources, etc. The computing resources are, for example, servers, etc. Taking the computing resources included in the cloud data center as servers running virtual machines as an example, the model generation system can be independently deployed on the servers or virtual machines in the cloud data center. The model generation system can also be distributedly deployed on multiple servers in the cloud data center, or distributedly deployed on multiple virtual machines in the cloud data center, or distributedly deployed on the servers and virtual machines in the cloud data center.

[0050] As Figure 1B shown, the model generation system can, for example, be abstracted by the cloud service provider into a model generation service on the cloud service platform and provided to users. After the user purchases the cloud service on the cloud service platform, the cloud environment uses the model generation system deployed in the cloud data center to provide the model generation service to the user. When the user uses the model generation service, the user can specify the task (i.e., the task objective) that needs to be completed by the model generation service through the application program interface (API) or the graphical user interface (Graphical User Interface, GUI), and upload the sample set to the cloud environment. The model generation system in the cloud environment receives the user's task objective and sample set, and performs the operation of automatically generating a model. The model generation system returns the automatically generated video review model to the user through the API or the GUI. The video review model can be downloaded or used online by the user.

[0051] The user classifies the samples in advance according to requirements, refining multiple different types of targets: ordinary flags, pedestrians, buildings, others (not any of the first four targets, but not a violation object either), a total of 5 targets. An image that does not contain any one or more of these 5 targets is a positive sample image. If a sample image contains any one of these 5 targets, then the sample image is a negative sample. After the user uploads the sample set, the specified task for the model generation service to complete is: train a video review model that includes a feature extraction network and a multi-target recognition network. The feature extraction network can output multi-level hierarchical features and generate an attention mechanism using different hierarchical features. Among them, the feature extraction network is used to extract the features of each of the aforementioned 5 targets, and the multi-target recognition network includes 5 branch networks, corresponding to different targets among the 5 targets respectively.

[0052] The video review method described in the embodiments of the present application includes two stages: a learning stage and a prediction stage. In the learning stage, use Figure 1B the shown architecture to train a video review model. After that, in the prediction stage, based on Figure 1A the shown implementation environment, use this video review model to conduct video review.

[0053] Next, based on Figure 1A and Figure 1B , the video review method described in the embodiments of the present application will be described in detail. Exemplarily, please refer to Figure 2 . Figure 2 Figure 18 is a flowchart of the video review method provided by the embodiments of the present application. The execution subject of this embodiment is an electronic device, and this electronic device is, for example, Figure 1A a server in

[0054] 201. Obtain multiple frames of images from the video to be reviewed.

[0055] In the embodiments of the present application, the video to be reviewed can be a video uploaded by the user through a terminal device and expected to be shown to the public, etc. The video to be reviewed can be a live video, an on-demand video, etc.

[0056] After the electronic device obtains the video to be reviewed, it performs frame splitting on the video to be reviewed to obtain multiple image frames. The electronic device can determine multiple frames of images from these image frames for video review. For example, when the video is relatively short, all the image frames obtained by frame splitting are used for video review; another example is that when the video is relatively long, one frame of image is extracted every 10 seconds to obtain multiple frames of images.

[0057] In addition, the electronic device can also obtain multiple frames of images according to the video type. For example, when the video to be audited is a live video, one frame of image can be selected every 10 seconds to obtain multiple frames of images; for another example, when the video to be audited is an on-demand video, the electronic device extracts 1 frame of image per second to obtain multiple frames of images. For example, if the length of an on-demand video is 3 minutes and there are 30 frames of images per second, then 180 frames of images are extracted.

[0058] 202. Input the multiple frames of images into the video audit model to obtain the recognition result of each frame of image in the multiple frames of images and the confidence level of the recognition result. The video audit model includes a feature extraction network and a multi-object recognition network. The multi-object recognition network includes multiple branch networks, and different branch networks in the multiple branch networks are used to recognize different objects. The recognition result of each frame of image in the multiple frames of images is determined according to the output result of each branch network in the multiple branch networks.

[0059] In the embodiments of the present application, for each frame of image, the frame of image may not contain a single target object, but may contain multiple targets, and even normal targets and illegal targets coexist, and both occupy the main position. For example, a frame of image may contain both sensitive parts with various different features and normal human features. At this time, if only a single category judgment is given, it is difficult to train a video audit model during the learning process, and it is difficult to accurately obtain the recognition result of the image to be recognized during the prediction process.

[0060] Therefore, in the embodiments of the present application, a video audit model is pre-deployed on the server. The video audit model includes two parts: a feature extraction network and a multi-object recognition network. The feature extraction network is used to extract the features of each object according to the pre-refined objects, and the multi-object recognition network is used to recognize each object. For example, if 11 types of objects are pre-refined, the multi-object recognition network includes 11 branch networks, each branch network corresponds to an object, and each branch network only focuses on the object of its own branch network. In this way, for the situation where multiple targets coexist in a frame of image, there is no need for a large number of training samples to reflect this phenomenon, reducing the cost of accumulating samples.

[0061] For each frame of image, the server uses the feature extraction network to extract features, and inputs the extracted features into the multi-object recognition network to obtain multiple output results and the confidence level of each output result. Then, according to the multiple output results and the confidence level of each output result, the recognition result of a frame of image and the confidence level corresponding to the recognition result are determined.

[0062] Among them, the confidence level of the recognition result is used to indicate the credibility of the recognition result. The higher the confidence level, the more accurate the recognition result.

[0063] 203. Determine whether the video to be audited is a violation video according to the recognition result of each frame of the multi-frame images and the confidence level of the recognition result.

[0064] After the server obtains the recognition result of each frame of image and the confidence level of the recognition result of each frame of image, it determines whether the video to be audited is a violation video according to these recognition results and confidence levels.

[0065] In one way, the server determines whether the number of images that meet the preset conditions in the multi-frame images exceeds the preset number. If the number of images that meet the conditions exceeds the preset number, it determines that the video to be audited is a violation video; if the number of images that meet the preset conditions is less than the preset number, it determines that the video to be audited is a legal video. The preset conditions include: the recognition result is the preset result and the confidence level exceeds the preset confidence level. Taking the preset confidence level as 0.8 and the preset number as 100 as an example, if 180 frames of images are extracted from a 3-minute on-demand video for video audit, when the recognition results of more than 100 frames of the 180 frames of images contain the violation object and the confidence levels of the recognition results of these 100 frames of images are all greater than 0.8, it indicates that the 3-minute on-demand video is a violation video. These 100 frames of images are continuous or discontinuous images among the 180 frames of images.

[0066] In another way, in addition to considering whether the recognition result meets the preset result and whether the quantity exceeds the preset number, continuity also needs to be considered. At this time, when there are more than the preset number of consecutive multi-frame images in the multi-frame images, the recognition results of the consecutive multi-frame images conform to the preset result, and the confidence levels of the recognition results of the consecutive multi-frame images are greater than the preset threshold, the server determines that the video to be audited is a violation video. Taking the preset result and the preset confidence level as 0.8 and the preset number as 1000 as an example, if 1800 frames of images are extracted from a 30-minute on-demand video for video audit, when the recognition results of more than 1000 frames of the 1800 frames of images contain the aforementioned violation object and the confidence levels of the recognition results of these 1000 frames of images are all greater than 0.8, if these 1000 frames of images are not continuous, it indicates that the 30-minute on-demand video is not a violation video. Among them, continuous can be continuous in the strict sense, that is, these 1000 frames of images are adjacent images in sequence among the 1800 frames of images, and there are no other image frames between two adjacent frames of images on the time axis. Or, among these 1000 frames of images, there are images that do not meet the preset conditions between two adjacent frames of images on the time axis, but the number of images that do not meet the preset conditions is relatively small.

[0067] Adopting this solution, considering continuity while considering the recognition result and quantity improves the accuracy of video audit.

[0068] The video review method provided by the embodiments of the present application is such that an electronic device obtains multiple frames of images from a video to be reviewed, sequentially inputs the multiple frames of images into a video review model to obtain the recognition result and confidence level of each frame of image, and determines whether the video to be reviewed is a violation video based on the recognition results and confidence levels of each frame of image. Among them, the recognition result of each frame of image is determined based on the output results of multiple branch networks included in the multi-object recognition network in the video review model. With this solution, the multi-object recognition network of the video review model includes multiple branch networks, different branch networks are used to recognize different objects, and each branch network only focuses on the object of its own branch. By refining the task objectives, the learning is more targeted, with high accuracy and fast speed.

[0069] The video review method described in the embodiments of the present application includes a learning stage and a prediction stage. Below, these two stages will be described in detail respectively.

[0070] The learning stage is the stage of collecting samples and training an initial model with a large number of samples to obtain a video review model.

[0071] Figure 3 It is a schematic diagram of the process of training a video review model in the video review method provided by the embodiments of the present application. This embodiment includes:

[0072] 301. Determine multiple categories, and the categories in the multiple categories correspond one-to-one with the branch networks in the multi-object recognition network.

[0073] In the embodiments of the present application, instead of simply dividing into positive samples and negative samples, the positive samples and negative samples are refined to obtain sample images of refined categories. In this way, when collecting samples, first determine multiple categories, and the categories in the multiple categories correspond one-to-one with the branch networks in the multi-object recognition network. Then, for each category, collect relevant images as sample images. For example, the categories of the collected image samples are as follows:

[0074] 1) Injury: including bleeding people or animals.

[0075] 2) Big fire: including forest fires, building fires, factory fires, vehicle explosions, etc., where there is fire or thick smoke in this scene.

[0076] 3) Flag: including various illegal flags.

[0077] 4) Controlled props: including knives, etc.

[0078] 5) Ordinary flag: such as national flags of various countries and flags of normal organizations.

[0079] 6) Ordinary people: non-special people, single and multiple people, without the situation of gathering.

[0080] 7) Scenic spots.

[0081] 8) Other normal targets, not belonging to any of the categories in 1)-7) above.

[0082] By refining the positive and negative samples in violation recognition, different business processing requirements can be met. For example, different requirements for the violation level and different restrictions on the violation objects. Additionally, through refinement, the deep learning can learn targeted and more guiding features, making the network better trained. For example, the violation flag is relative to the ordinary flag, and these targets may all appear in the image frames of the violation video. When refined separately, the deep learning network can strengthen their respective features.

[0083] 302. Obtain a sample set including image samples of each category in the multiple categories.

[0084] In this step, the electronic device collects image samples for the refined categories in a targeted manner to obtain a sample set.

[0085] Normally, multiple categories can be obtained after refining the positive and negative samples. However, some image samples may not belong to any category. Therefore, defining multiple categories also includes an other category, such as 11) above. Except for the other category, for any category (for example, any one of 1)-7) above), hereinafter referred to as the target category, the electronic device can obtain image samples according to the scope of the target category. For example, for 1), obtain images containing injured people or animals. For the other category, any image sample that does not belong to the categories 1)-7) can be used as the image sample of this category. Or, image samples can also be collected specifically. During the collection process, the electronic device determines the associated features according to the target category. Then, obtain image samples with the associated features and not belonging to the target category, so as to obtain image samples belonging to the other category.

[0086] Taking the target category as 1) as an example, if the associated feature is red, the electronic device obtains images with the red feature but not injured images as the images of the other category 8), such as images containing red flowers, red clothes, etc. Additionally, the electronic device can also classify the image samples of the other category and refine new categories, such as adding building category, vehicle category, etc.

[0087] Adopting this solution, the electronic device obtains image samples in a targeted manner, which can improve the accuracy and training speed of model training.

[0088] 303. Construct an initial model, where the initial model includes an initial feature extraction network and an initial multi-target recognition network.

[0089] 304. Use the image samples in the sample set to train the initial model to obtain the video review model.

[0090] Exemplarily, the electronic device continuously inputs the image samples in the sample set into the initial model. After each input of an image sample, the parameters of the initial feature extraction network and the parameters of the initial multi-object recognition network are optimized, and the initial model is iteratively trained until the parameters of the initial feature extraction network and the parameters of the initial multi-object recognition network reach the optimal values. The initial model with the optimal parameters is used as the final video review model. That is to say, the initial feature extraction network with the optimal parameters is the feature extraction network in the video review model, and the initial multi-object recognition network with the optimal parameters is the multi-object recognition network in the video review model.

[0091] In addition, the basis for determining whether the model training is completed can also be that the number of iterative training times reaches a preset number of iterative times, etc., which is not limited in the embodiments of the present application.

[0092] By adopting this solution, the purpose of training an accurate video review model is achieved.

[0093] The trained video review model includes a feature extraction network and a multi-object recognition network. The feature extraction network is used to output multi-level hierarchical features. Each branch network in the multiple branch networks includes an attention module and a fully connected classification layer, and the attention modules of at least two branch networks correspond to different hierarchical features in the multi-level hierarchical features.

[0094] Figure 4 It is a schematic structural diagram of video review in the video review method provided by the embodiments of the present application. Please refer to Figure 4 , this video review model includes a feature extraction network and a multi-object recognition network. Among them, the feature extraction network can be DenseNet121, ResNet50, EfficientNetB0, etc., which is not limited in the embodiments of the present application. The multi-object recognition network includes multiple branch networks. Each branch network (hereinafter referred to as the target branch network) includes a cascaded global pooling layer, a first fully connected layer, a inactivation layer, a second fully connected layer, a third fully connected layer, a fourth fully connected layer, and a classification output layer. An attention mechanism is established between the feature extraction network and the multi-object recognition network. Next, the feature extraction network, the multi-object recognition network, and the attention mechanism will be described separately.

[0095] Figure 5 It is a schematic structural diagram of the feature extraction network in the video review model provided by the embodiments of the present application. Figure 5Among them, the feature extraction network adopts the classic structure of DenseNet121, including a convolution layer, a pooling layer, dense blocks, and transitions (DenseBlock+Transition)(1), DenseBlock+Transition(2), DenseBlock+Transition(3), and DenseBlock(4). Among them, DenseBlock(4) outputs the features of the classification layer, and the features of the classification layer are, for example, a 7×7 vector. During the training process, a pre-trained model of imagenet can be used to initialize the initial feature extraction network.

[0096] The feature extraction network is used to output multi-level hierarchical features, and the multi-level hierarchical features include the features of the classification layer and the intermediate layer features. Figure 5 Among them, the output of DenseBlock(4) of the feature extraction network is the feature of the classification layer (also called the deep feature), and this feature of the classification layer is input into each branch network. The output of any one of DenseBlock+Transition(1), DenseBlock+Transition(2), and DenseBlock+Transition(3) of the feature extraction network is the intermediate layer feature. These intermediate layer features and the features of the classification layer are also called hierarchical features at different levels, shallow features, etc. The electronic device can establish an attention mechanism based on these hierarchical features.

[0097] Please also refer to Figure 4 and Figure 5 , Figure 4 The convolution (conv) used to generate shallow features in Figure 5 is the output of any one of DenseBlock+Transition(1), DenseBlock+Transition(2), and DenseBlock+Transition(3) in Figure 5 Taking DenseBlock+Transition(1) as an example, according to

[0098] Figure 6It is a schematic structural diagram of the fully-connected classification layer of the multi-object recognition network in the video review model provided by the embodiments of the present application. In the embodiments of the present application, each branch network of the multi-object recognition network includes an attention module and a fully-connected classification layer. The multi-object recognition network uses the attention module and the fully-connected classification layer to recognize the features extracted by the feature extraction network. Hereinafter, any one of the branch networks is referred to as the target branch network. The input of the fully-connected layer of the target branch network is the output of the attention module. The fully-connected classification layer includes: 1 global pooling layer, 4 fully-connected layers. The first 3 fully-connected layers include a relu activation layer, a dropout layer, and a classification output layer. The classification output layer includes a softmax layer and a classification probability layer, which are used to output 2 categories, that is, to make a binary classification judgment to determine whether it is the target of this branch network. In addition, the classification output layer also outputs the confidence of the output result of the target branch network. Each branch network makes a judgment separately and learns the parameters of the fully-connected layer separately.

[0099] Adopting this solution, the multi-object recognition network includes multiple branch networks, and different branch networks are used to recognize different targets. In the scenario where multiple targets coexist, the target recognition accuracy is high. Moreover, it helps to recognize small targets. A small target refers to: in an image, the main target occupies a relatively large area, and the targets other than the main target. For example, in an image, a car occupies 80% of the area, and in the remaining 20% of the image, there are illegal personnel, then the illegal personnel are small targets.

[0100] In the embodiments of the present application, the attention module of the target branch network can adopt an SE channel attention module, a spatial and channel attention module, a spatial attention module, or an SKnet multi-path attention module, etc. Hereinafter, the attention module of the target branch network will be described by taking the SE channel attention module as an example. Exemplarily, please refer to Figure 7 .

[0101] Figure 7 It is a schematic structural diagram of the attention module in the video review method provided by the embodiments of the present application. Please refer to Figure 7, the SE channel attention module includes a global pooling layer, two fully connected (FC) layers, an activation layer, etc. The activation function of the activation layer is, for example, the sigmoid function. For the target branch network, the electronic device determines the hierarchical features corresponding to the target branch network and inputs the hierarchical features into the attention module, so that the attention module outputs a feature vector, which is, for example, a 1×1×C vector, and this vector is also called a weight vector. At the same time, the classification layer features of the feature extraction network are input into the attention modules of each branch network. The classification layer features are, for example, an H×W×C vector. The electronic device multiplies the two vectors, etc., to obtain the input features of the full-face connection layer, and the input features can effectively guide the target branch network to identify the target corresponding to the target branch network. Here, H represents the number of pixels of the image sample or the image frame to be recognized in the length direction, W represents the number of pixels of the image sample or the image frame to be recognized in the width direction, and C represents the RGB channel data. Or, H represents the length of the feature output by the convolutional layer, W represents the width of the feature output by the convolutional layer, and C represents the number of channels.

[0102] Adopting this solution, by introducing the attention mechanism into the target recognition network, the learning efficiency and accuracy of the target object can be improved.

[0103] Optionally, in the above embodiments, different targets are recognized separately. Therefore, different attention mechanisms can be adopted based on the characteristics of each branch network. That is to say, among multiple branch networks, at least two branch networks exist, and the attention modules of the two branch networks correspond to different hierarchical features in the multi-level hierarchical features. For example, the hierarchical features corresponding to one branch network are the output of DenseBlock+Transition(1), and the hierarchical features corresponding to another branch network are the output of DenseBlock+Transition(2). Another example is that the hierarchical features corresponding to one branch network are the output of DenseBlock+Transition(3), and the hierarchical features corresponding to another branch network are the output of DenseBlock+Transition(4).

[0104] In this way, each branch network of the multi-target recognition network, in addition to using the classification layer features of the feature extraction network, can also integrate hierarchical features of different levels to improve the single-target recognition accuracy. Moreover, in the multi-target recognition network, by introducing the attention mechanism, each branch network screens the features entering itself, can better focus on the features of its own branch network, and at the same time improve the recognition success rate of small target objects.

[0105] Optionally, in the above embodiments, the hierarchical features corresponding to at least two branch networks are different. There may be at least one first branch network and at least one second branch network among the multiple branch networks. The hierarchical features corresponding to the attention module of the first branch network are at least one of the intermediate layer features of the feature extraction network, or the hierarchical features corresponding to the attention module of the first branch network are at least one of the classification layer features and the intermediate layer features; the hierarchical features corresponding to the attention module of the second branch network at least include the classification layer features of the feature extraction network.

[0106] Exemplarily, the attention module of the first branch network generates an attention mechanism using one or more different hierarchical features, and this hierarchical feature cannot be merely the classification layer feature. For example, the hierarchical features corresponding to the first branch network are at least one of DenseBlock+Transition(1), DenseBlock+Transition(2), DenseBlock+Transition(3), or the hierarchical features corresponding to the first branch network are DenseBlock(4) and at least one of DenseBlock+Transition(1), DenseBlock+Transition (2), DenseBlock+Transition(3). In this case, the attention module of the first branch network is called a multi-feature fusion attention module.

[0107] The attention module of the second branch network generates an attention mechanism using the classification layer features. That is, the hierarchical features corresponding to the second branch network are only the output of DenseBlock(4). In this case, the attention module of the second branch network is called a basic attention module.

[0108] Figure 8 It is a schematic diagram of the basic attention module in the video review method provided by the embodiments of the present application. Please refer to Figure 8 The attention module adopts an SE channel attention module, including a global pooling layer, 2 FCs, an activation layer, etc. The basis for the attention module to generate the weight vector is the classification layer features, that is, the features output by DenseBlock (4). The attention module generates a 1×1×C weight vector using the classification layer features, and uses this weight vector and the classification layer features (a H×W×C vector) to generate a new feature, and this new feature is the input feature of the fully connected classification layer.

[0109] Figure 9 It is a schematic diagram of the multi-feature fusion attention module in the video review method provided by the embodiments of the present application. Please refer to Figure 9, the attention module adopts the SE channel attention module, including a global pooling layer, two FCs, an activation layer, etc. The basis for the attention module to generate the weight vector is a fused feature, which is the fused feature of the intermediate feature and the classification layer feature. The intermediate feature can be any one of DenseBlock+Transition(1), DenseBlock+Transition(2), and DenseBlock+Transition(3), and the classification layer feature is, for example, the feature output by DenseBlock(4).

[0110] Figure 10 is another schematic diagram of the multi-feature fusion attention module in the video review method provided by the embodiments of the present application. Please refer to Figure 10 , the attention module adopts the SE channel attention module, including a global pooling layer, two FCs, an activation layer, etc. The basis for the attention module to generate the weight vector is an intermediate feature.

[0111] Adopting this solution, each branch network generates an attention mechanism according to different hierarchical features, which can enable each branch network to accurately identify its respective target.

[0112] In the above embodiments, there are multiple branch networks in the multi-target recognition network. The hierarchical features corresponding to the branch networks can be determined according to the targets of the branch networks. For example, the features of the positive samples of a branch network lie in texture, shape, etc., and the intermediate layer features in the feature extraction network are shape, color, texture, etc. Therefore, based on Figure 10 , an attention mechanism is generated using the intermediate layer features.

[0113] In the embodiments of the present application, different attention mechanisms are adopted for different targets, that is, different branch networks, which can effectively capture the features of relevant categories. Exemplarily, please refer to Figure 11A and Figure 11B , and Figure 12A and Figure 12B .

[0114] Figure 11A is a heat map in the video review method provided by the embodiments of the present application. Figure 11B is another heat map in the video review method provided by the embodiments of the present application. The electronic device can generate a heat map by calculating the partial derivatives of pixels and taking the global average of the partial derivatives, etc., based on the features extracted by the feature extraction network. Please refer to ​ , the heat map represents the area where one type of positive sample is located in the coexistence map, ​ represents the area where another type of positive sample is located in the coexistence map. Obviously, the video review model described in the embodiments of the present application can accurately learn the coexisting positive samples.

[0115] ​ It is another heat map in the video review method provided by the embodiments of the present application. ​ It is another heat map in the video review method provided by the embodiments of the present application. Please refer to ​ The heat map expresses the area where a kind of positive sample is located in the coexistence map, ​ which expresses the area where the large fire is located in the coexistence map. Obviously, the video review model described in the embodiments of the present application can accurately learn the coexisting positive samples.

[0116] Optionally, in the above embodiment, in order to accelerate the inference speed of the entire video review model, the electronic device uses a large number of training samples to train the initial model. After obtaining the video review model, the electronic device also constructs the video review model into a service or SDK.

[0117] Exemplarily, the electronic device uses Tensor RT, etc., to convert the video review model, accelerate the inference speed of the video review model, and package it into an SDK or a service for the convenience of users.

[0118] The above has described in detail the training process of the video review model and the structure of the trained video review model. Next, how to review the video to be reviewed using the video review model will be described in detail.

[0119] During the video review process, the electronic device obtains multiple frames of images from the video to be reviewed. For any one frame of the multiple frames of images, hereinafter referred to as the target image, the electronic device uses the feature extraction network to extract the features of the target image to obtain the multi-layer hierarchical features. The target image is any one frame of the multiple frames of images, and the multi-layer hierarchical features include classification layer features and intermediate layer features. Then, the multi-layer hierarchical features are input into each of the multiple branch networks to obtain the output result of each branch network and the confidence of the output result. Finally, according to the output result of each branch network in the multiple branch networks and the confidence of the output result, the recognition result of the target image and the confidence of the recognition result are determined.

[0120] Exemplarily, the classification layer features extracted by the feature extraction network are input into the attention modules of each branch network. At the same time, each branch network's attention module has corresponding hierarchical features. The attention module calculates the input features of the fully connected layer according to the classification layer features and the hierarchical features, and the fully connected layer obtains a binary classification structure according to the input features. For example, if the target object of a branch network is a large fire, the input of the fully connected layer of this branch network is whether the target image contains a large fire, the probability of containing a large fire, and the probability of not containing a large fire. Herein, containing a large fire means that the target image has elements such as a large fire.

[0121] By adopting this solution, the purpose of accurately determining the recognition result of each frame of image and the confidence level of the recognition result is achieved.

[0122] ​ It is another flowchart of the video review method provided by the embodiments of the present application. The execution subject of this embodiment is an electronic device, and a codec module, an identification module, and a judgment module are integrated on the electronic device. This embodiment includes:

[0123] 1301. Receive the video to be reviewed and perform frame extraction on the video to be reviewed.

[0124] Exemplarily, the codec module performs frame extraction on the received video. In one embodiment, a multimedia video processing tool, such as Fast Forward Mpeg, etc., is used to perform frame extraction on the video to be reviewed, so as to obtain multiple frames of images. Among them, the video to be reviewed is in RGB format, etc., and the embodiments of the present application do not limit this. The frame extraction frequency can be determined according to the length and type of the video. For example, when the video to be reviewed is a live video, the frame extraction frequency is 1 frame per 10 seconds; when the video to be reviewed is an on-demand video, the frame extraction frequency is 1 frame per 1 second. After obtaining multiple frames of images, the multiple frames of images are sequentially input into the recognition module.

[0125] 1302. Use the video review model to review and identify each frame of image to obtain the recognition result of each frame of image and the confidence level of the recognition result.

[0126] Any one frame of image in the multiple frames of images is hereinafter referred to as the target image. Since there are multiple branch networks in the video review model, each branch network will identify the target image, so as to obtain the output results of each branch network and the confidence levels of each output result. The electronic device determines the recognition result of the target image and the confidence level of the recognition result according to the output results and confidence levels of these branch networks.

[0127] Optionally, step 1302 includes the following steps:

[0128] 13021. For the target image in the multiple frames of images, use the feature extraction network to extract the features of the target image to obtain multi-level hierarchical features. The target image is any one frame of image in the multiple frames of images, and the multi-level hierarchical features include classification layer features and intermediate layer features.

[0129] 13022. For the target branch network in the multi-target recognition network, input the corresponding hierarchical features into the attention module of the target branch network to obtain the feature vector corresponding to the target branch network. The target branch network is any one branch network in the multiple branch networks.

[0130] Among them, the target branch network includes a global pooling layer, 2 fully connected (FC) layers, an activation layer, etc. The activation function of the activation layer is, for example, the sigmoid function. The output of the activation layer is the above-mentioned feature vector, which can be expressed as 1×1×C.

[0131] 13023. Determine the input features of the fully connected classification layer of the target branch network according to the feature vector and the classification layer features.

[0132] Taking the feature extraction network as the classic DenseNet121 as an example, the classification layer features are the output of DenseBlock (4), which can be expressed as H×W×C.

[0133] 13024. Input the input features into the fully connected classification layer of the target branch network to obtain the recognition result of the target branch network and the confidence of the recognition result.

[0134] 13025. Sort the recognition results of the multiple branch networks in descending order of confidence, and use the sum of the recognition results of the preset number of branch networks with the top ranking as the recognition result of the target image.

[0135] For example, if there are 11 branch networks in the multi-target branch network, then for the target image, the multi-target recognition result will obtain 11 output results. Assuming that the preset number is 3, and the output results of the top three branch networks are all illegal objects, and the confidence levels are 0.8, 0.75, and 0.7 in sequence, then the electronic device considers the recognition result of the target image to be 3 illegal objects, and the confidence of the recognition result can be the minimum value of the confidence levels of the three output results, or the average value, etc.

[0136] Adopting this solution, a certain number of output results are determined from the output results of multiple branch networks, and the sum of these output results is used as the recognition result of the target image, and this recognition result can more accurately describe the target image.

[0137] 1303. Determine whether the video to be audited is an illegal video according to the recognition result and the confidence of each frame of the multi-frame images.

[0138] Exemplarily, after obtaining the recognition results and confidence levels of each frame of image in multiple frames of images, the recognition results and confidence levels of each frame of image can be input into a judgment module for judgment by the judgment module. Assume that a confidence level greater than 0.9 is a strong confidence level, a confidence level between 0.7 and 0.9 is a medium confidence level, and a confidence level less than 0.7 is not credible. The judgment module determines whether each frame of image is a violation image according to the confidence levels of the recognition results of each image in the multiple frames of images. For example, if the recognition result of a frame of image conforms to a violation and the confidence level is greater than 0.7 and less than 0.9, then this frame of image is considered a violation image with a medium confidence level.

[0139] After that, the electronic device judges the concentration in time of the images with strong confidence levels and medium confidence levels in the multiple frames of images. For example, if the recognition results of 10 consecutive frames of images conform to a violation and the confidence level is greater than 0.7, then the video to be audited is considered a violation video.

[0140] In addition, the violation level of the video to be audited can also be determined according to the recognition results and confidence levels of each frame of image in the multiple frames of images, and videos with a medium violation confidence level can be manually verified according to business requirements. By adopting this solution, the workload of manual verification is greatly reduced, and at the same time, the accuracy of violation recognition is ensured.

[0141] The following is an embodiment of the apparatus of the present application, which can be used to execute the method embodiment of the present application. For details not disclosed in the embodiment of the apparatus of the present application, please refer to the method embodiment of the present application.

[0142] ​ It is a schematic diagram of a video audit apparatus provided in an embodiment of the present application. The video audit apparatus 1400 includes: an acquisition module 141, a processing module 142, and a determination module 143.

[0143] The acquisition module 141 is configured to acquire multiple frames of images from the video to be audited;

[0144] The processing module 142 is configured to input the multiple frames of images into a video audit model to obtain the recognition result of each frame of image in the multiple frames of images and the confidence level of the recognition result. The video audit model includes a feature extraction network and a multi-object recognition network. The multi-object recognition network includes multiple branch networks, and different branch networks in the multiple branch networks are used to recognize different targets. The recognition result of each frame of image in the multiple frames of images is determined according to the output result of each branch network in the multiple branch networks;

[0145] The determination module 143 is configured to determine whether the video to be audited is a violation video according to the recognition result of each frame of image in the multiple frames of images and the confidence level of the recognition result.

[0146] In a feasible implementation manner, the feature extraction network is used to output multi-level hierarchical features, and each branch network in the multiple branch networks includes an attention module and a fully connected classification layer.

[0147] In a feasible implementation manner, the attention modules of at least two branch networks in the multiple branch networks correspond to different hierarchical features in the multi-level hierarchical features.

[0148] In a feasible implementation manner, the processing module 142 is configured to, for a target image in the multiple frames of images, use the feature extraction network to extract the features of the target image to obtain the multi-level hierarchical features, where the target image is any one of the multiple frames of images, and the multi-level hierarchical features include classification layer features and intermediate layer features; input the multi-level hierarchical features into each branch network in the multiple branch networks to obtain the output result of each branch network and the confidence of the output result; and determine the recognition result of the target image and the confidence of the recognition result according to the output result of each branch network in the multiple branch networks and the confidence of the output result.

[0149] In a feasible implementation manner, when the processing module 142 inputs the multi-level hierarchical features into each branch network in the multiple branch networks to obtain the output result of each branch network and the confidence of the output result, for a target branch network in the multi-target recognition network, it is configured to input corresponding hierarchical features into the attention module of the target branch network to obtain a feature vector corresponding to the target branch network, where the target branch network is any one of the multiple branch networks; determine the input features of the fully connected classification layer of the target branch network according to the feature vector and the classification layer features; and input the input features into the fully connected classification layer of the target branch network to obtain the output result of the target branch network and the confidence of the output result.

[0150] In a feasible implementation manner, when the processing module 142 determines the recognition result of the target image and the confidence of the recognition result according to the output result of each branch network in the multiple branch networks and the confidence of the output result, it is configured to sort the output results of the multiple branch networks in descending order of the confidence of the output results, and use the sum of the output results of the top preset number of branch networks in the sorting as the recognition result of the target image.

[0151] In a feasible implementation manner, there is at least one first branch network among the multiple branch networks, and the hierarchical feature corresponding to the attention module of the first branch network is at least one of the intermediate layer features of the feature extraction network, or the hierarchical feature corresponding to the attention module of the first branch network is at least one of the classification layer feature and the intermediate layer features.

[0152] In a feasible implementation manner, there is at least one second branch network among the multiple branch networks, and the hierarchical feature corresponding to the attention module of the second branch network only includes the classification layer feature of the feature extraction network.

[0153] In a feasible implementation manner, the fully connected classification layer of the target branch network in the multi-target recognition network includes a cascaded global pooling layer, a first fully connected layer, a dropout layer, a second fully connected layer, a third fully connected layer, a fourth fully connected layer, and a classification output layer.

[0154] In a feasible implementation manner, the determining module 143 is configured to determine that the video to be reviewed is a violation video when there are more than a preset number of consecutive multi-frame images in the multi-frame images, the recognition results of the consecutive multi-frame images conform to a preset result, and the confidence levels of the recognition results of the consecutive multi-frame images are greater than a preset threshold.

[0155] ​ It is a schematic diagram of another video review device provided by an embodiment of the present application. The video review device 1500 includes: an acquisition module 151, a processing module 152, a determining module 153, and a training module 154. Among them, the descriptions of the acquisition module 151, the processing module 152, and the determining module 153 can be seen in ​ the description.

[0156] The training module 154 is configured to determine multiple categories before the processing module 152 inputs the multi-frame images into the video review model to obtain the recognition results of each frame image in the multi-frame images and the confidence levels of the recognition results, where the categories in the multiple categories correspond one by one to the branch networks in the multi-target recognition network; obtain a sample set including image samples of each category in the multiple categories; construct an initial model, where the initial model includes an initial feature extraction network and an initial multi-target recognition network; and train the initial model using the image samples in the sample set to obtain the video review model.

[0157] In a feasible design, when the training module 154 obtains a sample set including image samples of each of the multiple categories, for a target category among the multiple categories, the associated features are determined, where the target category is a category other than the other categories among the multiple categories; image samples having the associated features and not belonging to the target category are obtained to obtain image samples belonging to the other categories.

[0158] Please refer again to ​ , in a feasible implementation manner, the above video review device 1500 further includes:

[0159] A construction module 155, configured to construct the video review model into a service or a software development kit (SDK).

[0160] The video review device provided by the embodiments of the present application can perform the actions of the electronic device in the above embodiments, and its implementation principle and technical effects are similar, and will not be elaborated here.

[0161] ​ This is a schematic structural diagram of an electronic device provided by an embodiment of the present application. As ​ shown, the electronic device 1600 includes:

[0162] A processor 161 and a memory 162;

[0163] The memory 162 stores computer instructions;

[0164] The processor 161 executes the computer instructions stored in the memory 162, so that the processor 161 executes the video review method implemented by the above electronic device.

[0165] The specific implementation process of the processor 161 can be referred to the above method embodiments, and its implementation principle and technical effects are similar, and will not be elaborated here in this embodiment.

[0166] Optionally, the electronic device 1600 further includes a communication component 163. Among them, the processor 161, the memory 162, and the communication component 163 can be connected through a bus 164.

[0167] The embodiments of the present application further provide a computer-readable storage medium, in which computer instructions are stored, and when the computer instructions are executed by a processor, they are used to implement the video review method implemented by the above electronic device.

[0168] The embodiments of the present application further provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the video review method implemented by the above electronic device.

[0169] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include the common general knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and examples are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the following claims.

[0170] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A video review method, characterized in that, Including: Obtaining multiple frames of images from the video to be audited; Inputting the multiple frames of images into a video audit model to obtain the recognition result of each frame of image in the multiple frames of images and the confidence of the recognition result. The video audit model includes a feature extraction network and a multi-object recognition network. The feature extraction network is used to extract multiple features from each frame of image according to multiple preset targets. The features in the multiple features correspond to the targets in the multiple targets. The multi-object recognition network includes multiple branch networks. Different branch networks in the multiple branch networks are respectively used to recognize different targets in the multiple preset targets. The recognition result of each frame of image in the multiple frames of images is determined according to the output result of each branch network in the multiple branch networks; Determining whether the video to be audited is a violation video according to the recognition result of each frame of image in the multiple frames of images and the confidence of the recognition result.

2. The method according to claim 1, wherein The feature extraction network is used to output multi-level hierarchical features. Each branch network in the multiple branch networks includes an attention module and a fully connected classification layer.

3. The method according to claim 2, wherein Attention modules of at least two branch networks in the multiple branch networks correspond to different hierarchical features in the multi-level hierarchical features.

4. The method according to claim 2, wherein The inputting the multiple frames of images into a video audit model to obtain the recognition result of each frame of image in the multiple frames of images and the confidence of the recognition result includes: For a target image in the multiple frames of images, using the feature extraction network to extract the features of the target image to obtain the multi-level hierarchical features. The target image is any one frame of image in the multiple frames of images. The multi-level hierarchical features include classification layer features and intermediate layer features; Inputting the multi-level hierarchical features into each branch network in the multiple branch networks to obtain the output result of each branch network and the confidence of the output result; Determining the recognition result of the target image and the confidence of the recognition result according to the output result of each branch network in the multiple branch networks and the confidence of the output result.

5. The method according to claim 4, characterized in that, The inputting the multi-level hierarchical features into each branch network in the multiple branch networks to obtain the output result of each branch network and the confidence of the output result includes: For a target branch network in the multi-object recognition network, inputting the corresponding hierarchical features into the attention module of the target branch network to obtain the feature vector corresponding to the target branch network. The target branch network is any one branch network in the multiple branch networks; Determining the input features of the fully connected classification layer of the target branch network according to the feature vector and the classification layer features; Inputting the input features into the fully connected classification layer of the target branch network to obtain the output result of the target branch network and the confidence of the output result.

6. The method according to claim 4, wherein The determining the recognition result of the target image and the confidence of the recognition result according to the output result of each branch network in the multiple branch networks and the confidence of the output result includes: Sort the output results of the multiple branch networks in descending order according to the confidence of the output results, and use the sum of the output results of a preset number of branch networks with higher rankings as the recognition result of the target image.

7. The method according to claim 5, wherein there is at least one first branch network among the multiple branch networks, and the hierarchical feature corresponding to the attention module of the first branch network is at least one of the intermediate layer features of the feature extraction network, or the hierarchical feature corresponding to the attention module of the first branch network is at least one of the classification layer feature and the intermediate layer features.

8. The method according to claim 5, wherein there is at least one second branch network among the multiple branch networks, and the hierarchical feature corresponding to the attention module of the second branch network only includes the classification layer feature of the feature extraction network.

9. The method according to any one of claims 2-8, characterized in that, The fully connected classification layer of the target branch network in the multi-target recognition network includes a cascaded global pooling layer, a first fully connected layer, a dropout layer, a second fully connected layer, a third fully connected layer, a fourth fully connected layer, and a classification output layer.

10. The method according to any one of claims 1-8, characterized in that, Determining whether the video to be audited is a violation video according to the recognition result and the confidence of the recognition result of each frame of the multi-frame images includes: When there are more than a preset number of consecutive multi-frame images among the multi-frame images, the recognition results of the consecutive multi-frame images conform to a preset result, and the confidence of the recognition results of the consecutive multi-frame images is greater than a preset threshold, determine that the video to be audited is a violation video.

11. The method according to any one of claims 1-8, characterized in that, Before inputting the multi-frame images into the video audit model to obtain the recognition result and the confidence of the recognition result of each frame of the multi-frame images, it further includes: Determine a plurality of categories, and there is a one-to-one correspondence between the categories in the plurality of categories and the branch networks in the multi-target recognition network; Obtain a sample set including image samples of each category in the plurality of categories; Construct an initial model, and the initial model includes an initial feature extraction network and an initial multi-target recognition network; Train the initial model with the image samples in the sample set to obtain the video audit model.

12. The method according to claim 11, wherein The obtaining a sample set including image samples of each category in the plurality of categories includes: For a target category among the plurality of categories, determine an associated feature, and the target category is a category other than the other categories among the plurality of categories; Obtain image samples having the associated feature and not belonging to the target category to obtain image samples belonging to the other categories.

13. The method according to claim 11, wherein After training the initial model with the image samples in the sample set to obtain the video audit model, it further includes: Construct the video audit model into a service or a software development kit (SDK).

14. A video review device, characterized in that, Including: An obtaining module, configured to obtain multi-frame images from a video to be audited; A processing module, configured to input the multi-frame images into a video review model to obtain an identification result of each frame image in the multi-frame images and a confidence level of the identification result. The video review model includes a feature extraction network and a multi-object recognition network. The feature extraction network is configured to extract multiple features from each frame image according to multiple preset targets, and the features in the multiple features correspond to the targets in the multiple targets. The multi-object recognition network includes multiple branch networks, and different branch networks in the multiple branch networks are respectively configured to recognize different targets in the multiple preset targets. The identification result of each frame image in the multi-frame images is determined according to the output results of each branch network in the multiple branch networks; A determination module, configured to determine whether the video to be reviewed is a violation video according to the identification result of each frame image in the multi-frame images and the confidence level of the identification result.

15. An electronic device, comprising a processor, a memory, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, the electronic device is caused to implement the method according to any one of claims 1 to 13.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the method according to any one of claims 1 to 13 is implemented.

Images