Video recognition method, device, equipment and storage medium

Abstract

The present application discloses a video recognition method, device, equipment and storage medium, and relates to the field of computer technology. The method comprises: obtaining a video clip; extracting the features of the video clip through a video encoder to obtain the video features of the video clip; obtaining the similarity corresponding to at least one illegal text feature in the illegal text feature library according to the video features and the illegal text feature library, and the similarity corresponding to the illegal text feature refers to the similarity between the illegal text feature and the video feature; obtaining the hit information of at least one illegal text feature according to the similarity corresponding to at least one illegal text feature and the judgment threshold of the illegal text feature library; in the case where the video clip contains illegal content, generating prompt information of the video clip based on the hit information. The present application only needs to extract the video features once through the video encoder, which greatly reduces the computational complexity of feature extraction and reduces the R&D cost of the video recognition algorithm.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video recognition method, device, equipment and storage medium. Background Art

[0002] Videos uploaded by users to the application may contain illegal content, so the uploaded videos need to be identified and reviewed to ensure a healthy and positive product environment.

[0003] The related art provides a multimodal video understanding method based on a large language model, which obtains a video to be understood containing multimodal data, extracts features of the multimodal data of the video to be understood, obtains a first feature of the video frame of the video to be understood, a second feature of the text of the video to be understood, and a third feature of the audio of the video to be understood, and the first feature, the second feature, and the third feature have corresponding timestamps. The timestamps of the first feature, the second feature, and the third feature are aligned, and the first feature, the second feature, and the third feature are integrated into preset format data. According to the preset format data, preset user questions, and preset prompt words, the understanding result of the video to be understood is generated and output, which can integrate the multimodal data of the video, understand the video content more comprehensively and accurately, overcome the limitations of single modal information processing, align the data based on the timestamp and integrate the data to ensure the temporal consistency of the multimodal data and the structured output of the understanding result.

[0004] However, the above method requires multiple feature extractions on the video to be understood, which requires a lot of calculations and results in high video recognition costs. Summary of the invention

[0005] The embodiments of the present application provide a video recognition method, apparatus, device and storage medium. The technical solutions provided by the embodiments of the present application are as follows:

[0006] According to one aspect of an embodiment of the present application, a video recognition method is provided, the method comprising:

[0007] Get video clips;

[0008] Extracting features of the video clip by a video encoder to obtain video features of the video clip;

[0009] According to the video feature and the illegal text feature library, a similarity corresponding to at least one illegal text feature in the illegal text feature library is obtained, wherein the similarity corresponding to the illegal text feature refers to the similarity between the illegal text feature and the video feature, and the illegal text feature library is a feature library obtained by extracting features from at least one illegal text in the illegal text library;

[0010] Obtaining hit information of the at least one illegal text feature according to the similarities respectively corresponding to the at least one illegal text feature and the determination threshold of the illegal text feature library, wherein the hit information is used to indicate illegal content contained in the video clip;

[0011] In the case that the video clip contains illegal contents, prompt information of the video clip is generated based on the hit information, and the prompt information of the video clip is used to prompt the illegal contents contained in the video clip.

[0012] According to one aspect of an embodiment of the present application, a video recognition device is provided, the device comprising:

[0013] A video acquisition module, used to acquire video clips;

[0014] A feature extraction module, used to extract features of the video clip through a video encoder to obtain video features of the video clip;

[0015] A similarity determination module, for obtaining, based on the video feature and the illegal text feature library, the similarity corresponding to at least one illegal text feature in the illegal text feature library, wherein the similarity corresponding to the illegal text feature refers to the similarity between the illegal text feature and the video feature, and the illegal text feature library is a feature library obtained by extracting features from at least one illegal text in the illegal text library;

[0016] A hit determination module, used to obtain hit information of the at least one illegal text feature according to the similarities respectively corresponding to the at least one illegal text feature and the determination threshold of the illegal text feature library, wherein the hit information is used to indicate illegal content contained in the video clip;

[0017] The prompt generating module is used to generate prompt information of the video segment based on the hit information when the video segment contains illegal content, and the prompt information of the video segment is used to prompt the illegal content contained in the video segment.

[0018] According to one aspect of an embodiment of the present application, a computer device is provided, the computer device comprising a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-mentioned video recognition method.

[0019] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program is stored. The computer program is loaded and executed by a processor to implement the above-mentioned video recognition method.

[0020] According to one aspect of an embodiment of the present application, a computer program product is provided. The computer program product includes a computer program. The computer program is loaded and executed by a processor to implement the above-mentioned video recognition method.

[0021] The technical solution provided in the embodiments of the present application can bring the following beneficial effects:

[0022] By using a video encoder to extract video features, and judging whether the video features contain the illegal content represented by the illegal text based on the similarity between the video features and at least one illegal text feature, compared with the related art that requires the extraction of multiple features of video features, this application only needs to extract video features once through a video encoder, which greatly reduces the amount of computation for feature extraction, and after obtaining the video features, video recognition is achieved through similarity calculation and judgment of similarity and judgment threshold, avoiding the development of multiple sub-models to handle various illegal content, reducing the development cost of video recognition algorithms, and supporting large-scale and low-cost video recognition. And by annotating the illegal content of the video segment through the illegal text library, it is possible to quickly respond to and identify new video segments by updating the illegal text library, improving the efficiency of video recognition and further reducing the cost of video recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of an implementation environment of a solution provided by an embodiment of the present application;

[0024] Figure 2 is a flow chart of a video recognition method provided by an embodiment of the present application;

[0025] Figure 3 It is a schematic diagram of a calculation process of similarity corresponding to illegal text features provided by an embodiment of the present application;

[0026] Figure 4 This is a schematic diagram of a process for determining a violation of a video clip provided by an embodiment of the present application;

[0027] Figure 5 is a schematic diagram of a process for generating a video text pair provided by an embodiment of the present application;

[0028] Figure 6 is a schematic diagram of a training process of a video encoder provided by an embodiment of the present application;

[0029] Figure 7 is a schematic diagram of a process for determining a judgment threshold provided by an embodiment of the present application;

[0030] Figure 8 is a schematic diagram of the structure of a multimodal video understanding model provided by an embodiment of the present application;

[0031] Fig. 9 It is a schematic diagram of the complete process of the video recognition method provided by an embodiment of the present application;

[0032] Fig.10 is a block diagram of a video recognition device provided by an embodiment of the present application;

[0033] Fig.11 It is a structural block diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0035] Please refer to Figure 1 , which shows a schematic diagram of a solution implementation environment provided by an embodiment of the present application. The solution implementation environment can be implemented as a video recognition system. The solution implementation environment may include: a terminal device 10 and a server 20.

[0036] The number of terminal devices 10 can be one or more. The terminal device 10 can be an electronic device such as a mobile phone, a tablet computer, a laptop computer, a desktop computer, a game console, an e-book reader, a multimedia player, a wearable device, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc.

[0037] The terminal device 10 may be installed with a client of a target application, and the target application has a video playback function. The user may upload a video clip in the target application, and after the video clip passes the video review, the uploaded video clip is displayed in the target application. The present application does not limit the type of the target application, including but not limited to video applications, live broadcast applications, game applications, web applications, etc. Optionally, the target application may be an application that needs to be downloaded and installed, or may be a click-to-use application, and the present application does not limit this.

[0038] The server 20 is used to provide background services for the client of the target application installed and running in the terminal device 10. For example, the server 20 can be the background server of the above-mentioned target application. The server 20 can be an independent physical server, or a server cluster composed of multiple servers, or a cloud computing service center. Optionally, the server 20 provides background services for the target applications in multiple terminal devices 10 at the same time. The terminal device 10 and the server 20 can communicate with each other through the network.

[0039] In an embodiment of the present application, a user uploads a video clip through the client of a target application. After receiving the video clip, the terminal device sends the video clip to a server, and the server performs video recognition and review on the video clip. A video encoder is configured in the server, and the video encoder extracts the features of the video clip to obtain the video features of the video clip. According to the video features and the illegal text feature library corresponding to the illegal text library, the similarity between at least one illegal text feature in the illegal text feature library and the video features is calculated to obtain the similarity corresponding to at least one illegal text feature. Then, according to the similarity corresponding to at least one illegal text feature and the judgment threshold of the illegal text feature library, the hit information of at least one illegal text feature is obtained, and the hit information is used to indicate the illegal content contained in the video clip. In the case where the video clip contains illegal content, the prompt information of the video clip is generated based on the hit information, the prompt information is returned to the client of the target application, and the prompt information is displayed on the display interface of the target application to prompt the user of the illegal content contained in the video clip and prompt the user that the video clip fails to upload. In the case where the video clip does not contain illegal content, it means that the video clip passes the video review, and the video clip is uploaded to the client of the target application.

[0040] Please refer to Figure 2 , which shows a flow chart of a video recognition method provided by an embodiment of the present application. The execution subject of each step of the method may be a server. The method may include at least one of the following steps 210 to 250:

[0041] Step 210, obtaining a video clip.

[0042] The video clip is a video clip uploaded by the user on the client of the target application. Optionally, the video clip can be a video clip generated in real time in a live broadcast scene, or a complete video clip that has been pre-recorded. This application does not limit this. The video clip contains at least two video frames.

[0043] Step 220: extracting features of the video clip by using a video encoder to obtain video features of the video clip.

[0044] The video encoder is used to extract features from the video clips to obtain video features of the video clips. For the training process of the video encoder, reference may be made to the description of the following embodiment, which will not be introduced here.

[0045] Step 230, based on the video features and the illegal text feature library, obtain the similarity corresponding to at least one illegal text feature in the illegal text feature library, the similarity corresponding to the illegal text feature refers to the similarity between the illegal text feature and the video feature, and the illegal text feature library is a feature library obtained by extracting features from at least one illegal text in the illegal text library.

[0046] The illegal text library contains at least one illegal text, and the illegal text is a text used to represent illegal content. There is at least one illegal text in the illegal text library used to represent the same illegal content, and the number of illegal texts used to represent different illegal contents can be the same or different.

[0047] The illegal text feature library is a feature library obtained by extracting features of at least one illegal text in the illegal text library. The features of at least one illegal text in the illegal text library are extracted by a text encoder to obtain illegal text features corresponding to at least one illegal text, and the illegal text feature library is obtained based on the illegal text features corresponding to at least one illegal text. For the training process of the text encoder, reference may be made to the description of the following embodiment, which will not be introduced here.

[0048] It should be noted that the illegal text feature library is obtained by extracting features from at least one illegal text in the illegal text library in the offline stage, that is, the illegal text feature library is not generated after the server obtains the video clip, but is generated offline in advance by the text encoder at the same time as the illegal text library is generated. If the illegal text library changes, such as adding new illegal text to the illegal text library, or deleting the stored illegal text, the illegal text feature library will be updated according to the changed illegal text library, and the video clip will be recognized based on the updated illegal text feature library later.

[0049] Based on the video feature and at least one illegal text feature in the illegal text feature library, the similarity between the at least one illegal text feature and the video feature is calculated to obtain the similarity corresponding to the at least one illegal text feature.

[0050] The calculation process of the similarity corresponding to the illegal text features can be referred to Figure 3 As shown, in the offline stage, at least one illegal text in the illegal text library is input into the text encoder respectively, and the features of at least one illegal text are extracted by the text encoder respectively to obtain at least one illegal text feature, that is, to obtain the illegal text feature library. After receiving the video clip, the server extracts the features of the video clip by the video encoder to obtain the video features of the video clip, and respectively calculates the similarity between at least one illegal text feature and the video feature in the illegal text feature library to obtain the similarity corresponding to at least one illegal text feature. For example, the illegal text feature library contains n illegal text features, the similarity between the first illegal text feature and the video feature is S1, the similarity between the second illegal text feature and the video feature is S2, and the similarity between the nth illegal text feature and the video feature is Sn, where n is a positive integer.

[0051] Step 240, obtaining hit information of at least one illegal text feature according to the similarity corresponding to the at least one illegal text feature and the determination threshold of the illegal text feature library, wherein the hit information is used to indicate illegal content contained in the video clip.

[0052] The hit information of the illegal text feature is that the video clip hits the illegal text feature, or the video clip does not hit the illegal text feature. The video clip hits the illegal text feature, which means that the video clip hits the illegal text corresponding to the illegal text feature, and the illegal text corresponding to the illegal text feature hit by the video clip indicates that the video clip contains the illegal content represented by the illegal text. The video clip does not hit the illegal text feature, which means that the video clip does not hit the illegal text corresponding to the illegal text feature, and the video clip does not hit the illegal text corresponding to the illegal text feature, which means that the video clip does not contain the illegal content represented by the illegal text.

[0053] For example, if the hit information of the first illegal text feature is that the video clip hits the first illegal text feature, it means that the video clip contains the illegal content represented by the first illegal text. If the hit information of the first illegal text feature is that the video clip does not hit the first illegal text feature, it means that the video clip does not contain the illegal content represented by the first illegal text.

[0054] The determination threshold is used to determine the hit information of the illegal text feature, and the similarity corresponding to at least one illegal text feature is compared with the determination threshold of the illegal text feature library. If the similarity corresponding to the illegal text feature is greater than or equal to the determination threshold of the illegal text feature library, it is determined that the video clip hits the illegal text corresponding to the illegal text feature. If the similarity corresponding to the illegal text feature is less than the determination threshold of the illegal text feature library, it is determined that the video clip does not hit the illegal text corresponding to the illegal text feature.

[0055] In some embodiments, each illegal text feature in the illegal text feature library has a corresponding determination threshold. Then, according to the similarity corresponding to at least one illegal text feature and the determination threshold corresponding to at least one illegal text feature, the hit information of at least one illegal text feature is obtained.

[0056] Since each illegal text in the illegal text library represents different illegal contents, and the illegal degree of the illegal contents represented by each illegal text is also different, each illegal text in the illegal text library has a corresponding determination threshold, that is, each illegal text feature in the illegal text feature library has a corresponding determination threshold, and the determination threshold of the illegal text feature library refers to the determination threshold corresponding to at least one illegal text feature in the illegal text feature library. Therefore, when determining the hit information of each illegal text feature, it is necessary to compare the similarity corresponding to the illegal text feature and the determination threshold corresponding to the illegal text feature to obtain the hit information of the illegal text feature.

[0057] For the process of determining the judgment thresholds corresponding to at least one illegal text feature, reference may be made to the description of the following embodiment, which will not be introduced here.

[0058] By setting a corresponding judgment threshold for each illegal text feature, more accurate hit information can be obtained based on the textual meaning of different illegal texts, so that the hit information can more accurately indicate whether the video clip contains the illegal content represented by the illegal text, thereby improving the accuracy of video recognition.

[0059] In some embodiments, if the similarity corresponding to the illegal text feature is greater than or equal to the determination threshold corresponding to the illegal text feature, it is determined that the video segment hits the illegal text corresponding to the illegal text feature. If the similarity corresponding to the illegal text feature is less than the determination threshold corresponding to the illegal text feature, it is determined that the video segment does not hit the illegal text corresponding to the illegal text feature.

[0060] If the video clip hits the illegal text corresponding to the illegal text feature, it means that the video clip contains the illegal content represented by the illegal text. If the video clip does not hit the illegal text corresponding to the illegal text feature, it means that the video clip does not contain the illegal content represented by the illegal text.

[0061] The process of judging the violation of the video clip can be referred to Figure 4 As shown, in Figure 3 After the similarity (S1, S2, ..., Sn) between the video feature in the illegal text feature library and at least one illegal text feature in the illegal text feature library is determined, the hit information of the first illegal text feature is obtained according to the similarity S1 corresponding to the first illegal text feature and the determination threshold 1 corresponding to the first illegal text feature, and the hit information of other illegal text features is obtained in the same way. If S1 is greater than or equal to the threshold 1, it is determined that the video clip hits the first illegal text. If S1 is less than the threshold 1, it is determined that the video clip does not hit the first illegal text. Similarly, the recognition results of the video clip for other illegal texts are obtained.

[0062] By comparing the similarity corresponding to the illegal text features and the judgment threshold corresponding to the illegal text features, the recognition result of the video clip for the illegal text can be obtained, which reduces the recognition difficulty of the video clip and improves the video recognition efficiency.

[0063] In some embodiments, each illegal text feature in the illegal text feature library has the same determination threshold, that is, the determination threshold of the illegal text feature library is a determination threshold, then if the similarity corresponding to the illegal text feature is greater than or equal to the determination threshold of the illegal text feature library, it is determined that the video clip hits the illegal text corresponding to the illegal text feature. If the similarity corresponding to the illegal text feature is less than the determination threshold of the illegal text feature library, it is determined that the video clip does not hit the illegal text corresponding to the illegal text feature.

[0064] Step 250: When the video clip contains illegal content, generate prompt information of the video clip based on the hit information, and the prompt information of the video clip is used to prompt the illegal content contained in the video clip.

[0065] The video clip contains illegal content means that the video clip hits the illegal text corresponding to at least one illegal text feature, that is, the video clip contains illegal content represented by at least one illegal text in the illegal text library. Optionally, the video clip may only hit one illegal text in the illegal text library, that is, the video clip may only contain illegal content represented by one illegal text, and the hit information of at least one illegal text feature only has the hit information of one illegal text feature, which is the illegal text corresponding to the illegal text feature hit by the video clip, and the other hit information in the hit information of at least one illegal text feature is the illegal text corresponding to the illegal text feature that the video clip did not hit. Optionally, the video clip may also hit multiple illegal texts in the illegal text library, that is, the video clip may contain illegal content represented by multiple illegal texts, and the hit information of at least one illegal text feature has the hit information of multiple illegal text features, which is the illegal text corresponding to the illegal text feature hit by the video clip, and the other hit information in the hit information of at least one illegal text feature is the illegal text corresponding to the illegal text feature that the video clip did not hit.

[0066] Based on the hit information of at least one illegal text feature, which is used to indicate that the video clip hits the illegal text, prompt information of the video clip is generated. The prompt information of the video clip is used to prompt the illegal text hit by the video clip. It can also be understood that the prompt information of the video clip is used to prompt the illegal content contained in the video clip.

[0067] Exemplarily, if among the hit information of at least one illegal text feature, there is hit information of the first illegal text feature indicating that the video clip hits the first illegal text, and hit information of the second illegal text feature indicating that the video clip hits the second illegal text, then the prompt information of the generated video clip is used to prompt that the video clip contains the illegal content represented by the first illegal text and the illegal content represented by the second illegal text.

[0068] If the video clip contains illegal content, the review result of the video clip is not passed. If the video clip does not contain illegal content, the review result of the video clip is passed.

[0069] After generating the prompt information of the video clip, the server returns the prompt information of the video clip to the terminal device and displays the prompt information on the display interface of the target application to remind the user of the illegal content contained in the video clip and remind the user that the video clip has not passed the review and the video clip upload has failed. If the video clip does not contain illegal content, the video clip has passed the review, and the display interface of the target application displays a prompt information of successful upload after the video clip is successfully uploaded.

[0070] The technical solution provided by the embodiment of the present application uses a video encoder to extract video features, and judges whether the video features contain the illegal content represented by the illegal text according to the similarity between the video features and at least one illegal text feature. Compared with the related art that requires the extraction of multiple features of video features, the present application only needs to extract video features once through the video encoder, which greatly reduces the computational complexity of feature extraction. After obtaining the video features, video recognition is achieved through similarity calculation and judgment of similarity and judgment threshold, avoiding the development of multiple sub-models to handle various illegal contents, reducing the development cost of video recognition algorithms, and supporting large-scale and low-cost video recognition. And by annotating the illegal content of the video segment through the illegal text library, it is possible to quickly respond to and identify new video segments by updating the illegal text library, thereby improving the efficiency of video recognition and further reducing the cost of video recognition.

[0071] In some embodiments, the training process of the video encoder includes at least one of the following steps A1 to A3.

[0072] Step A1, obtaining at least one video-text pair, each video-text pair including a sample video and a video description text corresponding to the sample video.

[0073] The sample video may be any video, and the sample video includes at least two video frames. The present application does not limit the video content of the sample video. The video description text corresponding to the sample video is a text used to describe the video content of the sample video.

[0074] In some embodiments, the process of acquiring the video-text pair includes at least one of the following sub-steps A11 to A14.

[0075] Sub-step A11, obtaining a video data set, where the video data set includes at least one sample video.

[0076] The video data set is a data set obtained based on a sample video in at least one video-text pair, which means that at least one sample video in the video data set is the same as a sample video in at least one video-text pair.

[0077] Sub-step A12, obtaining at least one video description corresponding to the sample video according to the sample video through at least one multimodal video understanding model.

[0078] The at least one multimodal video understanding model here can be any publicly available large video language model, such as a natural language model based on a transformer structure trained with a large amount of data. The large amount of data can reach a sample level of more than 100 million, and this application does not limit this.

[0079] At least one multimodal video understanding model is a different publicly available multimodal video understanding model, and the multimodal video understanding model is used to extract the video content of the input video in text form. For each sample video, the sample video is input into at least one multimodal video understanding model to obtain at least one video description (videocaption) corresponding to the sample video, and each multimodal video understanding model outputs a corresponding video description. The video description is a descriptive text for the video content of the sample video, and the video descriptions output by different multimodal video understanding models may be different.

[0080] Sub-step A13, aggregating at least one video description through a large language model to obtain a video description text corresponding to the sample video.

[0081] The large language model here can be any publicly available large language model, such as a natural language model based on a transformer structure trained with a large amount of data. The large amount of data can reach a sample level of more than 100 million, and this application does not limit this.

[0082] At least one video description is input into a large language model, and semantic aggregation is performed on the at least one video description through the large language model to obtain an aggregated video description, that is, a video description text of a sample video.

[0083] Sub-step A14, obtaining a video-text pair according to the sample video and the video description text corresponding to the sample video.

[0084] The generation process of video text pairs can be referred to Figure 5As shown in the figure, for each sample video in the video dataset, the sample video is input into m multimodal video understanding models respectively, and the m multimodal video understanding models output m video descriptions respectively, and the m video descriptions are input into the large language model, and the large language model performs semantic aggregation on the m video descriptions and outputs the video description text corresponding to the sample video. According to each sample video in the video dataset and the video description text corresponding to each sample video, a video text pair is obtained.

[0085] It should be noted that, since the training process of the video encoder is performed in advance, the video-text pairs are also pre-generated offline based on at least one sample video, and do not occupy additional computing resources required for video recognition.

[0086] Since there is less data on video-text pairs, it is necessary to generate a training data set for the video encoder using the above method. By using at least one multimodal video understanding model to understand the sample video and summarizing at least one video description through a large language model, the obtained video description text can more comprehensively and accurately describe the video content of the sample video, improve the adaptability and accuracy of the video-text pair, and thus train the video encoder based on the video-text pair, which can improve the accuracy of the video encoder's feature extraction and thus improve the accuracy of video recognition.

[0087] Step A2, extracting features of the sample video by a pre-trained video encoder to obtain video features of the sample video, and extracting features of the video description text by a pre-trained text encoder to obtain text features of the video description text.

[0088] Step A3, with the goal of reducing the difference between the video features of the sample video and the text features of the video description text, adjust the parameters of the pre-trained video encoder and the pre-trained text encoder to obtain the trained video encoder and the trained text encoder, and the text encoder is used to extract the features of the illegal text.

[0089] According to the video features of the sample video and the text features of the video description text, a first loss function value is calculated, and according to the first loss function value, the parameters of the pre-trained video encoder and the pre-trained text encoder are adjusted to obtain a trained video encoder and a trained text encoder. The trained video encoder is used to extract features of the video clip in step 220, and the trained text encoder is used to extract features of at least one illegal text in the illegal text library in the offline stage.

[0090] Optionally, the first loss function value is a loss function value calculated based on a contrast loss function, and the video features of the sample video and the text features of the video description text are cross-modally aligned through the first loss function value to obtain a trained video encoder and a trained text encoder.

[0091] By using video-text pairs to train the video encoder and the text encoder, the features extracted by the trained video encoder and the trained text encoder are aligned as much as possible, thereby ensuring the effectiveness and accuracy of video recognition.

[0092] In some embodiments, the video encoder includes an image encoder and a criss-cross attention module, and the pre-trained video encoder includes a pre-trained image encoder and a pre-trained criss-cross attention module.

[0093] In some embodiments, step A2 is implemented as follows: extracting features of at least two video frames in a sample video through a pre-trained image encoder to obtain image features of at least two video frames; extracting sequence features between image features of at least two video frames through a pre-trained cross-attention module to obtain video features of the sample video.

[0094] The structure of the video encoder is obtained by migrating and transforming the image encoder. Since a video clip can be regarded as consisting of at least two consecutive video frames, in order to better understand multiple video frames based on time series, after extracting image features from at least two video frames, the cross-attention module is used to further extract and understand the correlation features between the image features of at least two video frames based on time series to obtain the video features of the sample video.

[0095] In some embodiments, step A3 is implemented as follows: with the goal of reducing the difference between the video features of the sample video and the text features of the video description text, adjusting the parameters of the pre-trained image encoder and the pre-trained cross-attention module to obtain the trained image encoder and the trained text encoder.

[0096] The parameters of the pre-trained image encoder and the pre-trained cross attention module are adjusted based on the first loss function value to obtain the trained image encoder and the trained text encoder. Optionally, the first loss function value may be a loss function value calculated based on a contrast loss function.

[0097] By setting the structure of the video encoder to a structure including an image encoder and a cross-attention module, the difficulty of building the video encoder is reduced, the operation of the design of the video encoder is facilitated, and thus the difficulty of training the video encoder is reduced.

[0098] In some embodiments, the training process of the video encoder includes at least one of the following steps B1 to B6.

[0099] Step B1, obtaining at least one image-text pair, each image-text pair including a sample image and an image description text corresponding to the sample image.

[0100] The sample image may be any image. The present application does not limit the image content of the sample image. The image description text corresponding to the sample image is the text used to describe the image content of the sample image.

[0101] Step B2, extracting features of the sample image through the initial image encoder to obtain image features of the sample image, and extracting features of the image description text through the initial text encoder to obtain text features of the image description text.

[0102] Step B3, with the goal of reducing the difference between the image features of the sample image and the text features of the image description text, adjust the parameters of the initial image encoder and the initial text encoder to obtain a pre-trained image encoder and a pre-trained text encoder.

[0103] A second loss function value is calculated according to the image features of the sample image and the text features of the image description text, and the parameters of the initial image encoder and the initial text encoder are adjusted according to the second loss function value to obtain a pre-trained image encoder and a pre-trained text encoder.

[0104] Optionally, the second loss function value is a loss function value calculated based on the contrast loss function, and the image features of the sample image and the text features of the image description text are cross-modally aligned through the second loss function value to obtain the above-mentioned pre-trained image encoder and pre-trained text encoder.

[0105] By aligning and training the image encoder and the text encoder before the video encoder, the image encoder and the text encoder are preliminarily trained, and the parameters of the pre-trained image encoder and the pre-trained text encoder are used for the training of the video encoder, which can reduce the training difficulty of the video encoder, shorten the training time of the video encoder, and improve the training efficiency of the video encoder.

[0106] After obtaining the pre-trained image encoder and the pre-trained text encoder, steps B4 to B6 are performed.

[0107] Step B4, obtaining at least one video-text pair, each video-text pair including a sample video and a video description text corresponding to the sample video.

[0108] Step B5, extracting features of the sample video by using a pre-trained video encoder to obtain video features of the sample video, and extracting features of the video description text by using a pre-trained text encoder to obtain text features of the video description text.

[0109] Step B6, with the goal of reducing the difference between the video features of the sample video and the text features of the video description text, adjust the parameters of the pre-trained video encoder and the pre-trained text encoder to obtain the trained video encoder and the trained text encoder, and the text encoder is used to extract the features of the illegal text.

[0110] The training process of the video encoder can be referred to Figure 6 As shown, the training process of the video encoder includes two stages. The first stage is to pre-train the image encoder and text encoder through image-text pairs, adjust the parameters of the image encoder and text encoder using the contrast loss function, and align the image features and text features across modalities to obtain the trained image encoder and text encoder of the first stage. The second stage is to add a cross-attention module after the trained image encoder of the first stage to form a video encoder, and use the trained text encoder of the first stage for the second stage, and then train the video encoder and text encoder again through video-text pairs, adjust the parameters of the video encoder and text encoder using the contrast loss function, and align the video features and text features across modalities to obtain the trained video encoder and text encoder of the second stage. The trained video encoder of the second stage will be used for feature extraction of video clips in the video recognition process, and the trained text encoder of the second stage will be used for feature extraction of illegal text in the offline stage.

[0111] In some embodiments, if each illegal text feature in the illegal text feature library has a corresponding determination threshold, the process of determining the determination threshold corresponding to the illegal text feature includes at least one of the following steps C1 to C4.

[0112] Step C1, obtaining a library of illegal videos and a library of compliant videos, wherein the library of illegal videos includes at least one illegal video, and the library of compliant videos includes at least one compliant video. An illegal video refers to a video containing illegal content, and a compliant video refers to a video not containing illegal content.

[0113] This application does not limit the content and type of illegal videos and compliant videos.

[0114] Step C2, obtaining, by a video encoder, a violation video feature library corresponding to the violation video library and a compliance video feature library corresponding to the compliance video library according to the violation video library and the compliance video library.

[0115] The features of at least one illegal video in the illegal video library are extracted by a video encoder to obtain an illegal video feature library corresponding to the illegal video library. The illegal video feature library contains illegal video features corresponding to at least one illegal video.

[0116] The features of at least one compliant video in the compliant video library are extracted by a video encoder to obtain a compliant video feature library corresponding to the compliant video library. The compliant video feature library contains compliant video features corresponding to at least one compliant video.

[0117] Step C3, for each illegal text feature in the illegal text feature library, obtain the similarity corresponding to at least one illegal video feature in the illegal video feature library and the similarity corresponding to at least one compliant video feature in the compliant video feature library according to the illegal video feature library, the compliant video feature library and the illegal text feature, wherein the similarity corresponding to the illegal video feature refers to the similarity between the illegal video feature and the illegal text feature, and the similarity corresponding to the compliant video feature refers to the similarity between the compliant video feature and the illegal text feature.

[0118] For each illegal text feature in the illegal text feature library, the similarity between at least one illegal video feature and the illegal text feature in the illegal video feature library is calculated respectively to obtain the similarity corresponding to at least one illegal video feature. And the similarity between at least one compliant video feature and the illegal text feature in the compliant video feature library is calculated respectively to obtain the similarity corresponding to at least one compliant video feature.

[0119] Step C4, obtaining a determination threshold corresponding to the illegal text feature according to the similarity corresponding to at least one illegal video feature and the similarity corresponding to at least one compliant video feature.

[0120] For each illegal text feature in the illegal text feature library, the illegal similarity set and the compliant similarity set of the illegal text feature are obtained through the similarity between the illegal text feature and the illegal video feature and the compliant video feature. Then, based on the threshold search algorithm, the judgment threshold corresponding to the illegal text feature is obtained, so that the obtained judgment threshold is consistent with the hit judgment of the video content of a large number of videos, thereby ensuring the rationality of the threshold design.

[0121] In some embodiments, step C4 includes at least one of sub-steps C41 to C43.

[0122] Sub-step C41, based on at least one evaluation threshold, obtains the accuracy and recall rate corresponding to at least one evaluation threshold according to the similarity corresponding to at least one illegal video feature and the similarity corresponding to at least one compliant video feature, and the evaluation threshold is used to determine the hit information of the illegal text feature.

[0123] The numerical range of at least one evaluation threshold is 0 to 1. The evaluation threshold is set by the technician according to the threshold determination requirements, and is used to determine the threshold of the hit information of the illegal text feature, and is not limited in this application. Exemplarily, at least one evaluation threshold can be an evaluation threshold set composed of 0.01 as an interval unit, and this application does not limit the minimum and maximum values ​​of at least one evaluation threshold. For example, at least one evaluation threshold can be 0.51, 0.52, 0.53, 0.54, and so on.

[0124] In some embodiments, based on the evaluation threshold, the similarity corresponding to at least one illegal video feature and the similarity corresponding to at least one compliant video feature, the hit information of at least one illegal video feature for the illegal text feature and the hit information of at least one compliant video feature for the illegal text feature are obtained. Based on the hit information of at least one illegal video feature for the illegal text feature and the hit information of at least one compliant video feature for the illegal text feature, the precision and recall corresponding to the evaluation threshold are obtained.

[0125] Exemplarily, for each illegal text feature in the illegal text feature library, if the similarity corresponding to the illegal video feature is greater than or equal to the evaluation threshold, it is determined that the illegal video corresponding to the illegal video feature hits the illegal text corresponding to the illegal text feature, that is, it is determined that the illegal video corresponding to the illegal video feature contains the illegal content represented by the illegal text. If the similarity corresponding to the illegal video feature is less than the evaluation threshold, it is determined that the illegal video corresponding to the illegal video feature does not hit the illegal text corresponding to the illegal text feature, that is, it is determined that the illegal video corresponding to the illegal video feature does not contain the illegal content represented by the illegal text. If the similarity corresponding to the compliant video feature is greater than or equal to the evaluation threshold, it is determined that the compliant video corresponding to the compliant video feature hits the illegal text corresponding to the illegal text feature, that is, it is determined that the compliant video corresponding to the compliant video feature contains the illegal content represented by the illegal text. If the similarity corresponding to the compliant video feature is less than the evaluation threshold, it is determined that the compliant video corresponding to the compliant video feature does not hit the illegal text corresponding to the illegal text feature, that is, it is determined that the compliant video corresponding to the compliant video feature does not contain the illegal content represented by the illegal text.

[0126] The illegal videos that contain illegal content can be regarded as true positives (TP), the illegal videos that do not contain illegal content in the compliant videos can be regarded as true negatives (TN), the compliant videos that contain illegal content in the compliant videos can be regarded as false positives (FP), and the compliant videos that do not contain illegal content in the illegal videos can be regarded as false negatives (FN).

[0127] Accuracy refers to the probability of making a correct judgment among all judgments, that is, the accuracy includes the probability of judging an illegal video as containing illegal content in an illegal video, and judging a compliant video as not containing illegal content in a compliant video. The accuracy corresponding to the evaluation threshold can be expressed as: (TP+TN) / (TP+TN+FN+FP). The recall rate refers to the probability of judging an illegal video as containing illegal content in the judgment of an illegal video. The recall rate corresponding to the evaluation threshold can be expressed as: TP / (TP+FN). Therefore, based on the hit information of at least one illegal video feature for the illegal text feature and the hit information of at least one compliant video feature for the illegal text feature, the accuracy and recall rate corresponding to the evaluation threshold can be calculated.

[0128] The above method can be used to obtain the hit information of the illegal video features and the compliant video features for the illegal text features, and the actual classification of each illegal video and the compliant video can be obtained, so that the accuracy and recall rate corresponding to the evaluation threshold can be calculated, which is used to determine the subsequent judgment threshold, so as to ensure the accuracy and rationality of the judgment threshold as much as possible.

[0129] Sub-step C42, obtaining a balance score corresponding to at least one evaluation threshold value according to the precision and recall rate corresponding to at least one evaluation threshold value.

[0130] Exemplarily, the balance score may be an F1 score, and the balance score=2 / (1 / precision+1 / recall).

[0131] In sub-step C43, the evaluation threshold corresponding to the maximum value of the balance scores corresponding to at least one evaluation threshold is determined as the determination threshold corresponding to the illegal text feature.

[0132] Each evaluation threshold can be calculated to obtain a corresponding balance score, at least one evaluation threshold corresponds to at least one balance score, and the evaluation threshold corresponding to the maximum value of the balance score in the at least one balance score is determined as the determination threshold corresponding to the illegal text feature.

[0133] By determining the evaluation threshold corresponding to the maximum value of the balance score as the decision threshold corresponding to the illegal text feature, the obtained decision threshold can balance the accuracy and recall rate to the greatest extent, improve the rationality of the decision threshold, and allow the decision threshold to be more accurately used to determine the hit information of the illegal text feature.

[0134] In some embodiments, step C4 includes at least one of sub-steps C44 to C47.

[0135] Sub-step C44, based on at least one evaluation threshold, obtains the accuracy, recall and false alarm rate corresponding to at least one evaluation threshold according to the similarity corresponding to at least one illegal video feature and the similarity corresponding to at least one compliant video feature.

[0136] The false alarm rate refers to the probability of judging a compliant video as containing illegal content in the judgment of compliant videos. The false alarm rate corresponding to the evaluation threshold can be expressed as: FP / (TN+FP).

[0137] Sub-step C45, obtaining a balance score corresponding to at least one first threshold according to the precision and recall rate corresponding to at least one evaluation threshold.

[0138] The calculation of the balance score may refer to the above embodiment, which will not be described in detail here.

[0139] Sub-step C46, ​​obtaining a comprehensive score corresponding to at least one evaluation threshold value according to the false alarm rate corresponding to at least one evaluation threshold value and the balance score corresponding to at least one evaluation threshold value.

[0140] In some embodiments, for each evaluation threshold of at least one evaluation threshold, a comprehensive score corresponding to the evaluation threshold is calculated according to the false alarm rate corresponding to the evaluation threshold and the weight corresponding to the false alarm rate, and the balance score corresponding to the evaluation threshold and the weight corresponding to the balance score, and the sum of the weight corresponding to the false alarm rate and the weight corresponding to the balance score is 1. Comprehensive score corresponding to the evaluation threshold=false alarm rate corresponding to the evaluation threshold×weight corresponding to the false alarm rate+balance score corresponding to the evaluation threshold×weight corresponding to the balance score.

[0141] The specific values ​​of the weight corresponding to the false alarm rate and the weight corresponding to the balance score are set by the technician according to the hit requirement, and this application does not limit it. Exemplarily, if the weight corresponding to the false alarm rate is equal to the weight corresponding to the balance score, then the comprehensive score corresponding to the evaluation threshold = the false alarm rate corresponding to the evaluation threshold + the balance score corresponding to the evaluation threshold.

[0142] Sub-step C47, determining the evaluation threshold corresponding to the maximum value of the comprehensive scores corresponding to at least one evaluation threshold as the determination threshold corresponding to the illegal text feature.

[0143] Each evaluation threshold can be calculated to obtain a corresponding comprehensive score. At least one evaluation threshold corresponds to at least one comprehensive score. The evaluation threshold corresponding to the maximum value of the comprehensive score in the at least one comprehensive score is determined as the determination threshold corresponding to the illegal text feature.

[0144] By determining the evaluation threshold corresponding to the maximum value of the comprehensive score as the decision threshold corresponding to the illegal text feature, the obtained decision threshold can balance the false alarm rate and the balance score to the greatest extent, avoiding the false alarm rate being too high and causing the negative example to be judged as a positive example, while avoiding the balance score, and avoiding the imbalance between the precision rate and the recall rate due to the balance score being too low, thereby improving the rationality of the decision threshold and allowing the decision threshold to be more accurately used to determine the hit information of the illegal text feature.

[0145] The determination process of the judgment threshold can refer to Figure 7 As shown, a text encoder is used to obtain a violation text feature library corresponding to the violation text library, and a video encoder is used to obtain a violation video feature library corresponding to the violation video library and a compliance video feature library corresponding to the compliance video library according to the violation video library and the compliance video library. For each violation text feature in the violation text feature library, the similarity between the violation text feature and the violation video feature and the compliance video feature is calculated respectively to obtain a violation similarity set and a compliance similarity set of the violation text feature. According to the violation similarity set and the compliance similarity set of the violation text feature, at least one evaluation index is calculated, and based on the threshold search algorithm, a determination threshold corresponding to the violation text feature is obtained. In sub-steps C41 to C43, at least one evaluation index includes accuracy, recall rate and balance score. In sub-steps C44 to C47, at least one evaluation index includes accuracy, recall rate, false alarm rate and balance score.

[0146] In some embodiments, step 240 may be followed by step 260 (not shown in the figure), and step 260 is located before step 250 .

[0147] Step 260, when the first similarity among the similarities corresponding to at least one illegal text feature is greater than or equal to the judgment threshold, and the first similarity is less than or equal to the first threshold, obtain the answer text corresponding to the first question text according to the video clip and the first question text through the multimodal video understanding model.

[0148] Among them, the first question text is a question text regarding whether the video clip contains illegal content corresponding to the first illegal text, the first illegal text is the illegal text corresponding to the first similarity, and the answer text corresponding to the first question text is used to indicate whether the video clip contains illegal content corresponding to the first illegal text.

[0149] The first threshold is greater than the judgment threshold. The specific value of the first threshold is set by the technical personnel according to the hit requirements and is not limited in this application.

[0150] The first similarity refers to the similarity corresponding to the first illegal text feature, and the first illegal text is the illegal text corresponding to the first illegal text feature.

[0151] When there is a first similarity in the similarities corresponding to at least one illegal text feature, the first similarity is greater than or equal to the judgment threshold, and the first similarity is less than or equal to the first threshold, there is doubt as to whether the judgment on the hit information of the first illegal text feature obtained in step 240 is accurate, that is, there is doubt as to whether the video clip contains the illegal content represented by the first illegal text. Therefore, it is necessary to use a multimodal understanding model to judge again whether the video clip contains the illegal content represented by the first illegal text, and the obtained answer text is used to indicate whether the video clip finally contains the illegal content corresponding to the first illegal text.

[0152] According to the hit information of the first illegal text feature obtained in step 240, a first question text is generated. The first question text is a question text for whether the video clip contains illegal content corresponding to the first illegal text, and is used to ask whether the video clip contains illegal content corresponding to the first illegal text. The video clip and the first question text are input into the multimodal video understanding model, and the answer text corresponding to the first question text is output. The answer text corresponding to the first question text is used to indicate whether the video clip contains illegal content corresponding to the first illegal text.

[0153] The multimodal video understanding model used in step 260 can be any publicly available large video language model, such as a natural language model based on a transformer structure obtained by training with a large amount of data, and the large amount of data can reach a sample level of more than 100 million, which is not limited in this application. Exemplarily, the multimodal video understanding model used in step 260 can be any multimodal video understanding model of at least one multimodal video understanding model used in the above sub-step A12.

[0154] The structural diagram of the multimodal video understanding model can be referred to Figure 8 As shown in Figure 1, the multimodal video understanding model includes a video encoder, a text encoder, and a large language model. v and question text X q Input to the multimodal video understanding model, extract the video segment X through the video encoder v The features of the video clip are obtained by v , extract question text X through text encoder q The features of the question text are obtained by q , the video feature H of the video clip v and the text features H of the question text q Input into the large language model and output the answer text X corresponding to the question text a .

[0155] If there are multiple similarities among the similarities corresponding to at least one illegal text feature, all of which are greater than or equal to the judgment threshold, and all of the multiple similarities are less than or equal to the first threshold, then based on the hit information of the illegal text features corresponding to the multiple similarities, question texts for the illegal text features corresponding to the multiple similarities are generated. The multimodal video understanding model obtains the corresponding answer text based on the video clip and the question texts of the illegal text features corresponding to the multiple similarities.

[0156] The above method can be used to re-judge the illegal text features that have been determined to be hits, thereby avoiding inaccurate initial judgments that lead to illegal content that is not actually included in the final video recognition results, affecting the user's video upload, and improving the accuracy of video recognition.

[0157] In some embodiments, step 240 may further include step 270 (not shown in the figure), and step 270 is located before step 250. Step 270 may be performed simultaneously with step 260, or one of step 260 and step 270 may be performed. Exemplarily, if the video recognition method needs to improve the accuracy of video recognition, step 260 may be performed after step 240 and before step 250. If the video recognition method needs to improve the recall rate of video recognition, step 270 may be performed after step 240 and before step 250.

[0158] Step 270, when the second similarity among the similarities corresponding to at least one illegal text feature is less than the determination threshold and the second similarity is greater than or equal to the second threshold, an answer text corresponding to the second question text is obtained according to the video clip and the second question text by using a multimodal video understanding model.

[0159] Among them, the second question text is a question text regarding whether the video clip contains illegal content corresponding to the second illegal text, the second illegal text is the illegal text corresponding to the second similarity, and the answer text corresponding to the second question text is used to indicate whether the video clip contains illegal content corresponding to the second illegal text.

[0160] The second threshold is smaller than the judgment threshold. The specific value of the second threshold is set by the technical personnel according to the hit requirement, and this application does not limit it.

[0161] The second similarity refers to the similarity corresponding to the second illegal text feature, and the second illegal text is the illegal text corresponding to the second illegal text feature.

[0162] When there is a second similarity in the similarities corresponding to at least one illegal text feature, the second similarity is less than the judgment threshold, and the second similarity is greater than or equal to the second threshold, there is doubt as to whether the judgment on the hit information of the second illegal text feature obtained in step 240 is accurate, that is, there is doubt as to whether the video clip contains the illegal content represented by the second illegal text. Therefore, it is necessary to use a multimodal understanding model to judge again whether the video clip contains the illegal content represented by the second illegal text, and the obtained answer text is used to indicate whether the video clip finally contains the illegal content corresponding to the second illegal text.

[0163] According to the hit information of the second illegal text feature obtained in step 240, a second question text is generated. The second question text is a question text for whether the video clip contains illegal content corresponding to the second illegal text, and is used to ask whether the video clip contains illegal content corresponding to the second illegal text. The video clip and the second question text are input into the multimodal video understanding model, and the answer text corresponding to the second question text is output. The answer text corresponding to the second question text is used to indicate whether the video clip contains illegal content corresponding to the second illegal text.

[0164] The multimodal video understanding model used in step 270 can be any publicly available large video language model, such as a natural language model based on a transformer structure obtained by training with a large amount of data, such as a sample level of more than 100 million, which is not limited in this application. Exemplarily, the multimodal video understanding model used in step 270 can be any multimodal video understanding model of at least one multimodal video understanding model used in the above sub-step A12.

[0165] If step 270 is performed simultaneously with step 260, the multimodal video understanding model used in step 260 is the same as the multimodal video understanding model used in step 270. If either step 260 or step 270 is performed, the multimodal video understanding model used in step 260 may be the same as or different from the multimodal video understanding model used in step 270, and this application does not limit this.

[0166] If there are multiple similarities among the similarities corresponding to at least one illegal text feature that are all less than the judgment threshold, and the multiple similarities are all greater than or equal to the second threshold, then based on the hit information of the illegal text features corresponding to the multiple similarities, a question text for the illegal text features corresponding to the multiple similarities is generated. The multimodal video understanding model obtains the corresponding answer text based on the video clip and the question text of the illegal text features corresponding to the multiple similarities.

[0167] The above method can be used to re-judge the illegal text features that have been judged as missed, so as to avoid the omission of the actual illegal content in the final video recognition results due to inaccurate initial judgment, which may lead to errors in video review. It improves the accuracy of video recognition and the generalization ability of video recognition, and can recall more illegal content.

[0168] Fig. 9 A schematic diagram of the complete process of the video recognition method is shown. First, a video clip is obtained, and the features of the video clip are extracted by a video encoder to obtain the video features of the video clip. Then, based on the illegal text feature library corresponding to the illegal text library extracted in the offline stage, the similarity between at least one illegal text feature in the illegal text feature library and the video feature is calculated. According to the similarity corresponding to at least one illegal text feature and the judgment threshold of the illegal text feature library, a strategy decision is made to obtain the hit information of at least one illegal text feature. The hit information of a part of the illegal text features is recalled and reviewed by the multimodal video understanding model, and the hit information of the part of the illegal text features includes the hit information of the illegal text features initially judged to be hits and the hit information of the illegal text features initially judged to be misses.

[0169] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0170] Please refer to Fig.10 , which shows a block diagram of a video recognition device provided by an embodiment of the present application. The device has the function of implementing the above-mentioned video recognition method, and the function can be implemented by hardware, or by hardware executing corresponding software. The device can be the server introduced above, or it can be set in the server. Fig.10 As shown, the device 1000 may include: a video acquisition module 1010 , a feature extraction module 1020 , a similarity determination module 1030 , a hit determination module 1040 and a prompt generation module 1050 .

[0171] The video acquisition module 1010 is used to acquire video clips.

[0172] The feature extraction module 1020 is used to extract the features of the video clip through a video encoder to obtain the video features of the video clip.

[0173] The similarity determination module 1030 is used to obtain the similarity corresponding to at least one illegal text feature in the illegal text feature library based on the video feature and the illegal text feature library, wherein the similarity corresponding to the illegal text feature refers to the similarity between the illegal text feature and the video feature, and the illegal text feature library is a feature library obtained by extracting features from at least one illegal text in the illegal text library.

[0174] The hit determination module 1040 is used to obtain hit information of the at least one illegal text feature according to the similarities respectively corresponding to the at least one illegal text feature and the determination threshold of the illegal text feature library, wherein the hit information is used to indicate the illegal content contained in the video clip.

[0175] The prompt generating module 1050 is used to generate prompt information of the video segment based on the hit information when the video segment contains illegal content, and the prompt information of the video segment is used to prompt the illegal content contained in the video segment.

[0176] In some embodiments, the video encoder is trained as follows:

[0177] Acquire at least one video-text pair, each video-text pair comprising a sample video and a video description text corresponding to the sample video;

[0178] Extracting features of the sample video by a pre-trained video encoder to obtain video features of the sample video, and extracting features of the video description text by a pre-trained text encoder to obtain text features of the video description text;

[0179] With the goal of reducing the difference between the video features of the sample video and the text features of the video description text, the parameters of the pre-trained video encoder and the pre-trained text encoder are adjusted to obtain the trained video encoder and the trained text encoder, and the text encoder is used to extract the features of the illegal text.

[0180] In some embodiments, the video encoder includes an image encoder and a cross-attention module; the training process of the video encoder includes:

[0181] Extracting features of at least two video frames in the sample video by using a pre-trained image encoder to obtain image features of the at least two video frames;

[0182] Extracting sequence features between image features of the at least two video frames through a pre-trained cross attention module to obtain video features of the sample video;

[0183] With the goal of reducing the difference between the video features of the sample video and the text features of the video description text, the parameters of the pre-trained image encoder and the pre-trained cross-attention module are adjusted to obtain the trained image encoder and the trained text encoder.

[0184] In some embodiments, the video encoder training process further includes:

[0185] Acquire at least one image-text pair, each image-text pair comprising a sample image and an image description text corresponding to the sample image;

[0186] Extracting features of the sample image through an initial image encoder to obtain image features of the sample image, and extracting features of the image description text through an initial text encoder to obtain text features of the image description text;

[0187] With the goal of reducing the difference between the image features of the sample image and the text features of the image description text, the parameters of the initial image encoder and the initial text encoder are adjusted to obtain the pre-trained image encoder and the pre-trained text encoder.

[0188] In some embodiments, the process of obtaining the video text pair is as follows:

[0189] Acquire a video data set, wherein the video data set includes at least one of the sample videos;

[0190] Obtaining at least one video description corresponding to the sample video according to the sample video through at least one multimodal video understanding model;

[0191] Aggregating the at least one video description through a large language model to obtain a video description text corresponding to the sample video;

[0192] The video-text pair is obtained according to the sample video and the video description text corresponding to the sample video.

[0193] In some embodiments, each illegal text feature in the illegal text feature library has a corresponding determination threshold; the hit determination module 1040 is used to:

[0194] According to the similarities respectively corresponding to the at least one illegal text feature and the determination thresholds respectively corresponding to the at least one illegal text feature, the hit information of the at least one illegal text feature is obtained.

[0195] In some embodiments, the hit determination module 1040 is used to:

[0196] If the similarity corresponding to the illegal text feature is greater than or equal to the determination threshold corresponding to the illegal text feature, it is determined that the video clip hits the illegal text corresponding to the illegal text feature;

[0197] If the similarity corresponding to the illegal text feature is less than the determination threshold corresponding to the illegal text feature, it is determined that the video clip does not hit the illegal text corresponding to the illegal text feature.

[0198] In some embodiments, the apparatus 1000 further includes a threshold determination module, the threshold determination module being configured to:

[0199] Obtaining a library of illegal videos and a library of compliant videos, wherein the illegal video library includes at least one illegal video, and the compliant video library includes at least one compliant video, wherein the illegal video refers to a video containing illegal content, and the compliant video refers to a video not containing illegal content;

[0200] Obtaining, by the video encoder, a violation video feature library corresponding to the violation video library and a compliance video feature library corresponding to the compliance video library according to the violation video library and the compliance video library;

[0201] For each illegal text feature in the illegal text feature library, according to the illegal video feature library, the compliant video feature library and the illegal text feature, obtain the similarity corresponding to at least one illegal video feature in the illegal video feature library and the similarity corresponding to at least one compliant video feature in the compliant video feature library, wherein the similarity corresponding to the illegal video feature refers to the similarity between the illegal video feature and the illegal text feature, and the similarity corresponding to the compliant video feature refers to the similarity between the compliant video feature and the illegal text feature;

[0202] A determination threshold corresponding to the illegal text feature is obtained according to the similarity corresponding to the at least one illegal video feature and the similarity corresponding to the at least one compliant video feature.

[0203] In some embodiments, the threshold determination module is used to:

[0204] Based on at least one evaluation threshold, according to the similarity corresponding to the at least one illegal video feature and the similarity corresponding to the at least one compliant video feature, the accuracy and recall rate corresponding to the at least one evaluation threshold are obtained, and the evaluation threshold is used to determine the hit information of the illegal text feature;

[0205] Obtaining a balance score corresponding to the at least one evaluation threshold value according to the precision rate and the recall rate corresponding to the at least one evaluation threshold value;

[0206] Among the balance scores corresponding to the at least one evaluation threshold, the evaluation threshold corresponding to the maximum value of the balance score is determined as the determination threshold corresponding to the illegal text feature.

[0207] In some embodiments, the threshold determination module is used to:

[0208] Obtaining, according to the evaluation threshold, the similarity corresponding to the at least one illegal video feature and the similarity corresponding to the at least one compliant video feature, hit information of the at least one illegal video feature for the illegal text feature and hit information of the at least one compliant video feature for the illegal text feature;

[0209] According to the hit information of the at least one illegal video feature for the illegal text feature and the hit information of the at least one compliant video feature for the illegal text feature, the accuracy and recall rate corresponding to the evaluation threshold are obtained.

[0210] In some embodiments, the threshold determination module is used to:

[0211] Based on at least one evaluation threshold, according to the similarity corresponding to the at least one illegal video feature and the similarity corresponding to the at least one compliant video feature, obtaining the accuracy, recall rate and false alarm rate corresponding to the at least one evaluation threshold respectively;

[0212] Obtaining a balance score corresponding to each of the at least one first threshold according to the precision and recall rate corresponding to each of the at least one evaluation threshold;

[0213] Obtaining a comprehensive score corresponding to each of the at least one evaluation threshold according to the false alarm rate corresponding to each of the at least one evaluation threshold and the balance score corresponding to each of the at least one evaluation threshold;

[0214] Among the comprehensive scores corresponding to the at least one evaluation threshold, the evaluation threshold corresponding to the maximum value of the comprehensive score is determined as the determination threshold corresponding to the illegal text feature.

[0215] In some embodiments, the apparatus 1000 further includes a model answering module, wherein the model answering module is configured to:

[0216] When a first similarity among the similarities respectively corresponding to the at least one illegal text feature is greater than or equal to the determination threshold, and the first similarity is less than or equal to the first threshold, obtaining an answer text corresponding to the first question text according to the video clip and the first question text by using a multimodal video understanding model;

[0217] Among them, the first question text is a question text regarding whether the video clip contains illegal content corresponding to the first illegal text, the first illegal text is the illegal text corresponding to the first similarity, and the answer text corresponding to the first question text is used to indicate whether the video clip contains illegal content corresponding to the first illegal text.

[0218] In some embodiments, the model answer module is used to:

[0219] When a second similarity among the similarities respectively corresponding to the at least one illegal text feature is less than the determination threshold, and the second similarity is greater than or equal to the second threshold, obtaining an answer text corresponding to the second question text according to the video clip and the second question text by using a multimodal video understanding model;

[0220] Among them, the second question text is a question text regarding whether the video clip contains illegal content corresponding to the second illegal text, the second illegal text is the illegal text corresponding to the second similarity, and the answer text corresponding to the second question text is used to indicate whether the video clip contains illegal content corresponding to the second illegal text.

[0221] It should be noted that the device provided in the above embodiment, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the content structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0222] Please refer to Fig.11 , which shows a block diagram of a computer device 1100 provided in one embodiment of the present application. The computer device 1100 may be any electronic device with data calculation, processing and storage functions. The computer device 1100 may be used to implement the video recognition method provided in the above embodiment.

[0223] Typically, the computer device 1100 includes a processor 1101 and a memory 1102 .

[0224] The processor 1101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1101 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1101 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1101 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1101 may also include an AI processor for processing computing operations related to machine learning.

[0225] The memory 1102 may include one or more computer-readable storage media, which may be non-transitory. The memory 1102 may also include a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1102 is used to store a computer program, which is configured to be executed by one or more processors to implement the above-mentioned video recognition method.

[0226] Those skilled in the art will understand that Fig.11 The structure shown in the figure does not constitute a limitation on the computer device 1100, and the computer device 1100 may include more or less components than shown in the figure, or combine some components, or adopt a different component arrangement.

[0227] In an illustrative embodiment, a computer-readable storage medium is also provided, in which a computer program is stored, and when the computer program is executed by a processor of a computer device, the video recognition method is implemented. Optionally, the computer-readable storage medium may be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0228] In an exemplary embodiment, a computer program product is also provided, the computer program product includes a computer program, the computer program is stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the computer device performs the above-mentioned video recognition method.

[0229] It should be noted that this application can display a prompt interface, pop-up window or output voice prompt information before collecting relevant data of users and during the process of collecting relevant data of users. The prompt interface, pop-up window or voice prompt information is used to prompt the user that relevant data is currently being collected, so that this application only starts to execute the relevant steps of obtaining relevant data of users after obtaining the confirmation operation issued by the user to the prompt interface or pop-up window, otherwise (that is, when the confirmation operation issued by the user to the prompt interface or pop-up window is not obtained), the relevant steps of obtaining relevant data of users are terminated, that is, the relevant data of users are not obtained. In other words, all user data collected by this application are processed strictly in accordance with the requirements of relevant national laws and regulations, and the informed consent or separate consent of the subject of personal information is obtained with the consent and authorization of the user. The subsequent data use and processing behavior is carried out within the scope of authorization of laws and regulations and the subject of personal information, and the collection, use and processing of relevant user data shall comply with the relevant laws, regulations and standards of relevant countries and regions.

[0230] It should be understood that the "multiple" mentioned in this article refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. In addition, the step numbers described in this article only illustrate a possible execution sequence between the steps. In some other embodiments, the above steps may not be executed in the order of the numbers, such as two steps with different numbers are executed at the same time, or two steps with different numbers are executed in the opposite order to the diagram. The embodiments of the present application are not limited to this.

[0231] The above description is only an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A video recognition method, characterized in that: The method comprises: Get video clips; Extracting features of the video clip by a video encoder to obtain video features of the video clip; According to the video feature and the illegal text feature library, a similarity corresponding to at least one illegal text feature in the illegal text feature library is obtained, wherein the similarity corresponding to the illegal text feature refers to the similarity between the illegal text feature and the video feature, and the illegal text feature library is a feature library obtained by extracting features from at least one illegal text in the illegal text library; Obtaining hit information of the at least one illegal text feature according to the similarities respectively corresponding to the at least one illegal text feature and the determination threshold of the illegal text feature library, wherein the hit information is used to indicate illegal content contained in the video clip; In the case that the video clip contains illegal contents, prompt information of the video clip is generated based on the hit information, and the prompt information of the video clip is used to prompt the illegal contents contained in the video clip.

2. The method according to claim 1, characterized in that: The training process of the video encoder is as follows: Acquire at least one video-text pair, each video-text pair comprising a sample video and a video description text corresponding to the sample video; Extracting features of the sample video by a pre-trained video encoder to obtain video features of the sample video, and extracting features of the video description text by a pre-trained text encoder to obtain text features of the video description text; With the goal of reducing the difference between the video features of the sample video and the text features of the video description text, the parameters of the pre-trained video encoder and the pre-trained text encoder are adjusted to obtain the trained video encoder and the trained text encoder, and the text encoder is used to extract the features of the illegal text.

3. The method according to claim 2, characterized in that The video encoder includes an image encoder and a cross attention module; the extracting the features of the sample video by the pre-trained video encoder to obtain the video features of the sample video includes: Extracting features of at least two video frames in the sample video by using a pre-trained image encoder to obtain image features of the at least two video frames; Extracting sequence features between image features of the at least two video frames through a pre-trained cross attention module to obtain video features of the sample video; The method aims to reduce the difference between the video features of the sample video and the text features of the video description text, and adjusts the parameters of the pre-trained video encoder and the pre-trained text encoder to obtain the trained video encoder and the trained text encoder, including: With the goal of reducing the difference between the video features of the sample video and the text features of the video description text, the parameters of the pre-trained image encoder and the pre-trained cross-attention module are adjusted to obtain the trained image encoder and the trained text encoder.

4. The method according to claim 3, characterized in that Before obtaining at least one video-text pair, the method further includes: Acquire at least one image-text pair, each image-text pair comprising a sample image and an image description text corresponding to the sample image; Extracting features of the sample image through an initial image encoder to obtain image features of the sample image, and extracting features of the image description text through an initial text encoder to obtain text features of the image description text; With the goal of reducing the difference between the image features of the sample image and the text features of the image description text, the parameters of the initial image encoder and the initial text encoder are adjusted to obtain the pre-trained image encoder and the pre-trained text encoder.

5. The method according to claim 2, characterized in that: The process of obtaining the video text pair is as follows: Acquire a video data set, wherein the video data set includes at least one of the sample videos; Obtaining at least one video description corresponding to the sample video according to the sample video through at least one multimodal video understanding model; Aggregating the at least one video description through a large language model to obtain a video description text corresponding to the sample video; The video-text pair is obtained according to the sample video and the video description text corresponding to the sample video.

6. The method according to claim 1, characterized in that Each illegal text feature in the illegal text feature library has a corresponding determination threshold; The step of obtaining hit information of the at least one illegal text feature according to the similarities respectively corresponding to the at least one illegal text feature and the determination threshold of the illegal text feature library comprises: According to the similarities respectively corresponding to the at least one illegal text feature and the determination thresholds respectively corresponding to the at least one illegal text feature, the hit information of the at least one illegal text feature is obtained.

7. The method according to claim 6, characterized in that The obtaining, according to the similarities respectively corresponding to the at least one illegal text feature and the determination threshold respectively corresponding to the at least one illegal text feature, the hit information of the at least one illegal text feature comprises: If the similarity corresponding to the illegal text feature is greater than or equal to the determination threshold corresponding to the illegal text feature, it is determined that the video clip hits the illegal text corresponding to the illegal text feature; If the similarity corresponding to the illegal text feature is less than the determination threshold corresponding to the illegal text feature, it is determined that the video clip does not hit the illegal text corresponding to the illegal text feature.

8. The method according to claim 6, characterized in that The method further comprises: Obtaining a library of illegal videos and a library of compliant videos, wherein the illegal video library includes at least one illegal video, and the compliant video library includes at least one compliant video, wherein the illegal video refers to a video containing illegal content, and the compliant video refers to a video not containing illegal content; Obtaining, by the video encoder, a violation video feature library corresponding to the violation video library and a compliance video feature library corresponding to the compliance video library according to the violation video library and the compliance video library; For each illegal text feature in the illegal text feature library, according to the illegal video feature library, the compliant video feature library and the illegal text feature, obtain the similarity corresponding to at least one illegal video feature in the illegal video feature library and the similarity corresponding to at least one compliant video feature in the compliant video feature library, wherein the similarity corresponding to the illegal video feature refers to the similarity between the illegal video feature and the illegal text feature, and the similarity corresponding to the compliant video feature refers to the similarity between the compliant video feature and the illegal text feature; A determination threshold corresponding to the illegal text feature is obtained according to the similarity corresponding to the at least one illegal video feature and the similarity corresponding to the at least one compliant video feature.

9. The method according to claim 8, characterized in that The obtaining, according to the similarity respectively corresponding to the at least one illegal video feature and the similarity respectively corresponding to the at least one compliant video feature, a determination threshold corresponding to the illegal text feature comprises: Based on at least one evaluation threshold, according to the similarity corresponding to the at least one illegal video feature and the similarity corresponding to the at least one compliant video feature, the accuracy and recall rate corresponding to the at least one evaluation threshold are obtained, and the evaluation threshold is used to determine the hit information of the illegal text feature; Obtaining a balance score corresponding to the at least one evaluation threshold value according to the precision rate and the recall rate corresponding to the at least one evaluation threshold value; Among the balance scores corresponding to the at least one evaluation threshold, the evaluation threshold corresponding to the maximum value of the balance score is determined as the determination threshold corresponding to the illegal text feature.

10. The method according to claim 9, characterized in that The obtaining, based on at least one evaluation threshold, the accuracy and recall rate corresponding to the at least one evaluation threshold respectively according to the similarity respectively corresponding to the at least one illegal video feature and the similarity respectively corresponding to the at least one compliant video feature, comprises: Obtaining, according to the evaluation threshold, the similarity corresponding to the at least one illegal video feature and the similarity corresponding to the at least one compliant video feature, hit information of the at least one illegal video feature for the illegal text feature and hit information of the at least one compliant video feature for the illegal text feature; According to the hit information of the at least one illegal video feature for the illegal text feature and the hit information of the at least one compliant video feature for the illegal text feature, the accuracy and recall rate corresponding to the evaluation threshold are obtained.

11. The method according to claim 8, characterized in that The obtaining, according to the similarity respectively corresponding to the at least one illegal video feature and the similarity respectively corresponding to the at least one compliant video feature, a determination threshold corresponding to the illegal text feature comprises: Based on at least one evaluation threshold, according to the similarity corresponding to the at least one illegal video feature and the similarity corresponding to the at least one compliant video feature, obtaining the accuracy, recall rate and false alarm rate corresponding to the at least one evaluation threshold respectively; Obtaining a balance score corresponding to each of the at least one first threshold according to the precision and recall rate corresponding to each of the at least one evaluation threshold; Obtaining a comprehensive score corresponding to each of the at least one evaluation threshold according to the false alarm rate corresponding to each of the at least one evaluation threshold and the balance score corresponding to each of the at least one evaluation threshold; Among the comprehensive scores corresponding to the at least one evaluation threshold, the evaluation threshold corresponding to the maximum value of the comprehensive score is determined as the determination threshold corresponding to the illegal text feature.

12. The method according to claim 1, characterized in that The method further comprises: When a first similarity among the similarities respectively corresponding to the at least one illegal text feature is greater than or equal to the determination threshold, and the first similarity is less than or equal to the first threshold, obtaining an answer text corresponding to the first question text according to the video clip and the first question text by using a multimodal video understanding model; Among them, the first question text is a question text regarding whether the video clip contains illegal content corresponding to the first illegal text, the first illegal text is the illegal text corresponding to the first similarity, and the answer text corresponding to the first question text is used to indicate whether the video clip contains illegal content corresponding to the first illegal text.

13. The method according to claim 1, characterized in that The method further comprises: When a second similarity among the similarities respectively corresponding to the at least one illegal text feature is less than the determination threshold, and the second similarity is greater than or equal to the second threshold, obtaining an answer text corresponding to the second question text according to the video clip and the second question text by using a multimodal video understanding model; Among them, the second question text is a question text regarding whether the video clip contains illegal content corresponding to the second illegal text, the second illegal text is the illegal text corresponding to the second similarity, and the answer text corresponding to the second question text is used to indicate whether the video clip contains illegal content corresponding to the second illegal text.

14. A video recognition device, characterized in that: The device comprises: A video acquisition module, used to acquire video clips; A feature extraction module, used to extract features of the video clip through a video encoder to obtain video features of the video clip; A similarity determination module, for obtaining, based on the video feature and the illegal text feature library, the similarity corresponding to at least one illegal text feature in the illegal text feature library, wherein the similarity corresponding to the illegal text feature refers to the similarity between the illegal text feature and the video feature, and the illegal text feature library is a feature library obtained by extracting features from at least one illegal text in the illegal text library; A hit determination module, used to obtain hit information of the at least one illegal text feature according to the similarities respectively corresponding to the at least one illegal text feature and the determination threshold of the illegal text feature library, wherein the hit information is used to indicate illegal content contained in the video clip; The prompt generating module is used to generate prompt information of the video segment based on the hit information when the video segment contains illegal content, and the prompt information of the video segment is used to prompt the illegal content contained in the video segment.

15. A computer device, characterized in that: The computer device includes a processor and a memory, wherein a computer program is stored in the memory, and the computer program is loaded and executed by the processor to implement the video recognition method according to any one of claims 1 to 13.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the video recognition method according to any one of claims 1 to 13.

17. A computer program product, characterized in that The computer program product comprises a computer program, which is loaded and executed by a processor to implement the video recognition method according to any one of claims 1 to 13.

Images