Prohibited video identification method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a prohibited video identification method, device and equipment and a storage medium, and relates to the technical field of image processing, the method comprises the following steps: determining a current video frame of a current video and a current text corresponding to the current video frame, the current text comprising a first text and / or a second text, the first text comprises character information in the current video frame, and the second text is a text converted from an audio corresponding to the current video frame; the current video frame and the current text are input into a trained recognition model to obtain a current recognition result, the trained recognition model carries out prohibited statement recognition on the current text based on a set corpus, and the set corpus comprises at least two prohibited statements; and determining whether the current video is a prohibited video according to the current identification result. By adopting the technical scheme of the embodiment of the invention, the problem of low accuracy of the existing non-artificial prohibited video identification method can be solved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of image processing technology, and in particular to a prohibited video recognition method, device, equipment and storage medium. Background Art

[0002] At present, manual review is basically used to review whether the videos to be released or live videos are banned videos. Although the efficiency of manual review is low, it is still the mainstream method for reviewing banned videos.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following problems in the prior art:

[0004] The low accuracy of non-manual prohibited video identification methods limits their promotion and application. Summary of the invention

[0005] The embodiments of the present invention provide a prohibited video recognition method, device, equipment and storage medium to solve the problem of low accuracy in existing non-manual prohibited video recognition methods.

[0006] In a first aspect, an embodiment of the present invention provides a method for identifying prohibited videos, the method comprising:

[0007] Determine a current video frame of a current video and a current text corresponding to the current video frame, wherein the current text includes a first text and / or a second text, wherein the first text includes character information in the current video frame, and the second text is a text converted from an audio corresponding to the current video frame;

[0008] Inputting the current video frame and the current text into a trained recognition model to obtain a current recognition result, wherein the trained recognition model performs prohibited image recognition on the current video frame to obtain an image recognition result, performs prohibited sentence recognition on the current text based on a set corpus to obtain a text recognition result, and determines the current recognition result based on the image recognition result and the text recognition result, wherein the set corpus includes at least two prohibited sentences;

[0009] Determine whether the current video is a prohibited video according to the current recognition result.

[0010] In a second aspect, an embodiment of the present invention further provides a prohibited video recognition device, the device comprising:

[0011] A determination module, used to determine a current video frame of a current video and a current text corresponding to the current video frame, wherein the current text includes a first text and / or a second text, wherein the first text includes character information in the current video frame, and the second text is a text converted from the audio corresponding to the current video frame;

[0012] a recognition module, configured to input the current video frame and the current text into a trained recognition model to obtain a current recognition result, wherein the trained recognition model performs prohibited image recognition on the current video frame to obtain an image recognition result, performs prohibited sentence recognition on the current text based on a set corpus to obtain a text recognition result, and determines the current recognition result based on the image recognition result and the text recognition result, wherein the set corpus includes at least two prohibited sentences;

[0013] A determination module is used to determine whether the current video is a prohibited video according to the current recognition result.

[0014] In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising:

[0015] one or more processors;

[0016] A memory for storing one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the prohibited video identification method provided by any embodiment of the present invention.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a prohibited video identification method as provided in any embodiment of the present invention.

[0019] The embodiments of the above invention have the following advantages or beneficial effects:

[0020] The embodiment of the present invention performs prohibited image recognition on the current video frame through a trained recognition model to obtain an image recognition result, performs prohibited sentence recognition on the current text based on a set corpus to obtain a text recognition result, determines the current recognition result based on the image recognition result and the text recognition result, and determines whether the current video is a prohibited video based on the current recognition result; since the set corpus includes at least two prohibited sentences, its minimum unit is a sentence, which enables the trained recognition model to directly perform sentence recognition, and determines whether the recognized sentence is a prohibited sentence based on the set corpus. Compared with the prior art of first identifying sensitive words, inferring the semantics of the sentence where the sensitive words are located based on the sensitive words, and then determining whether the sentence where the sensitive words are located is a prohibited sentence based on the semantic inference result, it can identify prohibited sentences that do not include sensitive words or prohibited sentences that do not include significant sensitive words, thereby improving the accuracy of prohibited sentence recognition, that is, the accuracy of the text recognition result, and the accuracy of the text recognition result can improve the accuracy of judging whether the current video is a prohibited video. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a prohibited video identification method provided by an embodiment of the present invention;

[0022] Figure 2 It is a flowchart of a method for determining a current video frame and its corresponding current text provided by an embodiment of the present invention;

[0023] Figure 3 Another flow chart of the prohibited video identification method provided by the embodiment of the present invention;

[0024] Figure 4 Another flow chart of the prohibited video identification method provided by the embodiment of the present invention;

[0025] Figure 5A is a schematic diagram of the structure of a prohibited video identification device provided by an embodiment of the present invention;

[0026] Figure 5B is another structural schematic diagram of a prohibited video identification device provided by an embodiment of the present invention;

[0027] Figure 5C is another structural schematic diagram of a prohibited video identification device provided by an embodiment of the present invention;

[0028] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0030] Figure 1 The flowchart of a prohibited video identification method provided by an embodiment of the present invention is applicable to the case where a video is determined to be a prohibited video based on a video frame, text information in the video frame, and audio corresponding to the video frame. The method can be executed by a prohibited image identification device integrated in an electronic device, and the device can be implemented by software and / or hardware. Figure 1 As shown, the method specifically comprises the following steps:

[0031] S110. Determine a current video frame of a current video and a current text corresponding to the current video frame, where the current text includes a first text and / or a second text, the first text includes character information in the current video frame, and the second text is a text converted from the audio corresponding to the current video frame.

[0032] The current video is a video to be played or a live video.

[0033] In one embodiment, the current video is a current video segment obtained by sampling the target video based on a set time interval. This embodiment reduces the amount of data processing in the process of determining prohibited videos through a sampling method.

[0034] The first text includes subtitles and bullet comments in the current video frame. Subtitles are character information added to the video when the video is produced, and bullet comments refer to sentences input by viewers during video playback. In one embodiment, the first text also includes the annotation characters of the objects contained in the video frame.

[0035] S120, inputting the current video frame and the current text into the trained recognition model to obtain a current recognition result, wherein the trained recognition model performs prohibited image recognition on the current video frame to obtain an image recognition result, performs prohibited sentence recognition on the current text based on a set corpus to obtain a text recognition result, and determines the current recognition result based on the image recognition result and the text recognition result, and the set corpus includes at least two prohibited sentences.

[0036] The corpus is set to include at least two forbidden sentence combinations; the forbidden sentence combination includes a first forbidden sentence and at least one second forbidden sentence; the second forbidden sentence is a variation of the first forbidden sentence and is synonymous with the first forbidden sentence. It can be understood that the smallest unit of the corpus is set to be a sentence. Compared with the corresponding first forbidden sentence, the second forbidden sentence includes at least one pinyin and / or at least one synonym and / or at least one homophonic word.

[0037] In one embodiment, for each sentence of each text, the trained text sub-model is used to: determine the preliminary recognition result of the current sentence; if the preliminary recognition result includes a deformation element, and the set corpus includes the first prohibited sentence corresponding to the preliminary recognition result, then generate a text recognition result including prohibited sentence information; wherein the deformation element is a homophone or pinyin; the prohibited sentence information includes the current sentence identifier, and the first prohibited sentence and / or the preliminary recognition result. The current sentence identifier refers to the sequence number of the current sentence in the first text or the second text corresponding to the video frame, such as 001, 005, etc.

[0038] If the preliminary sentence recognition result of any sentence in the first text or the second text is a first forbidden sentence, then the first forbidden sentence is the forbidden sentence recognition result of the sentence; if the preliminary sentence recognition result is a second forbidden sentence, then the first forbidden sentence corresponding to the second forbidden sentence is used as the forbidden sentence recognition result of the sentence.

[0039] Compared with the prior art that identifies sensitive words based on a corpus whose smallest unit is sensitive words, infers sentence semantics based on the identified sensitive words, and then determines whether the corresponding sentence is a banned sentence based on the inferred sentence semantics, in this embodiment, since the smallest unit of the corpus is set to be a sentence, the correspondence between the second banned sentence and the first banned sentence is predetermined and unique, so the banned sentence recognition process is the process of searching for preliminary sentence recognition results in the set corpus, which does not involve semantic inference, and therefore the determined banned sentence recognition results have higher accuracy.

[0040] Exemplarily, assume that the corpus includes the abbreviation MN of sensitive word A, and the abbreviations of sensitive word A and non-sensitive word B are the same, and sentence C includes MN. After identifying MN, the prior art needs to reverse the semantics of the sentence where MN is located, and then determine whether the sentence where MN is located is a prohibited sentence based on the semantic reverse result. If the sentence is a simple sentence, the accuracy of the semantic reverse result is usually relatively high. If the sentence is a complex sentence, the probability of reverse failure is usually also relatively high, and there will be a situation where a non-prohibited sentence is misjudged as a prohibited sentence. In this embodiment, sentence C is directly identified to obtain a preliminary identification result. If the preliminary identification result includes a deformation element, it can be compared with all prohibited sentences included in the set corpus; if the comparison is successful, the first prohibited sentence corresponding to the preliminary sentence identification result is used as the sentence identification result of sentence C; if the comparison fails, it indicates that sentence C is not a prohibited sentence. If the preliminary recognition result does not include the deformation element, it is compared with all the first prohibited sentences included in the set corpus. If the comparison is successful, the first sentence compared is used as the recognition result of the sentence C; if the comparison fails, it indicates that the sentence C is not a prohibited sentence. This example shows that for the scenario of non-significant sensitive words corresponding to pinyin abbreviations, this embodiment can significantly improve the accuracy of prohibited sentence recognition.

[0041] In one embodiment, the trained recognition model includes a trained decision sub-model, a trained image sub-model and at least one trained text sub-model. The trained recognition model determines the image recognition result corresponding to the current video frame through the trained image sub-model; determines the text recognition results corresponding to the first text and / or the second text respectively through at least one trained text sub-model; and determines the current recognition result corresponding to the image recognition result and all text recognition results through the trained decision sub-model. This embodiment achieves the purpose of determining the current recognition result that can accurately reflect whether the current video is a prohibited video based on multimodal information, which helps to improve the accuracy of prohibited video recognition results.

[0042] S130: Determine whether the current video is a prohibited video according to the current recognition result.

[0043] If the current recognition result includes a prohibited image mark and / or a prohibited sentence mark, the current video is regarded as a prohibited video; if the current recognition result does not include a prohibited image video mark and a prohibited sentence mark, the current video is regarded as a non-prohibited video.

[0044] In one embodiment, if the current video is determined to be a prohibited video because the current recognition result includes prohibited sentence information, the prohibited sentence corresponding to the prohibited sentence information is determined; the prohibited sentence in the current video is deleted or blocked to update the current video, and the process returns to S110 until the current video frame is the last video frame of the current video, or the current recognition result includes an image prohibited mark. This embodiment completes the normalization of the current video by automatically deleting the prohibited sentence, thereby simplifying the video compliance management process.

[0045] In one embodiment, if the current recognition result includes a prohibited image mark and the current video is a live video, the current video is marked as a prohibited video and the playback of the current video is stopped; if the current video is a video to be played, the current video is marked as a prohibited video and a prompt message is output to indicate that the video is prohibited from being played.

[0046] In one embodiment, if the current video is determined to be a prohibited video, a prompt message indicating that the current video is a prohibited video is output, and the prohibited information corresponding to the current recognition result is stored at the same time; the prohibited information includes the prohibited video frame and / or prohibited sentence information, the prohibited information includes the prohibited video frame and / or prohibited sentence information, and the prohibited sentence information includes a text identifier, a sentence identifier of the prohibited sentence in the text corresponding to the corresponding text identifier, and the prohibited sentence content. By storing the prohibited information corresponding to the current recognition result, evidence for proving that the current video is a prohibited video can be retained, so that the user can query the evidence at any time.

[0047] The embodiment of the present invention performs prohibited image recognition on the current video frame through a trained recognition model to obtain an image recognition result, performs prohibited sentence recognition on the current text based on a set corpus to obtain a text recognition result, determines the current recognition result based on the image recognition result and the text recognition result, and determines whether the current video is a prohibited video based on the current recognition result; since the set corpus includes at least two prohibited sentences, its minimum unit is a sentence, which enables the trained recognition model to directly perform sentence recognition, and determines whether the recognized sentence is a prohibited sentence based on the set corpus. Compared with the prior art of first identifying sensitive words, inferring the semantics of the sentence where the sensitive words are located based on the sensitive words, and then determining whether the sentence where the sensitive words are located is a prohibited sentence based on the semantic inference result, it can identify prohibited sentences that do not include sensitive words or prohibited sentences that do not include significant sensitive words, thereby improving the accuracy of prohibited sentence recognition, that is, the accuracy of the text recognition result, and the accuracy of the text recognition result can improve the accuracy of judging whether the current video is a prohibited video.

[0048] Figure 2 The present invention provides a flowchart of a method for determining a current video frame and a current text corresponding thereto in an embodiment of the present invention. This embodiment is used to further refine the method of "determining a current video frame of a current video and a current text corresponding thereto" in the above-mentioned embodiment. Figure 2 As shown, the method includes:

[0049] S2101: extracting a current video segment from a current video based on a set time interval, and determining a depth feature of each video frame in the current video segment.

[0050] The time interval may be set according to specific circumstances and is not specifically limited in this embodiment.

[0051] The duration of the current video segment may be set according to specific circumstances, and this embodiment does not make any specific limitation here.

[0052] In one embodiment, the deep features can more effectively represent high-level semantic information and have stronger robustness in modeling appearance changes such as rotation and deformation of the object. The deep features of each video frame in the current video segment can be determined by setting a network model, such as a neural network model.

[0053] In one embodiment, a histogram and / or a point density distribution diagram of the depth features of each video frame is determined; for each video frame, if the current video frame is determined to be a redundant frame based on the histogram and / or the point density distribution diagram of the current video frame, the current video frame is deleted from the current video segment to update the current video segment. By deleting the redundant frames, the amount of data calculation in the subsequent prohibited video determination process is reduced, and the speed of prohibited video identification is improved.

[0054] In one embodiment, a first clustering algorithm is used to perform cluster analysis on the depth features corresponding to each video frame in the current video segment to obtain a first clustering result; based on the first clustering result, video frames that meet the first clarity condition are screened out from the current video segment to obtain a video frame combination; a second clustering algorithm is used to perform cluster analysis on all the depth features corresponding to the video frame combination to obtain a second clustering result; based on the second clustering result, video frames that meet the second clarity condition are screened out from the video frame combination to update the current video segment again. The first clustering algorithm may be a partitioning clustering algorithm, such as a K-means clustering algorithm or a K center point algorithm, and the second clustering algorithm may be a hierarchical clustering algorithm, such as an agglomerative hierarchical clustering algorithm or a balanced iterative reduction and clustering using hierarchies (BIRCH). This embodiment uses different clustering algorithms to perform cluster analysis on all depth features corresponding to each video frame in the current video segment in turn, and deletes video frames that do not meet the clarity requirements from the current video segment based on the clustering results, which helps to improve the accuracy of subsequent image recognition results, the accuracy of the first text determination, and the accuracy of text recognition results, thereby improving the accuracy of the current video determination result of whether it is a prohibited video.

[0055] S2102: Cluster the deep features of all negative samples in the negative sample library and the deep features of all video frames in the current video segment to obtain a clustering result.

[0056] The negative sample library includes at least two negative samples. Negative samples refer to video frames marked as prohibited video frames.

[0057] By clustering the deep features of all negative samples in the negative sample library and the deep features of all video frames of the current video segment, at least two clusters can be obtained. In this embodiment, the clustering method is not specifically limited, and an existing clustering algorithm can be used, such as the K-mean clustering algorithm.

[0058] S2103 . For each video frame, determine the category to which the current video frame belongs, and select a target negative sample corresponding to the current video frame from all negative samples corresponding to the category.

[0059] This step aims to select target negative samples from the negative sample library that have a high similarity with the current video frame.

[0060] It can be understood that for any cluster, the similarity between video frames within a class is greater than the similarity between video frames between classes. For this reason, this embodiment selects a negative sample from the class where the current video frame belongs, and uses the negative sample as the target negative sample corresponding to the current video frame.

[0061] In one embodiment, a negative sample is randomly selected from the category of the current video frame, and the negative sample is used as the target negative sample corresponding to the current video frame. This embodiment has a high speed of selecting negative samples.

[0062] In one embodiment, the negative sample in the category where the current video frame belongs that is closest to the current video frame is used as the target negative sample corresponding to the current video frame. The target negative sample selected in this embodiment has the highest similarity with the current video frame.

[0063] S2104: Determine the average of the similarities between all video frames in the current video segment and the corresponding target negative samples.

[0064] After the target negative samples corresponding to all the video frames in the current video segment are determined, the mean similarity between all the video frames in the current video segment and the corresponding target negative samples is determined. The mean can reflect the overall possibility that the current video segment is a prohibited video.

[0065] S2105: If the mean value meets the set similarity condition, determine the current text corresponding to the current video frame, and determine the current text corresponding to the current video frame.

[0066] In one embodiment, the similarity condition is set to a similarity threshold. If the mean value is greater than the similarity threshold, the current video segment is determined to be a suspected prohibited video, and the current text corresponding to the current video frame is determined; if the mean value is less than or equal to the similarity threshold, the current video segment is determined to be a non-prohibited video.

[0067] The embodiment of the present invention determines a target negative sample in the negative sample library that has a high similarity with the current video frame by clustering the deep features of all negative samples in the negative sample library and the deep features of all video frames in the current video segment; determines the mean of the similarities between all video frames in the current video segment and the corresponding target negative samples, and compares the mean with a set similarity condition to determine whether the current video segment is a suspected prohibited video, and in the case that the current video segment is a suspected prohibited video, determines the current text corresponding to the current video frame; achieves the technical effect of filtering out non-suspected prohibited videos through similarity comparison, reduces the amount of data calculation in the process of determining prohibited videos, and improves the speed of determining prohibited videos.

[0068] Figure 3 Flow chart of the prohibited video recognition method provided by an embodiment of the present invention. This embodiment is used to refine the "inputting the current video frame and the current text into the trained recognition model to obtain the current recognition result" in the above embodiment. Figure 3 As shown, the method includes:

[0069] S310. Determine a current video frame of a current video and a current text corresponding to the current video frame, wherein the current text includes a first text and / or a second text, wherein the first text includes character information in the current video frame, and the second text is a text converted from the audio corresponding to the current video frame.

[0070] S3201. Determine image feature data of a current video frame, and text feature data of a first text and / or text feature data of a second text.

[0071] In one embodiment, the image feature data of the current video frame and the text feature data of the first text and / or the text feature data of the second text are determined by setting a coding model. The set network model may be a coding model based on a neural network.

[0072] S3202: normalize the image feature data and the text feature data of the first text and / or the text feature data of the second text to obtain normalized image feature data and the text feature data of the first text and / or the text feature data of the second text.

[0073] In one embodiment, the current text includes a first text and a second text, and the image feature data, the text feature data of the first text, and the text feature data of the second text are normalized by the following formula, which is as follows:

[0074]

[0075] The image feature data of the current video frame, the text feature data of the first text, and the text feature data of the second text are substituted into x in the above formula. max 、x min They are respectively the maximum feature data and the minimum feature data among the image feature data of the current video frame, the text feature data of the first text, and the text feature data of the second text.

[0076] S3203: Input the normalized image feature data and the text feature data of the first text and / or the text feature data of the second text into the trained recognition model to obtain a current recognition result.

[0077] Since the normalization of feature data makes feature data of different dimensions comparable in value, the accuracy of the recognition result is improved. Therefore, the normalized image feature data and the text feature data of the first text and / or the text feature data of the second text are input into the trained recognition model, which can improve the data processing speed of the trained recognition model.

[0078] S330: Determine whether the current video is a prohibited video according to the current recognition result.

[0079] The embodiment of the present invention normalizes the image feature data and the text feature data of the first text and / or the text feature data of the second text, and inputs the normalized image feature data and the text feature data of the first text and / or the text feature data of the second text into a trained recognition model, thereby improving the accuracy of the current recognition result, thereby improving the accuracy of the result of determining whether the current video is a prohibited video.

[0080] Figure 4 Flow chart of the prohibited video identification method provided by the embodiment of the present invention. This embodiment is used to refine the above embodiment. Figure 4 As shown, the method includes:

[0081] S410. Determine a current video frame of a current video and a current text corresponding to the current video frame, wherein the current text includes a first text and a second text, wherein the first text includes character information in the current video frame, and the second text is a text converted from the audio corresponding to the current video frame.

[0082] An image character recognition method is used to recognize characters in the current video frame to obtain a first text; an automatic audio recognition method is used to convert the audio corresponding to the current video frame into text, and the text is used as the second text.

[0083] S420, inputting the current video frame and the current text into the trained recognition model to obtain a current recognition result, wherein the trained recognition model performs prohibited image recognition on the current video frame to obtain an image recognition result, performs prohibited sentence recognition on the current text based on a set corpus to obtain a text recognition result, and determines the current recognition result based on the image recognition result and the text recognition result, and the set corpus includes at least two prohibited sentences.

[0084] A neural network encoder is used to extract image feature data of the current video frame, and text feature data of the first text and text feature data of the second text are extracted; the image feature data, the text feature data of the first text and the text feature data of the second text are normalized, and the normalized image feature data, the text feature data of the first text and the text feature data of the second text are input into a trained recognition model to obtain a current recognition result.

[0085] The trained recognition model includes a trained image sub-model, a trained first text sub-model, a trained second text sub-model and a decision model; the normalized image feature data is input into the trained image sub-model to obtain an image recognition result; the text feature data of the first text is input into the trained first text sub-model to obtain a first text recognition result; the text feature data of the second text is input into the trained second text sub-model to obtain a second text recognition result. The image recognition result, the first text recognition result and the second text recognition result are input into the trained decision sub-model to obtain the current recognition result. Among them, the trained image sub-model performs prohibited image recognition through softmax to obtain an image recognition result; the trained first text sub-model and the trained second text sub-model perform corresponding prohibited sentence recognition through softmax to obtain corresponding first text recognition result and second text recognition result.

[0086] Among them, the first text sub-model and the second text sub-model are both constructed based on the Transformer model. When the trained first text sub-model and the second text sub-model perform sentence recognition on the corresponding text, the text corresponding to the index with the highest probability in each component in the prediction sequence is used as the character recognition result, and the sorting result of all characters in the corresponding sentence is used as the preliminary sentence recognition result to determine whether there are homophonic words or pinyin in the preliminary sentence recognition result; if so, reverse engineering is performed based on the set corpus to restore the normal sentence information of the preliminary sentence recognition result. Specifically, the set corpus is traversed to determine the first prohibited sentence corresponding to the preliminary recognized sentence, and the first prohibited sentence is used as the recognition result of the current sentence.

[0087] Among them, the image sub-model can select the Ghost model, so as to use fewer parameters, generate the same amount of image feature data as the ordinary convolution layer, and improve the determination speed of the image recognition result.

[0088] Among them, the trained image sub-model, the trained first text sub-model and the trained second text sub-model integrate the attention mechanism in the process of prohibited image recognition or the process of prohibited sentence recognition, and act on the original feature data by evaluating channel association through branch learning. Based on global pooling, more effective global information is extracted, and finally corresponding weights are assigned to each channel to indicate the importance of feature data of different channels.

[0089] The trained decision sub-model performs weighted averaging on the image recognition result, the first text recognition result and the second text recognition result based on the ensemble learning strategy of the weighted average algorithm to complete the decision-level feature fusion and obtain the multimodal fusion result.

[0090] Among them, each sub-model is a heterogeneous learner, which is used to realize training and learning in different dimensions; the method of determining the multimodal fusion result through the weighted average algorithm is as follows:

[0091]

[0092] in, is the result of multimodal fusion, n=f1+f2+…+f k , n is the number of sub-models or the number of recognition results, x i is the recognition result of the sub-model marked as i, f i is the weight of the recognition result of the sub-model identified as i.

[0093] A current video segment is extracted from the current video based on a set time interval, and a histogram and / or a point density distribution diagram of depth features of each video frame of the current video segment is determined; for each video frame, if the current video frame is determined to be a redundant frame based on the histogram and / or the point density distribution diagram of the current video frame, the current video frame is deleted from the current video segment to update the current video segment.

[0094] A k-means clustering algorithm is used to perform cluster analysis on the depth features corresponding to each video frame in the current video segment to obtain a first clustering result; based on the first clustering result, video frames that meet a first clarity condition are screened out from the current video segment to obtain a video frame combination; a cohesive layer clustering algorithm is used to perform cluster analysis on all depth features corresponding to the video frame combination to obtain a second clustering result; based on the second clustering result, video frames that meet a second clarity condition are screened out from the video frame combination to update the current video segment again.

[0095] Determine the deep features of each video frame in the current video segment; cluster the deep features of all negative samples in the negative sample library and the deep features of all video frames in the current video segment to obtain a clustering result; for each video frame, determine the category to which the current video frame belongs, and select a target negative sample from all negative samples corresponding to the category; determine the mean of the similarity between all video frames in the current video segment and the corresponding target negative samples; if the mean meets the set similarity condition, determine the current text corresponding to the current video frame.

[0096] The method for constructing the corpus in this embodiment includes: obtaining historical comment information of a video of a set category, or historical comment information and subtitle information. Take each sentence of the corpus as a first forbidden sentence; for each first forbidden sentence, divide it into several parts, remove stop words, and replace each word therein with a homophone, pinyin or synonym to obtain the corresponding second forbidden sentence. It can be understood that each first forbidden sentence corresponds to at least one second forbidden sentence, and the combination of the first forbidden sentence and all its corresponding second forbidden sentences is taken as a forbidden sentence combination. The set of forbidden sentence combinations corresponding to all corpora is taken as a corpus.

[0097] The feedback information of users on bad texts in the form of new words is obtained in real time to determine the prohibited sentence combination corresponding to the bad text, thereby achieving the technical effect of updating the corpus in real time.

[0098] In this embodiment, the image sub-model, the first text sub-model, and the second text sub-model need to be trained separately to obtain the trained image sub-model, the trained first text sub-model, and the trained second text sub-model. Based on the output results of the trained image sub-model, the trained first text sub-model, and the trained second text sub-model, the training of the decision sub-model is completed to obtain the trained decision sub-model, thereby obtaining a trained recognition model including the trained image sub-model, the trained first text sub-model, the trained second text sub-model, and the trained decision sub-model. In the training process of the decision sub-model, the activation of normalized neurons is adopted, and the LayerNorm layer (normalization layer) is introduced, that is, the variance and mean of the recognition result set of each training sample are calculated, and the recognition result set of each training sample is converted into data with a variance of 1 and a mean of 0. The recognition result set includes the image recognition result, the first text recognition result, and the second text recognition result.

[0099] S430: Determine whether the current video is a prohibited video according to the current recognition result.

[0100] The embodiments of the present invention significantly improve the accuracy of prohibited sentence recognition, that is, the accuracy of text recognition results, and the accuracy of text recognition results can improve the accuracy of determining whether the current video is a prohibited video.

[0101] The following is an embodiment of a prohibited video identification device provided by an embodiment of the present invention. The device and the prohibited video identification method of the above-mentioned embodiment belong to the same inventive concept. For details not described in detail in the embodiment of the prohibited video identification device, reference can be made to the contents of the above-mentioned embodiments.

[0102] Figure 5A Schematic diagram of the structure of the device provided by the embodiment of the present invention. Figure 5A As shown, the device comprises:

[0103] A determination module 510 is used to determine a current video frame of a current video and a current text corresponding to the current video frame, wherein the current text includes a first text and / or a second text, wherein the first text includes character information in the current video frame, and the second text is a text converted from the audio corresponding to the current video frame;

[0104] a recognition module 520, configured to input the current video frame and the current text into a trained recognition model to obtain a current recognition result, wherein the trained recognition model performs prohibited image recognition on the current video frame to obtain an image recognition result, performs prohibited sentence recognition on the current text based on a set corpus to obtain a text recognition result, and determines the current recognition result based on the image recognition result and the text recognition result, wherein the set corpus includes at least two prohibited sentences;

[0105] The determination module 530 is used to determine whether the current video is a prohibited video according to the current recognition result.

[0106] In one embodiment, the set corpus includes at least two prohibited sentence combinations;

[0107] The prohibited sentence combination includes a first prohibited sentence and at least one second prohibited sentence;

[0108] The second prohibited sentence is a variation of the first prohibited sentence and has the same meaning as the first prohibited sentence.

[0109] In one embodiment, the trained recognition model is used to:

[0110] Determine the image recognition result corresponding to the current video frame by using the trained image sub-model;

[0111] Determine text recognition results corresponding to the first text and / or the second text respectively through at least one trained text sub-model;

[0112] The current recognition result corresponding to the image recognition result and all the text recognition results is determined by the trained decision sub-model.

[0113] In one embodiment, for each sentence in each text, the trained text sub-model is used to:

[0114] Determine the preliminary recognition result of the current sentence;

[0115] If the preliminary recognition result includes a deformation element, determining a first prohibited sentence corresponding to the preliminary recognition result based on a set corpus;

[0116] A text recognition result including prohibited sentence information is generated, wherein the prohibited sentence information includes the current sentence identifier, the first prohibited sentence and / or the preliminary recognition result, and the deformation element is a homophonic word or pinyin.

[0117] In one embodiment, Figure 5B As shown, the device further includes an output module 540, which is used to:

[0118] Outputting prompt information indicating that the current video is a prohibited video, and storing prohibited information corresponding to the current recognition result;

[0119] The prohibited information includes the prohibited video frame and / or prohibited sentence information, and the prohibited sentence information includes a text identifier, a sentence identifier of the prohibited sentence in a text corresponding to the text identifier, and prohibited sentence content.

[0120] In one embodiment, the determination module 510 is specifically configured to:

[0121] Extracting a current video segment from the current video based on a set time interval, and determining a depth feature of each video frame in the current video segment;

[0122] Clustering the deep features of all negative samples in the negative sample library and the deep features of all video frames of the current video segment to obtain a clustering result;

[0123] For each of the video frames, determining the category of the current video frame, and selecting a target negative sample corresponding to the current video frame from all negative samples corresponding to the category;

[0124] Determining a mean value of similarities between all video frames in the current video segment and corresponding target negative samples;

[0125] If the mean value meets the set similarity condition, the current text corresponding to the current video frame is determined.

[0126] In one embodiment, the determination module 510 is further configured to:

[0127] Determine a histogram and / or a point density distribution diagram of the depth features of each of the video frames;

[0128] For each video frame, if it is determined that the current video frame is a redundant frame based on the histogram and / or the point density distribution diagram of the current video frame, the current video frame is deleted from the current video segment to update the current video segment.

[0129] In one embodiment, the determination module 510 is further configured to:

[0130] Using a first clustering algorithm to perform clustering analysis on the depth features corresponding to each video frame in the current video segment to obtain a first clustering result;

[0131] Filtering video frames that meet a first definition condition from the current video segment according to the first clustering result to obtain a video frame combination;

[0132] Using a second clustering algorithm to perform clustering analysis on all depth features corresponding to the video frame combination to obtain a second clustering result;

[0133] According to the second clustering result, video frames meeting a second definition condition are screened out from the video frame combination to update the current video segment again.

[0134] In one embodiment, Figure 5C As shown, the apparatus further includes an iteration module 550, and the iteration module 550 is used to:

[0135] If the current video is determined to be a prohibited video because the current recognition result includes prohibited sentence information, determining a prohibited sentence corresponding to the prohibited sentence information;

[0136] The prohibited sentence in the current video is deleted or blocked to update the current video, and the process returns to the step of determining a current video frame of the current video and a current text corresponding to the current video frame.

[0137] The embodiment of the present invention performs prohibited image recognition on the current video frame through a trained recognition model to obtain an image recognition result, performs prohibited sentence recognition on the current text based on a set corpus to obtain a text recognition result, determines the current recognition result based on the image recognition result and the text recognition result, and determines whether the current video is a prohibited video based on the current recognition result; since the set corpus includes at least two prohibited sentences, its minimum unit is a sentence, which enables the trained recognition model to directly perform sentence recognition, and determines whether the recognized sentence is a prohibited sentence based on the set corpus. Compared with the prior art of first identifying sensitive words, inferring the semantics of the sentence where the sensitive words are located based on the sensitive words, and then determining whether the sentence where the sensitive words are located is a prohibited sentence based on the semantic inference result, it can identify prohibited sentences that do not include sensitive words or prohibited sentences that do not include significant sensitive words, thereby improving the accuracy of prohibited sentence recognition, that is, the accuracy of the text recognition result, and the accuracy of the text recognition result can improve the accuracy of judging whether the current video is a prohibited video.

[0138] The prohibited video identification device provided in the embodiment of the present invention can execute the prohibited video identification method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the prohibited video identification method.

[0139] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 6 A block diagram of an exemplary server 12 suitable for use in implementing embodiments of the present invention is shown. Figure 6 The server 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0140] like Figure 6 As shown, the server 12 is in the form of a general purpose computing device. The components of the server 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0141] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0142] The server 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the server 12, including volatile and non-volatile media, removable and non-removable media.

[0143] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0144] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0145] The server 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the server 12, and / or may communicate with any device that enables the server 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. In addition, the server 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the server 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the server 12, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0146] The processing unit 16 executes various functional applications and data processing by running the program stored in the system memory 28, for example, implementing the steps of the prohibited video identification method provided in the first embodiment of the present invention, which includes:

[0147] Determine a current video frame of a current video and a current text corresponding to the current video frame, wherein the current text includes a first text and / or a second text, wherein the first text includes character information in the current video frame, and the second text is a text converted from an audio corresponding to the current video frame;

[0148] Inputting the current video frame and the current text into a trained recognition model to obtain a current recognition result, wherein the trained recognition model performs prohibited image recognition on the current video frame to obtain an image recognition result, performs prohibited sentence recognition on the current text based on a set corpus to obtain a text recognition result, and determines the current recognition result based on the image recognition result and the text recognition result, wherein the set corpus includes at least two prohibited sentences;

[0149] Determine whether the current video is a prohibited video according to the current recognition result.

[0150] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the prohibited video identification method provided by any embodiment of the present invention.

[0151] This embodiment provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, a prohibited video identification method as provided in the above embodiment of the present invention is implemented. The method includes:

[0152] Determine a current video frame of a current video and a current text corresponding to the current video frame, wherein the current text includes a first text and / or a second text, wherein the first text includes character information in the current video frame, and the second text is a text converted from an audio corresponding to the current video frame;

[0153] Inputting the current video frame and the current text into a trained recognition model to obtain a current recognition result, wherein the trained recognition model performs prohibited image recognition on the current video frame to obtain an image recognition result, performs prohibited sentence recognition on the current text based on a set corpus to obtain a text recognition result, and determines the current recognition result based on the image recognition result and the text recognition result, wherein the set corpus includes at least two prohibited sentences;

[0154] Determine whether the current video is a prohibited video according to the current recognition result.

[0155] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0156] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0157] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0158] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0159] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0160] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A prohibited video identification method, characterized in that: The method comprises: Determine a current video frame of a current video and a current text corresponding to the current video frame, wherein the current text includes a first text and / or a second text, wherein the first text includes character information in the current video frame, and the second text is a text converted from an audio corresponding to the current video frame; Inputting the current video frame and the current text into a trained recognition model to obtain a current recognition result, wherein the trained recognition model performs prohibited image recognition on the current video frame to obtain an image recognition result, performs prohibited sentence recognition on the current text based on a set corpus to obtain a text recognition result, and determines the current recognition result based on the image recognition result and the text recognition result, wherein the set corpus includes at least two prohibited sentences; Determine whether the current video is a prohibited video according to the current recognition result.

2. The method according to claim 1, characterized in that The set corpus includes at least two prohibited sentence combinations; The prohibited sentence combination includes a first prohibited sentence and at least one second prohibited sentence; The second prohibited sentence is a variation of the first prohibited sentence and has the same meaning as the first prohibited sentence.

3. The method according to claim 2, characterized in that The trained recognition model is used to: Determine the image recognition result corresponding to the current video frame by using the trained image sub-model; Determine text recognition results corresponding to the first text and / or the second text respectively through at least one trained text sub-model; The current recognition result corresponding to the image recognition result and all the text recognition results is determined by the trained decision sub-model.

4. The method according to claim 3, characterized in that For each sentence in each text, the trained text sub-model is used to: Determine the preliminary recognition result of the current sentence; If the preliminary recognition result includes a deformation element, and the set corpus includes a first prohibited sentence corresponding to the preliminary recognition result, generating a text recognition result including prohibited sentence information; Among them, the deformation element is a homophonic word or pinyin; the prohibited sentence information includes the current sentence identifier, and the first prohibited sentence and / or the preliminary recognition result.

5. The method according to claim 1, characterized in that While / after determining whether the current video is a prohibited video according to the current recognition result, the method further includes: Outputting prompt information indicating that the current video is a prohibited video, and storing prohibited information corresponding to the current recognition result; The prohibited information includes the prohibited video frame and / or prohibited sentence information, and the prohibited sentence information includes a text identifier, a sentence identifier of the prohibited sentence in a text corresponding to the text identifier, and prohibited sentence content.

6. The method according to claim 1, characterized in that The determining of a current video frame of a current video and a current text corresponding to the current video frame includes: Extracting a current video segment from the current video based on a set time interval, and determining a depth feature of each video frame in the current video segment; Clustering the deep features of all negative samples in the negative sample library and the deep features of all video frames of the current video segment to obtain a clustering result; For each of the video frames, determining the category of the current video frame, and selecting a target negative sample corresponding to the current video frame from all negative samples corresponding to the category; Determining a mean value of similarities between all video frames in the current video segment and corresponding target negative samples; If the mean value meets the set similarity condition, the current text corresponding to the current video frame is determined.

7. The method according to claim 6, characterized in that Before clustering the deep features of all negative samples in the negative sample library and the deep features of all video frames in the current video segment to obtain a clustering result, the method further includes: Determine a histogram and / or a point density distribution diagram of the depth features of each of the video frames; For each video frame, if it is determined that the current video frame is a redundant frame based on the histogram and / or the point density distribution diagram of the current video frame, the current video frame is deleted from the current video segment to update the current video segment.

8. The method according to claim 6, characterized in that Before clustering the deep features of all negative samples in the negative sample library and the deep features of all video frames in the current video segment to obtain a clustering result, the method further includes: Using a first clustering algorithm to perform clustering analysis on the depth features corresponding to each video frame in the current video segment to obtain a first clustering result; Filtering video frames that meet a first definition condition from the current video segment according to the first clustering result to obtain a video frame combination; Using a second clustering algorithm to perform clustering analysis on all depth features corresponding to the video frame combination to obtain a second clustering result; According to the second clustering result, video frames meeting a second definition condition are screened out from the video frame combination to update the current video segment again.

9. The method according to claim 1, characterized in that: After determining whether the current video is a prohibited video according to the current recognition result, the method further includes: If the current video is determined to be a prohibited video because the current recognition result includes prohibited sentence information, determining a prohibited sentence corresponding to the prohibited sentence information; The prohibited sentence in the current video is deleted or blocked to update the current video, and the process returns to the step of determining a current video frame of the current video and a current text corresponding to the current video frame.

10. A prohibited video recognition device, characterized in that: include: A determination module, used to determine a current video frame of a current video and a current text corresponding to the current video frame, wherein the current text includes a first text and / or a second text, wherein the first text includes character information in the current video frame, and the second text is a text converted from the audio corresponding to the current video frame; a recognition module, configured to input the current video frame and the current text into a trained recognition model to obtain a current recognition result, wherein the trained recognition model performs prohibited image recognition on the current video frame to obtain an image recognition result, performs prohibited sentence recognition on the current text based on a set corpus to obtain a text recognition result, and determines the current recognition result based on the image recognition result and the text recognition result, wherein the set corpus includes at least two prohibited sentences; A determination module is used to determine whether the current video is a prohibited video according to the current recognition result.

11. An electronic device, characterized in that: The electronic device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the prohibited video identification method as described in any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the prohibited video identification method as described in any one of claims 1-9 is implemented.

Images