Video barrage review method, device, electronic device and computer storage medium

Abstract

The present application provides a video bullet review method, device, electronic device and computer storage medium. After extracting frames of a target video and obtaining bullet review content and video information of each frame, the bullet review content and video information of each frame are input into a semantic review model to obtain the review score of the bullet review content of the frame, thereby determining the review result of the bullet review content of the frame according to the review score of the bullet review content of the frame; wherein, if the review result of the bullet review content of the frame indicates that it is passed, the bullet review content of the frame is exposed; if the review result of the bullet review content of the frame indicates that it is not passed, the bullet review content of the frame is intercepted. The purpose of fast, efficient and accurate automatic review of the bullet review content is achieved, which greatly reduces the burden of review, saves review costs and improves review efficiency.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a video commentary review method, device, electronic device and computer storage medium. Background Art

[0002] In the Internet industry, video barrage content review is an indispensable part. Traditional review systems mainly rely on reviewers for human review. Although this method has high accuracy, it has low execution efficiency and high cost. Another method is to combine machine review with manual review. The more representative ones are sensitive word filtering, BERT and other technologies, which can automatically review texts. Among them, the sensitive word filtering method has high accuracy and fast speed, but the recall is very low and it is difficult to cover the whole. BERT is a pre-trained language model that can also be used for text classification. Its accuracy is lower than that of human review, and its inference speed is very slow. It cannot be applied to review scenarios with high real-time requirements. Moreover, the above methods can only review the content of barrage, and it is difficult to review in combination with the video context. In fact, the same barrage may have different review results under different videos. Summary of the invention

[0003] In view of this, the present application provides a video barrage review method, device, electronic device and computer storage medium, which can quickly, efficiently and accurately automatically review the barrage content, greatly reducing the review burden, saving review costs and improving review efficiency.

[0004] The first aspect of the present application provides a method for reviewing video comments, comprising:

[0005] Extract frames from the target video to obtain the bullet comment content and video information of each frame;

[0006] For each frame of barrage content and video information, the barrage content and video information of the frame are input into a semantic review model to obtain a review score of the barrage content of the frame; wherein the semantic review model is obtained by training a text classification model according to a first training sample set; the first training sample set includes at least one first training sample; the first training sample includes the barrage content and video text information corresponding to the first historical frame;

[0007] According to the review score of the barrage content of the frame, the review result of the barrage content of the frame is determined; if the review result of the barrage content of the frame indicates passed, the barrage content of the frame is exposed; if the review result of the barrage content of the frame indicates failed, the barrage content of the frame is blocked.

[0008] Optionally, if the review result of the bullet comment content of the frame indicates that it is passed, after revealing the bullet comment content of the frame, the method further includes:

[0009] The exposed barrage content is used as pending barrage content;

[0010] At preset intervals, for each pending barrage content, the pending barrage content and the video information corresponding to the pending barrage content are input into the review model, and the inspection and review results of the pending barrage content are output; wherein the review model is obtained by fine-tuning the pre-trained model according to the second training sample set; the second training sample set includes at least one second training sample; the second training sample includes training task instructions, training sample questions and training sample answers; the training sample questions include the barrage content and video information corresponding to the historical frames;

[0011] If the inspection result of the pending barrage content indicates that it has passed, the pending barrage content is marked and stored in the database;

[0012] If the inspection and review result of the pending barrage content indicates that it fails, the pending barrage content will be recalled.

[0013] Optionally, the method for constructing the audit model includes:

[0014] Constructing a second training sample set; wherein the second training sample set includes at least one second training sample; the second training sample includes a training task instruction, a training sample question and a training sample answer; the training sample question includes the bullet comment content and video text information corresponding to the second historical frame;

[0015] Fix the pre-trained weights of the pre-trained model;

[0016] Feeding all the second training samples into the first low-rank matrix and the second low-rank matrix for training respectively, to obtain a trained first low-rank matrix and a trained second low-rank matrix;

[0017] Determining a target weight according to the trained first low-rank matrix, the trained second low-rank matrix and the pre-trained weight;

[0018] The target weights are updated to the audit model.

[0019] Optionally, determining the target weight according to the trained first low-rank matrix, the trained second low-rank matrix and the pre-trained weight includes:

[0020] Calculate the trained first low-rank matrix and the trained second low-rank matrix according to matrix multiplication to obtain trained low-rank matrix weights;

[0021] The trained low-rank matrix weights are added to the pre-trained weights to obtain target weights.

[0022] Optionally, the method for constructing the semantic audit model includes:

[0023] Constructing a first training sample set; wherein the first training sample set includes at least one first training sample; the first training sample includes the bullet comment content and video text information corresponding to the first historical frame;

[0024] For each first training sample, the first training sample is input into the audit model, and a label of the first training sample is obtained as an output;

[0025] Vectorizing the bullet comment content and the video text content corresponding to the first historical frame respectively to obtain a first vector and a second vector;

[0026] Generate a fusion vector of the first training sample according to the first vector and the second vector;

[0027] The text classification model is trained according to the fusion vectors of all the first training samples and the labels of the first training samples to obtain a semantic review model.

[0028] Optionally, generating a fusion vector of the first training sample according to the first vector and the second vector includes:

[0029] Inputting the first vector into a first gated recurrent unit, outputting a first target vector, and inputting the first vector into a first highway unit, outputting a second target vector;

[0030] Inputting the second vector into a second gated recurrent unit, outputting a third target vector, and inputting the second vector into a second highway unit, outputting a fourth target vector;

[0031] Performing soft attention processing according to the second target vector and the fourth target vector to obtain a first attention vector and a second attention vector;

[0032] Concatenate the first target vector, the second target vector, and the first attention vector to obtain a first concatenated vector;

[0033] Concatenate the third target vector, the fourth target vector, and the second attention vector to obtain a second concatenated vector;

[0034] Inputting the first splicing vector into the first gated recurrent unit to obtain a fifth target vector, and inputting the second splicing vector into the second gated recurrent unit to obtain a sixth target vector;

[0035] A function operation is performed according to the fifth target vector and the sixth target vector to obtain a fusion vector of the first training sample.

[0036] Optionally, performing a function operation on the fifth target vector and the sixth target vector to obtain a fusion vector of the first training sample includes:

[0037] taking the difference between the fifth target vector and the sixth target vector as a seventh target vector;

[0038] taking the product of the fifth target vector and the sixth target vector as an eighth target vector;

[0039] The fifth target vector, the seventh target vector and the eighth target vector are concatenated to obtain a fusion vector of the first training sample.

[0040] The second aspect of the present application provides a video comment review device, including:

[0041] A frame extraction unit is used to extract frames of the target video to obtain the bullet comment content and video information of each frame;

[0042] A first input unit is used to input the barrage content and video information of each frame into a semantic review model to obtain a review score of the barrage content of the frame; wherein the semantic review model is obtained by training a text classification model according to a first training sample set; the first training sample set includes at least one first training sample; the first training sample includes the barrage content and video text information corresponding to the first historical frame;

[0043] The first determination unit is used to determine the review result of the barrage content of the frame according to the review score of the barrage content of the frame; wherein, if the review result of the barrage content of the frame indicates passed, the barrage content of the frame is exposed; if the review result of the barrage content of the frame indicates failed, the barrage content of the frame is intercepted.

[0044] Optionally, the video comment review device further includes:

[0045] A second determining unit, configured to use the exposed bullet comment content as pending bullet comment content;

[0046] The second input unit is used to input the pending barrage content and the video information corresponding to the pending barrage content into the review model at preset intervals for each pending barrage content, and output the inspection and review results of the pending barrage content; wherein the review model is obtained by fine-tuning the pre-trained model according to the second training sample set; the second training sample set includes at least one second training sample; the second training sample includes training task instructions, training sample questions and training sample answers; the training sample questions include the barrage content and video information corresponding to the historical frames;

[0047] A storage unit, for marking and storing the pending barrage content in a database if the inspection and review result of the pending barrage content indicates that the pending barrage content has passed;

[0048] The recall unit is used to recall the pending barrage content if the inspection and review result of the pending barrage content shows that it fails.

[0049] Optionally, the construction unit of the audit model includes:

[0050] A first construction unit is used to construct a second training sample set; wherein the second training sample set includes at least one second training sample; the second training sample includes a training task instruction, a training sample question and a training sample answer; the training sample question includes the bullet comment content and video text information corresponding to the second historical frame;

[0051] A fixed unit is used to fix the pre-trained weights of the pre-trained model;

[0052] A feeding unit, used for feeding all the second training samples into the first low-rank matrix and the second low-rank matrix for training, to obtain a trained first low-rank matrix and a trained second low-rank matrix;

[0053] A third determining unit, configured to determine a target weight according to the trained first low-rank matrix, the trained second low-rank matrix and the pre-trained weight;

[0054] An updating unit is used to update the target weight to the audit model.

[0055] Optionally, the third determining unit includes:

[0056] A first calculation unit is used to calculate the trained first low-rank matrix and the trained second low-rank matrix according to matrix multiplication to obtain a trained low-rank matrix weight;

[0057] The second computing unit is used to add the trained low-rank matrix weight to the pre-trained weight to obtain a target weight.

[0058] Optionally, the construction unit of the semantic audit model includes:

[0059] A second construction unit is used to construct a first training sample set; wherein the first training sample set includes at least one first training sample; the first training sample includes the bullet comment content and video text information corresponding to the first historical frame;

[0060] A third input unit is used for inputting each first training sample into the audit model, and outputting a label of the first training sample;

[0061] A vectorization unit, using language to respectively vectorize the bullet comment content and the video text content corresponding to the first historical frame to obtain a first vector and a second vector;

[0062] a fusion unit, configured to generate a fusion vector of the first training sample according to the first vector and the second vector;

[0063] A training unit is used to train a text classification model according to the fusion vectors of all the first training samples and the labels of the first training samples to obtain a semantic review model.

[0064] Optionally, the fusion unit comprises:

[0065] a fourth input unit, configured to input the first vector into a first gated recurrent unit, output a first target vector, and input the first vector into a first highway unit, output a second target vector;

[0066] a fifth input unit, configured to input the second vector into a second gated recurrent unit, output a third target vector, and input the second vector into a second highway unit, output a fourth target vector;

[0067] a soft attention processing unit, configured to perform soft attention processing according to the second target vector and the fourth target vector to obtain a first attention vector and a second attention vector;

[0068] A first splicing unit, configured to splice the first target vector, the second target vector and the first attention vector to obtain a first splicing vector;

[0069] A second splicing unit, configured to splice the third target vector, the fourth target vector and the second attention vector to obtain a second splicing vector;

[0070] a sixth input unit, configured to input the first splicing vector into the first gated recurrent unit to obtain a fifth target vector, and input the second splicing vector into the second gated recurrent unit to obtain a sixth target vector;

[0071] A function operation unit is used to perform a function operation according to the fifth target vector and the sixth target vector to obtain a fusion vector of the first training sample.

[0072] Optionally, the function operation unit includes:

[0073] a third calculation unit, configured to use a difference between the fifth target vector and the sixth target vector as a seventh target vector;

[0074] a fourth calculation unit, configured to use the product of the fifth target vector and the sixth target vector as an eighth target vector;

[0075] The third concatenation unit is configured to concatenate the fifth target vector, the seventh target vector and the eighth target vector to obtain a fusion vector of the first training sample.

[0076] A third aspect of the present application provides an electronic device, including:

[0077] one or more processors;

[0078] a storage device having one or more programs stored thereon;

[0079] When the one or more programs are executed by the one or more processors, the one or more processors implement the video barrage review method as described in any one of the first aspects.

[0080] A fourth aspect of the present application provides a computer storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the method for reviewing video barrage as described in any one of the first aspects is implemented.

[0081] It can be seen from the above scheme that the present application provides a video barrage review method, device, electronic device and computer storage medium. After extracting frames of the target video and obtaining the barrage content and video information of each frame, the barrage content and video information of each frame are input into the semantic review model to obtain the review score of the barrage content of the frame, thereby determining the review result of the barrage content of the frame according to the review score of the barrage content of the frame; wherein, if the review result of the barrage content of the frame shows that it is passed, the barrage content of the frame is exposed; if the review result of the barrage content of the frame shows that it is not passed, the barrage content of the frame is intercepted. The purpose of fast, efficient and accurate automatic review of the barrage content is achieved, which greatly reduces the burden of review, saves review costs and improves review efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0083] Figure 1 A specific flow chart of a video comment review method provided in an embodiment of the present application;

[0084] Figure 2 A flowchart of a method for constructing a semantic audit model provided in another embodiment of the present application;

[0085] Figure 3 A flowchart of a method for constructing a large audit model provided in another embodiment of the present application;

[0086] Figure 4 A flowchart of a method for generating a fusion vector provided in another embodiment of the present application;

[0087] Figure 5 A schematic diagram of a gated recurrent unit provided in another embodiment of the present application;

[0088] Figure 6 A structural diagram of a semantic audit model provided in another embodiment of the present application;

[0089] Figure 7 A flowchart of a video comment review method provided in another embodiment of the present application;

[0090] Figure 8 A schematic diagram of a video comment review device provided by another embodiment of the present application;

[0091] Fig. 9 A schematic diagram of an electronic device for implementing a method for reviewing video barrages provided in accordance with another embodiment of the present application. DETAILED DESCRIPTION

[0092] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0093] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0094] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0095] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0096] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0097] The present application embodiment provides a method for reviewing video comments, such as Figure 1 As shown, the specific steps include:

[0098] S101, extract frames of the target video to obtain the bullet comment content and video information of each frame.

[0099] The video information includes but is not limited to video lines, video introduction, etc., which are not limited here.

[0100] It is understandable that videos usually exist in the form of video data, so it is necessary to extract frames from the video to obtain the barrage content and video information of each frame. The implementation method of frame extraction can be but is not limited to using Python programming language and OpenCV library for video frame extraction processing, which is not limited here.

[0101] S102. For each frame of barrage content and video information, the barrage content and video information of the frame are input into a semantic review model to obtain a review score for the barrage content of the frame.

[0102] Among them, the semantic review model is obtained by training the text classification model according to the first training sample set; the first training sample set includes at least one first training sample; the first training sample includes the barrage content and video text information corresponding to the first historical frame.

[0103] In the specific implementation process of this application, the semantic audit model can be called Bpool, which is a neural network double-tower model based on multiple features. In another embodiment of this application, an implementation method of constructing a semantic audit model is as follows: Figure 2 As shown, including:

[0104] S201: Construct a first training sample set.

[0105] The first training sample set includes at least one first training sample; the first training sample includes the bullet comment content and video text information corresponding to the first historical frame.

[0106] S202: For each first training sample, input the first training sample into the audit model, and output a label of the first training sample.

[0107] Among them, the review model is obtained by fine-tuning the pre-trained model according to the second training sample set; the second training sample set includes at least one second training sample; the second training sample includes training task instructions, training sample questions and training sample answers; the training sample questions include the barrage content and video information corresponding to the historical frames.

[0108] In the specific implementation process of this application, the pre-trained model can be but is not limited to GLM-9b, etc. GLM-9b is a transformer-based pre-trained large language model, which is not limited here.

[0109] Specifically, by using the audit model to annotate the first training sample, the label of the first training sample is obtained, and it is no longer necessary to manually annotate data as in the prior art, which effectively reduces labor costs. In addition, the audit model requires less training data than the semantic audit model, and the audit model has higher accuracy than the semantic audit model, so a small amount of data can be used to train the audit model first, and then the audit model can be used to generate training data for the semantic audit model, greatly reducing the cost of manual annotation.

[0110] In the specific implementation process of the present application, the first training sample may be preprocessed according to a format recognizable by the audit large model before the first training sample is input into the audit large model, or the audit large model may preprocess the first training sample after the first training sample is input into the audit large model to obtain a format recognizable by the audit large model. There is no limitation here.

[0111] Optionally, in another embodiment of the present application, an implementation method of the method for constructing a large model is reviewed, such as Figure 3 As shown, including:

[0112] S301: Construct a second training sample set.

[0113] Among them, the second training sample set includes at least one second training sample; the second training sample includes training task instructions, training sample questions and training sample answers; the training sample questions include barrage content and video text information corresponding to the second historical frame.

[0114] In the specific implementation process of the present application, the sample data will first be subjected to preprocessing steps such as regularization and data cleaning to obtain a second training sample that meets the format requirements, which is not limited here.

[0115] Specifically, the training task instructions are task instructions issued to the review model for learning, which can enable the review model to understand what the task is specifically about. For example, based on the content of the barrage and the video information, please determine whether the barrage belongs to the preset category: normal, abusive speech, dangerous speech, etc. There is no limitation here.

[0116] The second training sample that meets the format requirement may be in the format of, but is not limited to, training task instruction-training sample question-training sample answer.

[0117] Here is an example of the second training sample:

[0118] "instruct": "Please judge whether the barrage belongs to the preset categories: abusive speech, dangerous speech, based on the content of the barrage and the video information. If so, please output "no", otherwise output "yes".

[0119] “text”:”Haha, it’s really beautiful”;

[0120] “video_info”: “Video description: This video tells the story of Snow White. Video type: animation. Video lines: XXXXXXXX”;

[0121] "label":"pass".

[0122] Among them, instruct represents the training task instruction, text represents the content of the barrage, video_info represents the video information, and label represents the training sample answer.

[0123] S302: Fix the pre-training weights of the pre-training model.

[0124] Among them, the pre-trained weights of the pre-trained model are fixed, that is, only the pre-trained weights of the pre-trained model are loaded during training, and the pre-trained weights of the pre-trained model are not updated.

[0125] In the specific implementation process of this application, low-rank adaptive fine-tuning technology (LORA technology) can be used but is not limited to the pre-training weights of the fixed pre-training model, which is not limited here.

[0126] S303 , feeding all second training samples into the first low-rank matrix and the second low-rank matrix respectively for training, to obtain a trained first low-rank matrix and a trained second low-rank matrix.

[0127] First, embed the second training sample, i.e., vectorize it, and turn a single text data into a vector. The training target is the label of the data (the audit results are 0 and 1). These vectors are combined to form the final weight C. The training process is to update the parameters of the weight C. The weight C is a matrix that we want to fine-tune. Assuming it is m*n dimensional, lora fine-tuning is to perform a low-rank decomposition of the matrix m*n, splitting it into two low-rank matrices m*d and d*n, the first low-rank matrix A and the second low-rank matrix B, thereby reducing the amount of calculation and improving training efficiency.

[0128] S304: Determine a target weight according to the trained first low-rank matrix, the trained second low-rank matrix and the pre-trained weights.

[0129] Optionally, in another embodiment of the present application, an implementation of step S304 includes:

[0130] First, the trained first low-rank matrix and the trained second low-rank matrix are calculated according to matrix multiplication to obtain the trained low-rank matrix weights; then, the trained low-rank matrix weights are added to the pre-trained weights to obtain the target weights.

[0131] Among them, the pre-trained weight is a matrix. The trained first low-rank matrix and the trained second low-rank matrix are calculated according to matrix multiplication to obtain the trained low-rank matrix weight (m*n), m*n=m*d+d*n, and then the trained low-rank matrix weight is added to the pre-trained weight to obtain the target weight.

[0132] S305. Update the target weight to the audit model.

[0133] S203, respectively vectorize the bullet comment content and the video text content corresponding to the first historical frame to obtain a first vector and a second vector.

[0134] Among them, the first vector is a vector obtained by vectorizing the barrage content corresponding to the first historical frame; the second vector is a vector obtained by vectorizing the video text content corresponding to the first historical frame.

[0135] In the specific implementation process of the present application, the first vector and the second vector may be, but are not limited to, vectors of dimensions n*30*128.

[0136] S204: Generate a fusion vector of the first training sample according to the first vector and the second vector.

[0137] Optionally, in another embodiment of the present application, an implementation of step S204 is as follows: Figure 4 As shown, including:

[0138] S401, inputting a first vector into a first gated recurrent unit, outputting a first target vector, and inputting the first vector into a first highway unit, outputting a second target vector.

[0139] It should be noted that the gated recurrent unit (GRU) consists of two gates, such as Figure 5 As shown in the figure, one is the update gate and the other is the reset gate. After the msg vector (the first vector) comes in, the initial state is h(t-1). First, the gate values ​​of the update gate and the reset gate, z(t) and r(t), are calculated. Then, these two gate values ​​are calculated according to the preset calculation method. Finally, the two are added together to get the final output h(t). This final output h(t) is the first target vector.

[0140] The highway unit is a network structure that performs nonlinear transformation on a part of the input (such as a part of the first vector), directly passes over a part of the input (such as the other first part of the first vector) without any change, and then merges the two to obtain the first target vector.

[0141] S402, input the second vector into a second gated recurrent unit, output to obtain a third target vector, and input the second vector into a second highway unit, output to obtain a fourth target vector.

[0142] It should be noted that the specific implementation method of step S402 can refer to step S401 and will not be repeated here.

[0143] S403. Perform soft attention processing according to the second target vector and the fourth target vector to obtain a first attention vector and a second attention vector.

[0144] Specifically, first, the vector dot product of the second target vector and the fourth target vector is calculated to obtain the attention points of the second target vector and the fourth target vector, and then the attention points are softmax activated twice using the second target vector and the fourth target vector respectively to obtain the activated second target vector and the activated fourth target vector, and then the vector dot product of the activated second target vector and the second target vector is calculated to obtain the first attention vector, and the vector dot product of the activated fourth target vector and the fourth target vector is calculated to obtain the second attention vector.

[0145] S404: Concatenate the first target vector, the second target vector, and the first attention vector to obtain a first concatenated vector.

[0146] Specifically, the first target vector, the second target vector and the first attention vector are concatenated (concat operation, vector concatenation) to obtain a first concatenated vector.

[0147] S405: Concatenate the third target vector, the fourth target vector and the second attention vector to obtain a second concatenated vector.

[0148] It should be noted that the specific implementation method of step S405 can refer to step S404 and will not be repeated here.

[0149] S406 , input the first splicing vector into a first gated recurrent unit to obtain a fifth target vector, and input the second splicing vector into a second gated recurrent unit to obtain a sixth target vector.

[0150] Specifically, the way the first gated loop unit processes the first spliced ​​vector and the way the second gated loop unit processes the second spliced ​​vector can be referred to step S401. Both gated loop units process vectors, except that step S401 processes the first vector (which can also be understood as the initial vector), and step S406 processes the first spliced ​​vector obtained after various processing of the first vector, which will not be repeated here.

[0151] S407 . Perform a function operation on the fifth target vector and the sixth target vector to obtain a fusion vector of the first training sample.

[0152] Optionally, in another embodiment of the present application, an implementation of step S407 includes:

[0153] First, the difference between the fifth target vector and the sixth target vector is used as the seventh target vector. Then, the product of the fifth target vector and the sixth target vector is used as the eighth target vector. Finally, the fifth target vector, the seventh target vector and the eighth target vector are concatenated to obtain the fusion vector of the first training sample.

[0154] S205. Train the text classification model according to the fusion vectors of all first training samples and the labels of the first training samples to obtain a semantic review model.

[0155] Specifically, training is to update the parameters of the weights. The training target is the audit label (0~1). The parameters of the target weights are continuously updated through the training target. The loss function measures the difference between the predicted value and the true value. The smaller the loss function, the better the training. The training stops when the loss function drops to a reasonable position.

[0156] like Figure 6 The figure shows the structure diagram of the semantic audit model (Bpool). msg is the content of the bullet comment, and audio_info is the video text information. First, msg and audio_info are converted into a format that the model can process, and then vectorized to obtain the first vector and the second vector. The first vector and the second vector are sent to the two towers of Bpool, named msg tower and audio_info tower. Here, only the processing process of msg tower is described, and the processing of audio_info tower is the same as that of msg tower. The msg vector (first vector) is sent to GRU (Gated Recurrent Unit) and highway (Highway Unit) respectively, and a_encoder (first target vector) is obtained after passing through GRU; a_highway (second target vector) is obtained after passing through highway, and then soft_attention is performed on a_highway (second target vector) and b_highway (fourth target vector) to obtain a_attention (first attention vector) and b_attention (second attention vector). Next, concatenate a_encoder, a_highway, and a_attention to obtain a_merge (the first concatenated vector), and then send a_merge to GRU to obtain apool (the fifth target vector). In the same way, bpool (the sixth target vector) can be obtained, and then apool and bpool are operated by functions (diff, mul, etc.), and concatenated with apool to obtain a fusion vector. Then, the text classification model is trained based on all the fusion vectors and the labels output by the large audit model to obtain a semantic audit model.

[0157] S103. Determine the review result of the barrage content of the frame according to the review score of the barrage content of the frame.

[0158] Among them, if the review result of the barrage content of the frame shows that it is passed, the barrage content of the frame will be exposed; if the review result of the barrage content of the frame shows that it is not passed, the barrage content of the frame will be blocked.

[0159] In the specific implementation process of this application, it can be preset that if the review score is greater than 0.5, it means that the review has not passed, and if it is less than 0.5, it means that the review has passed. There is no limitation here.

[0160] Optionally, in another embodiment of the present application, if the review result of the bullet comment content of the frame indicates that it is passed, then after the bullet comment content of the frame is exposed, an implementation method of the video bullet comment review method is as follows: Figure 7 As shown, it also includes:

[0161] S701. Use the exposed barrage content as pending barrage content.

[0162] S702. At preset intervals, for each pending barrage content, the pending barrage content and the video information corresponding to the pending barrage content are input into the review model, and the inspection and review results of the pending barrage content are output.

[0163] Among them, the review model is obtained by fine-tuning the pre-trained model according to the second training sample set; the second training sample set includes at least one second training sample; the second training sample includes training task instructions, training sample questions and training sample answers; the training sample questions include the barrage content and video information corresponding to the historical frames.

[0164] S703: If the inspection and review result of the pending barrage content indicates that it has passed, the pending barrage content is marked and stored in the database.

[0165] In the specific implementation process of this application, the pending barrage content is marked and stored in the database. If the barrage content consistent with the pending barrage content is encountered later, there is no need to review it again and the barrage content can be directly exposed.

[0166] It is understandable that a time period can also be set, that is, if a bullet comment content that is consistent with the pending bullet comment content is encountered again within this time period, there is no need to review it again, and the bullet comment content is directly exposed. After this time period, the pending bullet comment content is removed from the database.

[0167] S704. If the inspection and review result of the pending barrage content indicates that it fails, the pending barrage content is recalled.

[0168] It should be noted that in the specific implementation process of this application, when the audit model is used for patrol audit, it is an offline inspection, and the barrage that fails the audit will be recalled and intercepted, thereby maximizing the accuracy and timeliness of the video barrage audit.

[0169] It can be seen from the above scheme that the present application provides a video barrage review method, which extracts frames from the target video, obtains the barrage content and video information of each frame, and then inputs the barrage content and video information of each frame into the semantic review model to obtain the review score of the barrage content of the frame, thereby determining the review result of the barrage content of the frame according to the review score of the barrage content of the frame; wherein, if the review result of the barrage content of the frame shows that it is passed, the barrage content of the frame is exposed; if the review result of the barrage content of the frame shows that it is not passed, the barrage content of the frame is intercepted. The purpose of fast, efficient and accurate automatic review of the barrage content is achieved, which greatly reduces the burden of review, saves review costs, and improves review efficiency.

[0170] The present application embodiment provides a video comment review device, such as Figure 8 As shown, specifically including:

[0171] The frame extraction unit 801 is used to extract frames of the target video to obtain the bullet comment content and video information of each frame.

[0172] The first input unit 802 is used to input the barrage content and video information of each frame into the semantic review model to obtain the review score of the barrage content of the frame.

[0173] Among them, the semantic review model is obtained by training the text classification model according to the first training sample set; the first training sample set includes at least one first training sample; the first training sample includes the barrage content and video text information corresponding to the first historical frame.

[0174] The first determination unit 803 is used to determine the review result of the barrage content of the frame according to the review score of the barrage content of the frame.

[0175] Among them, if the review result of the barrage content of the frame shows that it is passed, the barrage content of the frame will be exposed; if the review result of the barrage content of the frame shows that it is not passed, the barrage content of the frame will be blocked.

[0176] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 1 As shown, no further description is given here.

[0177] Optionally, in another embodiment of the present application, an implementation of the video comment review device further includes:

[0178] The second determining unit is used to use the exposed barrage content as the pending barrage content.

[0179] The second input unit is used to input the pending barrage content and the video information corresponding to the pending barrage content into the review model at preset intervals for each pending barrage content, and output the inspection and review results of the pending barrage content.

[0180] Among them, the review model is obtained by fine-tuning the pre-trained model according to the second training sample set; the second training sample set includes at least one second training sample; the second training sample includes training task instructions, training sample questions and training sample answers; the training sample questions include the barrage content and video information corresponding to the historical frames.

[0181] The storage unit is used to mark the pending barrage content and store it in the database if the inspection and review result of the pending barrage content shows that it has passed.

[0182] The recall unit is used to recall the pending barrage content if the inspection and review result of the pending barrage content shows that it fails.

[0183] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 7 As shown, no further description is given here.

[0184] Optionally, in another embodiment of the present application, an implementation of reviewing a construction unit of a large model includes:

[0185] The first construction unit is used to construct a second training sample set.

[0186] Among them, the second training sample set includes at least one second training sample; the second training sample includes training task instructions, training sample questions and training sample answers; the training sample questions include barrage content and video text information corresponding to the second historical frame.

[0187] Fixed unit, used to fix the pre-trained weights of the pre-trained model.

[0188] The feeding unit is used to feed all the second training samples into the first low-rank matrix and the second low-rank matrix respectively for training to obtain the trained first low-rank matrix and the trained second low-rank matrix.

[0189] The third determination unit is used to determine the target weight according to the trained first low-rank matrix, the trained second low-rank matrix and the pre-trained weight.

[0190] An updating unit is used to update the target weight to the audit model.

[0191] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 3 As shown, no further description is given here.

[0192] Optionally, in another embodiment of the present application, an implementation of the third determining unit includes:

[0193] The first calculation unit is used to calculate the trained first low-rank matrix and the trained second low-rank matrix according to matrix multiplication to obtain the trained low-rank matrix weights.

[0194] The second computing unit is used to add the trained low-rank matrix weights to the pre-trained weights to obtain the target weights.

[0195] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0196] Optionally, in another embodiment of the present application, an implementation of a construction unit of a semantic audit model includes:

[0197] The second construction unit is used to construct a first training sample set.

[0198] The first training sample set includes at least one first training sample; the first training sample includes the bullet comment content and video text information corresponding to the first historical frame.

[0199] The third input unit is used to input the first training sample into the audit model for each first training sample, and output the label of the first training sample.

[0200] The vectorization unit uses language to vectorize the bullet comment content and the video text content corresponding to the first historical frame to obtain a first vector and a second vector.

[0201] The fusion unit is used to generate a fusion vector of the first training sample according to the first vector and the second vector.

[0202] The training unit is used to train the text classification model according to the fusion vectors of all the first training samples and the labels of the first training samples to obtain a semantic review model.

[0203] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 2 As shown, no further description is given here.

[0204] Optionally, in another embodiment of the present application, an implementation of the fusion unit includes:

[0205] The fourth input unit is used to input the first vector into the first gated loop unit to output a first target vector, and to input the first vector into the first highway unit to output a second target vector.

[0206] The fifth input unit is used to input the second vector into the second gated recurrent unit to output a third target vector, and to input the second vector into the second highway unit to output a fourth target vector.

[0207] The soft attention processing unit is used to perform soft attention processing according to the second target vector and the fourth target vector to obtain the first attention vector and the second attention vector.

[0208] The first splicing unit is used to splice the first target vector, the second target vector and the first attention vector to obtain a first splicing vector.

[0209] The second splicing unit is used to splice the third target vector, the fourth target vector and the second attention vector to obtain a second splicing vector.

[0210] The sixth input unit is used to input the first splicing vector into the first gated recurrent unit to obtain a fifth target vector, and to input the second splicing vector into the second gated recurrent unit to obtain a sixth target vector.

[0211] The function operation unit is used to perform a function operation according to the fifth target vector and the sixth target vector to obtain a fusion vector of the first training sample.

[0212] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 4 As shown, no further description is given here.

[0213] Optionally, in another embodiment of the present application, an implementation of the function operation unit includes:

[0214] The third calculation unit is used to use the difference between the fifth target vector and the sixth target vector as the seventh target vector.

[0215] The fourth calculation unit is used to take the product of the fifth target vector and the sixth target vector as an eighth target vector.

[0216] The third concatenation unit is used to concatenate the fifth target vector, the seventh target vector and the eighth target vector to obtain a fusion vector of the first training sample.

[0217] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be repeated here.

[0218] It can be seen from the above scheme that the present application provides a video barrage review device, which extracts frames of the target video through the frame extraction unit 801, and obtains the barrage content and video information of each frame. For the barrage content and video information of each frame, the first input unit 802 inputs the barrage content and video information of the frame into the semantic review model to obtain the review score of the barrage content of the frame, and the first determination unit 803 determines the review result of the barrage content of the frame according to the review score of the barrage content of the frame; wherein, if the review result of the barrage content of the frame indicates that it is passed, the barrage content of the frame is exposed; if the review result of the barrage content of the frame indicates that it is not passed, the barrage content of the frame is intercepted. The purpose of fast, efficient and accurate automatic review of the barrage content is achieved, which greatly reduces the burden of review, saves review costs, and improves review efficiency.

[0219] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0220] Another embodiment of the present application provides an electronic device, such as Fig. 9 As shown, including:

[0221] One or more processors 901.

[0222] The storage device 902 stores one or more programs.

[0223] When the one or more programs are executed by the one or more processors 901, the one or more processors 901 implement the video barrage review method as described in any one of the above embodiments.

[0224] Another embodiment of the present application provides a computer storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the video barrage review method as described in any one of the above embodiments is implemented.

[0225] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0226] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0227] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0228] Another embodiment of the present application provides a computer program product, which, when executed, is used to execute the above-mentioned video barrage review method.

[0229] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment of the present application are executed.

[0230] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in this application is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing this application.

[0231] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.

[0232] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above application concept. For example, the above features are replaced with (but not limited to) technical features with similar functions applied in the present application.

Claims

1. A video comment review method, characterized in that: include: Extract frames from the target video to obtain the bullet comment content and video information of each frame; For each frame of barrage content and video information, the barrage content and video information of the frame are input into a semantic review model to obtain a review score of the barrage content of the frame; wherein the semantic review model is obtained by training a text classification model according to a first training sample set; the first training sample set includes at least one first training sample; the first training sample includes the barrage content and video text information corresponding to the first historical frame; Determine the review result of the barrage content of the frame according to the review score of the barrage content of the frame; if the review result of the barrage content of the frame indicates that it is passed, then expose the barrage content of the frame; if the review result of the barrage content of the frame indicates that it is not passed, then intercept the barrage content of the frame; The method for constructing the semantic audit model includes: Constructing a first training sample set; wherein the first training sample set includes at least one first training sample; the first training sample includes the bullet comment content and video text information corresponding to the first historical frame; For each first training sample, the first training sample is input into the review model, and the label of the first training sample is output, wherein the review model is obtained by fine-tuning the pre-trained model according to the second training sample set, and the second training sample set includes at least one second training sample, and the second training sample includes a training task instruction, a training sample question, and a training sample answer; the training sample question includes the barrage content and video information corresponding to the historical frame; Vectorizing the bullet comment content and the video text content corresponding to the first historical frame respectively to obtain a first vector and a second vector; Generate a fusion vector of the first training sample according to the first vector and the second vector; Training a text classification model according to the fusion vectors of all the first training samples and the labels of the first training samples to obtain a semantic review model; The step of generating a fusion vector of the first training sample according to the first vector and the second vector includes: Inputting the first vector into a first gated recurrent unit, outputting a first target vector, and inputting the first vector into a first highway unit, outputting a second target vector; Inputting the second vector into a second gated recurrent unit, outputting a third target vector, and inputting the second vector into a second highway unit, outputting a fourth target vector; Performing soft attention processing according to the second target vector and the fourth target vector to obtain a first attention vector and a second attention vector; Concatenate the first target vector, the second target vector, and the first attention vector to obtain a first concatenated vector; Concatenate the third target vector, the fourth target vector, and the second attention vector to obtain a second concatenated vector; Inputting the first splicing vector into the first gated recurrent unit to obtain a fifth target vector, and inputting the second splicing vector into the second gated recurrent unit to obtain a sixth target vector; A function operation is performed according to the fifth target vector and the sixth target vector to obtain a fusion vector of the first training sample.

2. The video commentary review method according to claim 1, characterized in that: If the review result of the bullet-screen content of the frame indicates that it is passed, after revealing the bullet-screen content of the frame, the method further includes: The exposed barrage content is used as pending barrage content; At preset intervals, for each pending barrage content, the pending barrage content and the video information corresponding to the pending barrage content are input into the review model, and the inspection and review result of the pending barrage content is output; if the inspection and review result of the pending barrage content indicates that it is passed, the pending barrage content is marked and stored in the database; If the inspection and review result of the pending barrage content indicates that it fails, the pending barrage content will be recalled.

3. The video commentary review method according to claim 2, characterized in that: The method for constructing the audit model includes: Constructing a second training sample set; wherein the second training sample set includes at least one second training sample; the second training sample includes a training task instruction, a training sample question and a training sample answer; the training sample question includes the bullet comment content and video text information corresponding to the second historical frame; Fix the pre-trained weights of the pre-trained model; Feeding all the second training samples into the first low-rank matrix and the second low-rank matrix for training respectively, to obtain a trained first low-rank matrix and a trained second low-rank matrix; Determining a target weight according to the trained first low-rank matrix, the trained second low-rank matrix and the pre-trained weight; The target weights are updated to the audit model.

4. The video commentary review method according to claim 3 is characterized in that: The determining the target weight according to the trained first low-rank matrix, the trained second low-rank matrix and the pre-trained weight comprises: Calculate the trained first low-rank matrix and the trained second low-rank matrix according to matrix multiplication to obtain trained low-rank matrix weights; The trained low-rank matrix weights are added to the pre-trained weights to obtain target weights.

5. The video comment review method according to claim 1, characterized in that: The performing a function operation according to the fifth target vector and the sixth target vector to obtain a fusion vector of the first training sample includes: taking the difference between the fifth target vector and the sixth target vector as a seventh target vector; taking the product of the fifth target vector and the sixth target vector as an eighth target vector; The fifth target vector, the seventh target vector and the eighth target vector are concatenated to obtain a fusion vector of the first training sample.

6. A video comment review device, characterized in that: include: A frame extraction unit is used to extract frames of the target video to obtain the bullet comment content and video information of each frame; A first input unit is used to input the barrage content and video information of each frame into a semantic review model to obtain a review score of the barrage content of the frame; wherein the semantic review model is obtained by training a text classification model according to a first training sample set; the first training sample set includes at least one first training sample; the first training sample includes the barrage content and video text information corresponding to the first historical frame; A first determining unit is used to determine the review result of the barrage content of the frame according to the review score of the barrage content of the frame; if the review result of the barrage content of the frame indicates that it is passed, the barrage content of the frame is exposed; if the review result of the barrage content of the frame indicates that it is not passed, the barrage content of the frame is blocked; The method for constructing the semantic audit model includes: Constructing a first training sample set; wherein the first training sample set includes at least one first training sample; the first training sample includes the bullet comment content and video text information corresponding to the first historical frame; For each first training sample, the first training sample is input into the review model, and the label of the first training sample is output, wherein the review model is obtained by fine-tuning the pre-trained model according to the second training sample set, and the second training sample set includes at least one second training sample, and the second training sample includes a training task instruction, a training sample question, and a training sample answer; the training sample question includes the barrage content and video information corresponding to the historical frame; Vectorizing the bullet comment content and the video text content corresponding to the first historical frame respectively to obtain a first vector and a second vector; Generate a fusion vector of the first training sample according to the first vector and the second vector; Training a text classification model according to the fusion vectors of all the first training samples and the labels of the first training samples to obtain a semantic review model; The step of generating a fusion vector of the first training sample according to the first vector and the second vector includes: Inputting the first vector into a first gated recurrent unit, outputting a first target vector, and inputting the first vector into a first highway unit, outputting a second target vector; Inputting the second vector into a second gated recurrent unit, outputting a third target vector, and inputting the second vector into a second highway unit, outputting a fourth target vector; Performing soft attention processing according to the second target vector and the fourth target vector to obtain a first attention vector and a second attention vector; Concatenate the first target vector, the second target vector, and the first attention vector to obtain a first concatenated vector; Concatenate the third target vector, the fourth target vector, and the second attention vector to obtain a second concatenated vector; Inputting the first splicing vector into the first gated recurrent unit to obtain a fifth target vector, and inputting the second splicing vector into the second gated recurrent unit to obtain a sixth target vector; A function operation is performed according to the fifth target vector and the sixth target vector to obtain a fusion vector of the first training sample.

7. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the video barrage review method as described in any one of claims 1 to 5.

8. A computer storage medium, characterized in that A computer program is stored thereon, wherein when the computer program is executed by a processor, the method for reviewing video barrage as described in any one of claims 1 to 5 is implemented.

Images