A data processing method, apparatus, device, and storage medium

Abstract

This application discloses a data processing method, apparatus, device, and storage medium in the field of artificial intelligence. The method includes: acquiring the target text of the target bullet comment and the corresponding target video segment for that bullet comment; performing feature extraction processing on the target text using a text processing module in the bullet comment recognition model to obtain a target text feature vector; performing feature extraction processing on the target video segment using a video processing module in the bullet comment recognition model to obtain a target video feature vector; and determining the recognition result corresponding to the target bullet comment based on the target text feature vector and the target video feature vector using a fusion feature processing module in the bullet comment recognition model. This recognition result is used to characterize the type of the target bullet comment. This method can accurately identify the type of bullet comment.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a data processing method, apparatus, device and storage medium. Background Technology

[0002] Many online video streaming platforms now feature a bullet screen (danmu) function, allowing users to edit and send comments about the currently playing video. Viewers of the video can then see these comments. However, in practice, the quality of user-sent bullet screen comments varies greatly, often including inappropriate, boring, or offensive content. These are known in the industry as "inappropriate bullet screen comments." To improve the viewing experience, some online video streaming platforms identify and filter inappropriate bullet screen comments, preventing them from being displayed to other users.

[0003] In related technologies, it is common to identify whether a bullet comment is inappropriate based solely on its content. For example, the words in the bullet comment are matched with words in a pre-built illegal word list. If the bullet comment contains words that successfully match words in the illegal word list, then the bullet comment can be determined to be inappropriate. Another example is to use a text classification model learned through machine learning algorithms to identify whether a bullet comment is inappropriate.

[0004] The above-mentioned bullet screen recognition methods only focus on the single dimension of information in the bullet screen text when identifying whether a bullet screen is inappropriate. This often makes it difficult to accurately identify inappropriate bullet screens and may result in situations such as missing inappropriate bullet screens or mistakenly identifying normal bullet screens as inappropriate bullet screens. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, and storage medium that can accurately identify whether a bullet screen is an inappropriate bullet screen, avoiding situations such as missing the identification of inappropriate bullet screens or misidentifying normal bullet screens as inappropriate bullet screens.

[0006] In view of the above, the first aspect of this application provides a data processing method, the method comprising:

[0007] For the target bullet screen to be identified, obtain the target text of the target bullet screen and the target video segment corresponding to the target bullet screen;

[0008] The text processing module in the bullet screen recognition model performs feature extraction processing on the target text to obtain the target text feature vector;

[0009] The video processing module in the bullet screen recognition model performs feature extraction processing on the target video segment to obtain a reference video feature vector, performs feature extraction processing on the video image captured from the target video segment to obtain a reference image feature vector, and generates a target video feature vector based on the reference video feature vector and the reference image feature vector.

[0010] The fusion feature processing module in the bullet screen recognition model determines the recognition result corresponding to the target bullet screen based on the target text feature vector and the target video feature vector; the recognition result corresponding to the target bullet screen is used to characterize the type of the target bullet screen.

[0011] A second aspect of this application provides a data processing apparatus, the apparatus comprising:

[0012] The data acquisition unit is used to acquire the target text of the target bullet screen and the target video segment corresponding to the target bullet screen for the target bullet screen to be identified;

[0013] The text feature extraction unit is used to perform feature extraction processing on the target text through the text processing module in the bullet screen recognition model to obtain the target text feature vector;

[0014] The video feature extraction unit is used to perform feature extraction processing on the target video segment to obtain a reference video feature vector through the video processing module in the bullet screen recognition model, perform feature extraction processing on the video image captured from the target video segment to obtain a reference image feature vector, and generate a target video feature vector based on the reference video feature vector and the reference image feature vector.

[0015] The bullet screen recognition unit is used to determine the recognition result corresponding to the target bullet screen based on the target text feature vector and the target video feature vector through the fusion feature processing module in the bullet screen recognition model; the recognition result corresponding to the target bullet screen is used to characterize the type of the target bullet screen.

[0016] A third aspect of this application provides an apparatus comprising a processor and a memory:

[0017] The memory is used to store computer programs;

[0018] The processor is configured to perform the steps of the data processing method as described in the first aspect above, according to the computer program.

[0019] A fourth aspect of this application provides a computer-readable storage medium for storing a computer program for performing the steps of the data processing method described in the first aspect.

[0020] A fifth aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the data processing method described in the first aspect.

[0021] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0022] In the data processing method provided in this application embodiment, for the target bullet screen to be identified, the target text of the target bullet screen and the target video segment corresponding to the target bullet screen are first obtained; then, the text processing module in the bullet screen recognition model performs feature extraction processing on the target text to obtain a target text feature vector; the video processing module in the bullet screen recognition model performs feature extraction processing on the target video segment to obtain a reference video feature vector, performs feature extraction processing on the video image captured from the target video segment to obtain a reference image feature vector, and generates a target video feature vector based on the reference video feature vector and the reference image feature vector; furthermore, the fusion feature processing module in the bullet screen recognition model determines the recognition result corresponding to the target bullet screen based on the target text feature vector and the target video feature vector, and the recognition result corresponding to the target bullet screen is used to characterize the type of the target bullet screen, for example, to characterize whether the target bullet screen belongs to an inappropriate bullet screen. Compared to related technologies that identify whether a bullet comment is inappropriate solely based on the text of the bullet comment, the data processing method provided in this application comprehensively considers both the textual information of the target bullet comment itself and the visual information of the target video segment corresponding to the target bullet comment when identifying whether a target bullet comment is inappropriate. It utilizes the visual information of the target video segment to assist in identifying whether a target bullet comment is inappropriate. This improves the accuracy of identifying inappropriate bullet comments, avoids missed or misidentified inappropriate bullet comments, and thus helps to effectively filter inappropriate bullet comments, prevents the display of inappropriate bullet comments to video viewers, and improves the user's video viewing experience. Attached Figure Description

[0023] Figure 1 This is a schematic diagram illustrating an application scenario of the data processing method provided in the embodiments of this application;

[0024] Figure 2 A flowchart illustrating the data processing method provided in an embodiment of this application;

[0025] Figure 3 A schematic diagram illustrating the working principle of an exemplary text processing module provided in this application embodiment;

[0026] Figure 4 A schematic diagram illustrating the working principle of an exemplary video processing module provided in the embodiments of this application;

[0027] Figure 5 A schematic diagram of the structure of an exemplary bullet screen recognition model provided in the embodiments of this application;

[0028] Figure 6 A flowchart illustrating the training method for the bullet screen recognition model provided in this application embodiment;

[0029] Figure 7 A flowchart illustrating the initial bullet screen recognition model pre-training method provided in this application embodiment;

[0030] Figure 8 A schematic diagram illustrating the working principle of an exemplary matching relationship recognition model provided in this application embodiment;

[0031] Figure 9 A flowchart illustrating the pre-training method for the initial text processing module provided in this application embodiment;

[0032] Figure 10 A schematic diagram illustrating the working principle of an exemplary text recognition model provided in this application embodiment;

[0033] Figure 11 A flowchart illustrating the pre-training method for the initial video processing module provided in this application embodiment;

[0034] Figure 12 A schematic diagram illustrating the working principle of an exemplary video image reconstruction model provided in this application embodiment;

[0035] Figure 13 A schematic diagram illustrating the training process of the bullet screen recognition model provided in this application embodiment;

[0036] Figure 14 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0037] Figure 15 This is a schematic diagram of another data processing apparatus provided in an embodiment of this application;

[0038] Figure 16 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application;

[0039] Figure 17 This is a schematic diagram of the server structure provided in an embodiment of this application. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0041] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0042] Computer vision (CV) is a science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0043] Natural Language Processing (NLP) is an important field within computer science and artificial intelligence. It studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field involves natural language—the language people use in daily life—and thus it has a close relationship with linguistic research. NLP techniques typically include text processing, semantic understanding, machine translation, question answering, and knowledge graphs.

[0044] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0045] The solutions provided in this application relate to computer vision technology, natural language processing technology, and machine learning technology in the field of artificial intelligence, and are specifically illustrated through the following embodiments:

[0046] This application provides a data processing method that can improve the accuracy of bullet screen recognition by comprehensively considering the text information of the bullet screen and the visual information of the video segment corresponding to the bullet screen when identifying the type of bullet screen (such as whether the bullet screen is a bad bullet screen).

[0047] Specifically, in the data processing method provided in this application embodiment, for the target bullet screen to be identified, the target text of the target bullet screen and the target video segment corresponding to the target bullet screen are first obtained; then, the text processing module in the bullet screen recognition model performs feature extraction processing on the target text to obtain a target text feature vector; the video processing module in the bullet screen recognition model performs feature extraction processing on the target video segment to obtain a reference video feature vector, performs feature extraction processing on the video image captured from the target video segment to obtain a reference image feature vector, and generates a target video feature vector based on the reference video feature vector and the reference image feature vector; furthermore, the fusion feature processing module in the bullet screen recognition model determines the recognition result corresponding to the target bullet screen based on the target text feature vector and the target video feature vector, and the recognition result corresponding to the target bullet screen is used to characterize the type of the target bullet screen, such as characterizing whether the target bullet screen belongs to inappropriate bullet screens.

[0048] Compared to related technologies that identify whether a bullet comment is inappropriate from a single dimension, the data processing method provided in this application takes into account the high reference value of the video information corresponding to the bullet comment when determining whether it is inappropriate. Based on this, when identifying whether a target bullet comment is inappropriate using a bullet comment recognition model, it comprehensively considers both the text information of the target bullet comment itself and the visual information of the target video segment corresponding to the target bullet comment. The visual information of the target video segment is used to assist in identifying whether the target bullet comment is inappropriate. This improves the accuracy of inappropriate bullet comment recognition, avoids missed or misidentified inappropriate bullet comments, and thus helps to effectively filter inappropriate bullet comments, preventing them from being displayed to video viewers and improving the user's video viewing experience.

[0049] It should be understood that the data processing method provided in this application embodiment can be applied to devices with text processing and image processing capabilities, such as terminal devices or servers. Specifically, the terminal device can be a smartphone, computer, tablet computer, personal digital assistant (PDA), vehicle terminal, etc.; the server can specifically be an application server or a web server. In actual deployment, it can be a standalone server, a cluster server, or a cloud server.

[0050] To facilitate understanding of the data processing method provided in the embodiments of this application, the following example uses a server as the execution subject of the data processing method to illustrate the application scenarios of the data processing method.

[0051] See Figure 1 , Figure 1 This is a schematic diagram illustrating an application scenario of the data processing method provided in the embodiments of this application. For example... Figure 1 As shown, this application scenario includes a terminal device 110 and a server 120, which can communicate with each other via a network. The terminal device 110 runs a video playback application (APP), which allows users to send bullet comments while watching videos. The server 120 serves as the backend server for the video playback APP, and is used to execute the data processing method provided in this embodiment.

[0052] In practical applications, when a user watches a target video through a video playback app running on terminal device 110, they can use the bullet screen sending function provided by the video playback app to edit and send comments on the target video, i.e., bullet screens. For example, the video playback interface of the video playback app includes a bullet screen editing bar and a bullet screen sending control. The user can enter target text in the bullet screen editing bar, and after confirming the completion of the target text input, click the bullet screen sending control to trigger the bullet screen sending operation. After detecting that the user has triggered the bullet screen sending operation, terminal device 110 will generate a target bullet screen based on the target text entered by the user in the bullet screen editing bar, and then send the target bullet screen to server 120 via the network.

[0053] After receiving the target comment, server 120 will obtain the target text within the comment and the corresponding target video segment. The target video segment corresponding to the target comment is a segment of the target video played by the video playback app at the time the comment was created or sent.

[0054] Furthermore, server 120 can invoke the pre-trained bullet screen recognition model 121 to identify the type of the target bullet screen based on the target text in the target bullet screen and the target video segment corresponding to the target bullet screen, such as identifying whether the target bullet screen belongs to inappropriate bullet screens. Specifically, server 120 can use the text processing module 1211 in the bullet screen recognition model 121 to perform feature extraction processing on the target text to obtain a target text feature vector; use the video processing module 1212 in the bullet screen recognition model 121 to perform feature extraction processing on the target video segment to obtain a reference video feature vector; perform feature extraction processing on the video image captured from the target video segment to obtain a reference image feature vector; and generate a target video feature vector based on the reference video feature vector and the reference image feature vector; furthermore, use the fusion feature processing module 1213 in the bullet screen recognition model 121 to determine the recognition result corresponding to the target bullet screen based on the target text feature vector and the target video feature vector. This recognition result can characterize the type of the target bullet screen, such as whether the target bullet screen belongs to inappropriate bullet screens.

[0055] If server 120 determines through the above operations that the target bullet comment is an inappropriate bullet comment, it will filter the bullet comment, discarding it and preventing it from being displayed to other viewers of the target video. Conversely, if server 120 determines through the above operations that the target bullet comment is not inappropriate, it can be displayed normally, allowing other viewers of the target video to see the bullet comment correctly.

[0056] It should be understood that Figure 1The application scenarios shown are merely examples. In practical applications, the data processing method provided in this application embodiment can also be applied to other scenarios. For example, the terminal device 110 can independently execute the data processing method to identify whether the bullet comments to be sent by the user are inappropriate bullet comments, and directly intercept and send bullet comments identified as inappropriate bullet comments. No limitations are made here regarding the application scenarios of the data processing method provided in this application embodiment.

[0057] The data processing method provided in this application will be described in detail below through method embodiments.

[0058] See Figure 2 , Figure 2 This is a flowchart illustrating the data processing method provided in an embodiment of this application. For ease of description, the following embodiments will still use a server as the execution subject of this data processing method. Figure 2 As shown, the data processing method includes the following steps:

[0059] Step 201: For the target bullet screen to be identified, obtain the target text of the target bullet screen and the target video segment corresponding to the target bullet screen.

[0060] In practical applications, when users watch videos through video playback applications or websites, they can use the bullet comment sending function provided by the application or website to trigger bullet comment editing and sending operations. In response to the user-triggered bullet comment sending operation, the terminal device supporting the video playback application or connected to the video playback website will send the bullet comment edited by the user through the bullet comment editing operation to the backend server of the application or website. It should be understood that the bullet comment sent by the terminal device to the server includes the bullet comment text edited by the user through the bullet comment editing operation; optionally, the bullet comment may also include the bullet comment style selected by the user through the bullet comment editing operation (such as bullet comment text font, bullet comment text color, bullet comment decorative frame, etc.).

[0061] After receiving a bullet comment from a terminal device, the server treats the bullet comment as a target bullet comment to be identified and extracts the target text from it. The target text here is the bullet comment text edited and sent by the user through the bullet comment sending function. For example, assuming the user edits and sends the target bullet comment "The plot is so exciting" through the bullet comment sending function provided by the video playback application, the target text extracted by the server from the target bullet comment is "The plot is so exciting".

[0062] In addition, the server also needs to determine the target video segment corresponding to the target bullet comment. This target video segment can be a video segment played at the creation time and / or sending time of the target bullet comment. For example, assuming a user edits and sends a target bullet comment while watching a target video through a video playback application, the server can determine the creation time of the target bullet comment and then determine the video segment played by the user's video playback application at that creation time as the target video segment corresponding to the target bullet comment; alternatively, the server can also determine the sending time of the target bullet comment and then determine the video segment played by the user's video playback application at that sending time as the target video segment corresponding to the target bullet comment; or, the server can simultaneously determine both the creation time and the sending time of the target bullet comment and then consider both the video segment played by the user's video playback application at the creation time and the video segment played at the sending time as the target video segments corresponding to the target bullet comment.

[0063] Because the time interval between the creation and sending of a bullet comment is usually short, in many cases, the video segment played at the creation time and the video segment played at the sending time may be the same video segment, or the two video segments may have a high degree of overlap. Whether to determine the target video segment based on the creation time, the sending time, or both, depends on the specific application requirements.

[0064] This application provides two exemplary implementation methods for determining the target video segment corresponding to the target bullet screen. These two implementation methods are described below.

[0065] The first implementation method divides the target video played by the user who posted the target bullet comment into at least one reference video segment. Then, it determines the reference video segment played at the reference time point corresponding to the target bullet comment as the target video segment corresponding to that target bullet comment; the reference time point corresponding to the target bullet comment may include at least one of the following: the creation time point of the target bullet comment, and the sending time point of the target bullet comment.

[0066] Specifically, the servers of video playback applications or websites can pre-divide the video resources they can provide into segments. For example, the server can divide the video resources into several reference video segments according to specific time units (such as 5 seconds, 1 minute, etc.); or, the server can divide the video resources into several reference video segments according to the plot of the video resources. In this way, all the video resources that the server can provide include several reference video segments.

[0067] After the server obtains the target bullet comment sent by the terminal device, it can determine the reference time point corresponding to the target bullet comment (i.e., the creation time point and / or sending time point of the target bullet comment), as well as the target video currently being played by the video playback application or video playback website running on the terminal device; then, it determines the reference video segment in the target video played by the video playback application or video playback website at the reference time point as the target video segment corresponding to the target bullet comment.

[0068] For example, assuming the reference time point corresponding to the target bullet comment is the creation time point of the target bullet comment, and the target video watched by the user who posted the target bullet comment through the video playback application has been pre-divided into several reference video segments according to the preset time unit of 5 seconds; if the creation time point of the target bullet comment is 10:24:15, and the video playback application plays the content of 3:17 in the target video at the creation time point, then the video segment to which the content of 3:17 in the target video belongs, 3:15-3:20, can be determined as the target video segment corresponding to the target bullet comment.

[0069] The second implementation method is to extract a video segment from the target video according to the preset video segment extraction rules, based on the reference time point corresponding to the target bullet comment and the preset video segment extraction duration, and use it as the target video segment corresponding to the target bullet comment; the reference time point corresponding to the target bullet comment may include at least one of the following: the creation time point of the target bullet comment, the sending time point of the target bullet comment.

[0070] Specifically, the servers of video playback applications or video playback websites can pre-define video segment extraction rules and video segment extraction durations. For example, they can extract a video segment 5 seconds before the creation time of a bullet comment as the corresponding video segment for the bullet comment, or extract a video segment 5 seconds before the sending time of a bullet comment as the corresponding video segment for the bullet comment, or extract a video segment between 5 seconds before the creation time of a bullet comment and 5 seconds after the sending time of a bullet comment as the corresponding video segment for the bullet comment, and so on.

[0071] Once the server receives the target bullet comment sent by the terminal device, it can extract the target video segment corresponding to the target bullet comment from the target video currently being watched by the user through the video playback application or video playback website, based on the creation time and / or publication time of the target bullet comment and in accordance with the video segment extraction rules and video segment extraction duration defined by the server.

[0072] For example, suppose the server's predefined video segment extraction rule is to extract the video segment between 5 seconds before the creation time of the bullet comment and 5 seconds after the sending time of the bullet comment, as the video segment corresponding to the bullet comment; if the creation time of the target bullet comment is 10:24:15 and the sending time is 10:24:25, and the content played at the creation time of the target bullet comment is 3:17 of the target video, and the content played at the sending time of the target bullet comment is 3:27 of the target video, then the video segment between 3:12 and 3:32 of the target video should be the target video segment corresponding to the target bullet comment.

[0073] It should be understood that the above video clip extraction rules and video clip extraction duration are only examples. In practical applications, other video clip extraction rules and video clip extraction durations can be set according to actual needs. This application does not impose any limitations on these video clip extraction rules and video clip extraction durations.

[0074] It should be noted that in practical applications, in addition to the server performing the above-mentioned operation of determining the target video segment, the terminal device that supports the operation of video playback application or the terminal device that has established a communication link with the video playback website can also perform the above-mentioned operation of determining the target video segment. That is, the terminal device can determine the time of the target video segment in the target video through either of the above two implementation methods, and then send the determined time of the target video segment in the target video to the server so that the server can obtain the target video segment from the target video based on the time.

[0075] Of course, in practical applications, other methods can also be used to determine the target video segment corresponding to the target bullet comment. For example, the entire target video that the user is watching when sending the target bullet comment can also be used as the target video segment. This application does not impose any restrictions on the method of determining the target video segment.

[0076] Step 202: The target text is processed by the text processing module in the bullet screen recognition model to obtain the target text feature vector.

[0077] Once the server obtains the target text of the target comment, it can call a pre-trained comment recognition model. The text processing module within this model then performs feature extraction on the target text. Specifically, the text processing module extracts features from the target text to obtain the corresponding target text feature vector.

[0078] As an example, the text processing module in the bullet screen recognition model performs feature extraction on the target text to obtain the target text feature vector, which may specifically include:

[0079] The word feature extraction structure in the text processing module is used to extract features from each word segment in the target text to obtain the word feature vector corresponding to each word segment. Then, the word feature fusion structure in the text processing module is used to determine the target text feature vector from the word feature vector corresponding to each word segment.

[0080] The following is combined with Figure 3 The illustrated diagram shows the working principle of an exemplary text processing module, illustrating the working principle of the aforementioned text processing module. For example... Figure 3 As shown, the text processing module includes a word feature extraction structure 301 and a word feature fusion structure 302. The word feature extraction structure 301 includes a recurrent neural network (RNN) layer and a fully connected layer, while the word feature fusion structure 302 includes a fully connected layer, a convolutional layer, and a pooling layer.

[0081] In practice, the RNN layer in the word feature extraction structure 301 performs preliminary feature extraction on each word in the input target text to obtain the primary word feature vector corresponding to each word; then, the fully connected layer in the word feature extraction structure 301 processes the primary word feature vector corresponding to each word again to obtain the word feature vector corresponding to each word.

[0082] It should be noted that the RNN layer in the word feature extraction structure 301 can also be replaced by a gated recurrent unit (GRU) or a long short-term memory network (LSTM). The specific structure can be set according to actual needs. This application does not impose any restrictions on the specific structure of the word feature extraction structure 301.

[0083] After word feature extraction structure 301 extracts the word feature vectors corresponding to each word in the target text, word feature fusion structure 302 further determines the target text feature vector based on the word feature vectors corresponding to each word. Specifically, the fully connected layer in word feature fusion structure 302 performs preliminary fusion processing on the word feature vectors corresponding to each word to obtain a preliminary fused feature vector. Then, the convolutional layer and pooling layer in word feature fusion structure 302 sequentially perform convolution and pooling processing on the preliminary fused feature vector to finally obtain the target text feature vector.

[0084] It should be understood that Figure 3 The word feature fusion structure 302 shown is only an example. In practical applications, other structural forms of word feature fusion structure 302 can be designed according to actual application needs. This application does not impose any restrictions on the specific structure of the word feature fusion structure 302.

[0085] Step 203: Through the video processing module in the bullet screen recognition model, feature extraction processing is performed on the target video segment to obtain a reference video feature vector, feature extraction processing is performed on the video image collected from the target video segment to obtain a reference image feature vector, and a target video feature vector is generated based on the reference video feature vector and the reference image feature vector.

[0086] Once the server obtains the target video segment corresponding to the target comment, it can invoke a pre-trained comment recognition model. The video processing module within this model then performs feature extraction on the target video segment. Specifically, the video processing module extracts features from the target video segment to obtain a reference video feature vector, extracts features from the video image captured from the target video segment to obtain a reference image feature vector, and finally generates the target video feature vector based on both the reference video feature vector and the reference image feature vector.

[0087] As an example, the video processing module in the bullet screen recognition model generates a target video feature vector based on the target video segment, which may specifically include:

[0088] The video feature extraction structure in the video processing module performs feature extraction on the target video segment to obtain a reference video feature vector. The image feature extraction structure in the same video processing module performs feature extraction on multiple frames of video images captured from the target video segment to obtain reference image feature vectors for each frame. The feature fusion layer in the same video processing module fuses the reference video feature vector and the reference image feature vectors for each frame to obtain the target video feature vector.

[0089] The following is combined with Figure 4 The illustrated diagram shows the working principle of an exemplary video processing module, illustrating the working principle of the aforementioned video processing module. Figure 4 As shown, the video processing module includes a video feature extraction structure 401, an image feature extraction structure 402, and a feature fusion layer 403. The video feature extraction structure 401 may include a 3D Convolutional Neural Network (3D-CNN) and two fully connected layers. The image feature extraction structure 402 may include n (n is an integer greater than 1) Convolutional Neural Networks (CNN) and RNNs. The feature fusion layer 403 may be a fully connected layer.

[0090] In practice, the 3D-CNN in the video feature extraction structure 401 models the input target video segment to obtain the primary video feature vector of the target video segment; then, the two fully connected layers in the video feature extraction structure 401 process the primary video feature vector step by step to obtain the reference video feature vector.

[0091] It should be noted that when the length of the input target video segment is long, for example, when the length of the target video segment exceeds the preset length, the target video segment can be divided into multiple sub-video segments according to the preset time unit, such as 5 seconds. Then, the 3D-CNN in the video feature extraction structure 401 is used to perform feature extraction processing on each sub-video segment to obtain the primary video feature vectors corresponding to each of the multiple sub-video segments. Then, the primary video feature vectors corresponding to each of the multiple sub-video segments are merged and concatenated through the two fully connected layers in the video feature extraction structure 401 to obtain the reference video feature vector.

[0092] It should be understood that Figure 4 The video feature extraction structure 401 shown is only an example. In practical applications, other structural forms of video feature extraction structures 401 can be designed according to actual application requirements. This application does not impose any limitations on the specific structure of the video feature extraction structure 401.

[0093] In addition, the server can capture n frames of video images from the target video segment; for example, the server can divide the target video segment into n sub-video segments of equal duration, and then capture one frame of video image from each sub-video segment to obtain n frames of video images.

[0094] Furthermore, the image feature extraction structure 402 in the video processing module can perform feature extraction processing on these n video images to obtain a reference image feature vector. Specifically, the n CNNs in the image feature extraction structure 402 can be used to perform feature extraction processing on these n video images respectively to obtain the primary image feature vectors corresponding to each of the n video images; then, the RNN in the image feature extraction structure 402 can determine the reference image feature vector based on the reference video feature vector extracted by the video feature extraction structure 401 and the primary image feature vectors corresponding to each of the n video images.

[0095] like Figure 4 As shown, the n CNNs in the image feature extraction structure 402 have the same structure, and each CNN can include convolutional layers, pooling layers, and fully connected layers. When performing feature extraction processing on each frame of video image, the video image can be processed step by step through the convolutional layers, pooling layers, and fully connected layers in the CNN to obtain the primary image feature vector corresponding to that frame of video image.

[0096] like Figure 4 As shown, the RNN in the image feature extraction structure 402 needs to determine the hidden layer vector corresponding to each frame of video image based on the primary image feature vector. Specifically, the RNN needs to determine the hidden layer vector corresponding to the i-th frame of video image (i is an integer greater than or equal to 1 and less than or equal to n). When i equals 1, the RNN needs to determine the hidden layer vector corresponding to the i-th frame of video image based on the reference video feature vector extracted by the video feature extraction structure 401 and the primary image feature vector corresponding to the i-th frame of video image. When i is not equal to 1, the RNN needs to determine the hidden layer vector corresponding to the i-th frame of video image based on the hidden layer vector corresponding to the (i-1)-th frame of video image and the primary image feature vector corresponding to the i-th frame of video image. The final hidden layer vector corresponding to the n-th frame of video image is the required reference image feature vector.

[0097] Specifically, the RNN merges the reference video feature vector with the primary image feature vectors corresponding to each of the n video frames to obtain the hidden vector corresponding to the last video frame. This hidden vector is the reference image feature vector, which carries the features of the n video frames and the target video segment. The RNN has two inputs: one is the primary image feature vector corresponding to the currently processed video frame, and the other is the hidden vector corresponding to the previous video frame. For the first video frame, since there are no preceding video frames, the reference video feature vector extracted by video feature extraction structure 401 can be used as a substitute.

[0098] It should be understood that Figure 4The image feature extraction structure 402 shown is only an example. In practical applications, other structural forms of image feature extraction structures 402 can be designed according to actual application requirements. This application does not impose any limitations on the specific structure of the image feature extraction structure 402.

[0099] After obtaining a reference video feature vector through video feature extraction structure 401 and a reference image feature vector through image feature extraction structure 402, the video processing module can further fuse the reference video feature vector and the reference image feature vector through feature fusion layer 403 to obtain the target video feature vector; for example Figure 4 As shown, the video processing module can use a fully connected layer to fuse and merge the feature vector of the reference video and the feature vector of the reference image to obtain the target video feature vector.

[0100] It should be understood that Figure 4 The feature fusion layer 403 shown is only an example. In practical applications, other structural forms of feature fusion layer 403 can be designed according to actual application requirements. This application does not impose any limitations on the specific structure of the feature fusion layer 403.

[0101] It should be noted that in practical applications, steps 202 and 203 can be executed simultaneously, or steps 202 can be executed first and then steps 203, or steps 203 can be executed first and then steps 202. This application does not impose any restrictions on the execution order of steps 202 and 203.

[0102] Step 204: Using the fusion feature processing module in the bullet screen recognition model, determine the recognition result corresponding to the target bullet screen based on the target text feature vector and the target video feature vector; the recognition result corresponding to the target bullet screen is used to characterize the type of the target bullet screen.

[0103] The server extracts the target text feature vector through the text processing module in the bullet screen recognition model, and then extracts the target video feature vector through the video processing module in the bullet screen recognition model. It can further use the fusion feature processing module in the bullet screen recognition model to determine the bullet screen recognition result corresponding to the target bullet screen to be identified based on the target text feature vector and the target video feature vector, that is, to determine the type of the target bullet screen, such as whether the target bullet screen belongs to inappropriate bullet screen.

[0104] Figure 5 The diagram shows a schematic representation of an exemplary bullet screen recognition model. Figure 5 As shown, the bullet screen recognition model includes a text processing module 501, a video processing module 502, and a fusion feature processing module 503. Specifically, the text processing module 501 can be... Figure 3The structure shown indicates that the video processing module 502 can specifically be... Figure 4 The structure shown includes a first fully connected layer, a convolutional layer, a pooling layer, and a second fully connected layer in the feature processing module 503.

[0105] like Figure 5 As shown, the text processing module 501 performs feature extraction processing on the target text to obtain a target text feature vector, and then inputs the target text feature vector into the fusion feature processing module 503; the video processing module 502 performs feature extraction processing on the target video segment to obtain a target video feature vector, and then inputs the target video feature vector into the fusion feature processing module 503. The first fully connected layer in the fusion feature processing module 503 performs primary fusion processing on the input target text feature vector and target video feature vector to obtain a primary fusion feature vector; then, the convolutional layer, pooling layer and the second fully connected layer in the fusion feature processing module 503 process the primary fusion feature vector step by step, and finally outputs the recognition result corresponding to the target bullet screen. The recognition result corresponding to the target bullet screen can characterize whether the target bullet screen belongs to the category of malicious bullet screens.

[0106] It should be understood that the recognition result corresponding to the target bullet comment can be the probability that the target bullet comment belongs to inappropriate bullet comments. If the probability exceeds a preset probability threshold, the target bullet comment can be determined to be inappropriate; conversely, if the probability does not exceed the preset probability threshold, the target bullet comment can be determined not to be inappropriate. Alternatively, the recognition result of the target bullet comment can also be 1 or 0. If the recognition result is 1, it indicates that the target bullet comment belongs to inappropriate bullet comments; if the recognition result is 0, it indicates that the target bullet comment does not belong to inappropriate bullet comments. This application does not impose any limitations on the recognition result output by the bullet comment recognition model.

[0107] It should be understood that Figure 5 The fusion feature processing module 503 in the bullet screen recognition model shown is only an example. In practical applications, other structural forms of fusion feature processing module 503 can be designed according to actual application requirements. This application does not limit the specific structure of the fusion feature processing module 503.

[0108] The data processing method provided in this application takes into account the high reference value of the video information corresponding to the bullet comments when determining whether they are inappropriate. Based on this, when identifying whether a target bullet comment is inappropriate using a bullet comment recognition model, it comprehensively considers both the text information of the target bullet comment itself and the visual information of the target video segment corresponding to the target bullet comment. The visual information of the target video segment is used to assist in identifying whether the target bullet comment is inappropriate. This improves the accuracy of inappropriate bullet comment recognition, avoids missed or misidentified inappropriate bullet comments, and thus helps to effectively filter inappropriate bullet comments, preventing them from being displayed to video viewers and improving the user's video viewing experience.

[0109] against Figure 2 The bullet screen recognition model used in the illustrated embodiment is further described below through a method embodiment.

[0110] See Figure 6 , Figure 6 This is a flowchart illustrating the training method for the bullet screen recognition model provided in this application embodiment. For ease of description, the following embodiment uses a server as the execution entity for the bullet screen recognition model training method. Figure 6 As shown, the training method for this bullet screen recognition model includes the following steps:

[0111] Step 601: Obtain the first training sample; the first training sample includes the first training text of the training bullet comments, the first training video segment corresponding to the training bullet comments, and the annotation result corresponding to the training bullet comments.

[0112] Before the server trains the bullet screen recognition model, it needs to obtain a large number of labeled first training samples. Based on the obtained first training samples, a supervised training task is performed to train the initial bullet screen recognition model and obtain a bullet screen recognition model that can be put into practical application.

[0113] It should be noted that the first training sample is the labeled sample used to train the initial bullet screen recognition model; it includes the first training text of the training bullet screen, the first training video segment corresponding to the training bullet screen, and the labeling result corresponding to the training bullet screen. The training bullet screen is the bullet screen used when training the initial bullet screen recognition model. It can be obtained by randomly scraping bullet screens from the internet, for example, randomly scraping several bullet screens sent to a specific video as training bullet screens, or manually creating training bullet screens. This application does not impose any restrictions on the method of obtaining the training bullet screen. The first training text of the training bullet screen is the text carried in the training bullet screen. The first training video segment corresponding to the training bullet screen can be a video segment played at the creation time and / or sending time of the training bullet screen. The method of obtaining the first training video segment corresponding to the training bullet screen is the same as... Figure 2 The method for obtaining the target video segment corresponding to the target bullet comment in the illustrated embodiment is the same; for details, please refer to the method for obtaining the target video segment above. The annotation results corresponding to the training bullet comments are used to characterize whether the training bullet comments are malicious bullet comments, and can usually be obtained by relevant technical personnel.

[0114] Step 602: Using the initial text processing module in the initial bullet screen recognition model to be trained, perform feature extraction processing on the first training text to obtain the first training text feature vector; using the initial video processing module in the initial bullet screen recognition model, perform feature extraction processing on the first training video segment to obtain the first training video feature vector.

[0115] Step 603: Using the initial fusion feature processing module in the initial bullet screen recognition model, determine the prediction result corresponding to the training bullet screen based on the first training text feature vector and the first training video feature vector.

[0116] Since steps 602 and 603 are highly related and both belong to the internal working process of the initial bullet screen recognition model, the implementation process of steps 602 and 603 will be introduced in general below.

[0117] After the server obtains the first training sample, it can use the first training sample to train the initial bullet screen recognition model. Specifically, the initial text processing module in the initial bullet screen recognition model can first perform feature extraction processing on the first training text included in the first training sample to obtain the first training text feature vector; and the initial video processing module in the initial bullet screen recognition model can perform feature extraction processing on the first training video segment included in the first training sample to obtain the first training video feature vector. Then, the initial fusion feature processing module in the initial bullet screen recognition model can determine the prediction result corresponding to the training bullet screen based on the first training text feature vector and the first training video feature vector, that is, predict whether the training bullet screen belongs to the category of inappropriate bullet screens.

[0118] It should be noted that the initial bullet comment recognition model is a pre-built bullet comment recognition model structure to be trained, which is given initial model parameters. The model structure of this initial bullet comment recognition model is similar to... Figure 2 The model structures of the bullet screen recognition models in the illustrated embodiments are the same. For example, the model structure of the initial bullet screen recognition model can be as follows: Figure 5 As shown, this includes... Figure 3 The initial text processing module shown in the diagram, and the... Figure 4 The initial video processing module with the structure shown; correspondingly, the implementation principle of this initial bullet screen recognition model in determining the prediction result corresponding to the training bullet screen based on the first training text and the first training video segment is the same as... Figure 2 The working principle of the bullet screen recognition model in the illustrated embodiment is the same; for details, please refer to [link / reference]. Figure 2 The relevant description in the illustrated embodiment. This initial bullet screen recognition model and Figure 2 The only difference between the bullet screen recognition models in the illustrated embodiments is the model parameters. This embodiment aims to use the first training samples obtained in step 601 to train the initial bullet screen recognition model, adjust the model parameters of the initial bullet screen model, and thus obtain... Figure 2 The bullet screen recognition model in the illustrated embodiment.

[0119] In one possible implementation, the initial bullet screen recognition model can be a model obtained through pre-training. For example, it can be a model obtained by pre-training a matching relationship recognition model used to identify whether there is a matching relationship between videos and text. Pre-training refers to training the neural network model on a different training task before formally training (fine-tuning) it. Formal training based on a pre-trained neural network model can effectively improve the performance of formal training and reduce the amount of training samples required, thus increasing the training speed. The pre-training process of this initial bullet screen recognition model will be described below through another method embodiment; details can be found in the relevant content below.

[0120] In one possible implementation, the initial text processing module in the initial bullet screen recognition model can also be a partial structure of the model obtained through pre-training. For example, it can be a partial structure of the model obtained by pre-training a text classification model. It should be understood that the pre-trained initial text processing module already possesses strong text feature extraction capabilities. Formal training of the initial bullet screen recognition model, including this initial text processing module, can also improve the effectiveness of formal training, reduce the amount of training samples required during formal training, and increase the model training speed. The pre-training process of this initial text processing module will be described below through another method embodiment; details can be found in the relevant content below.

[0121] In one possible implementation, the initial video processing module in the initial bullet screen recognition model can also be a partial structure of the model obtained through pre-training. For example, it can be a partial structure of the model obtained by pre-training a video image reconstruction model. It should be understood that the pre-trained initial video processing module already possesses strong video feature extraction capabilities. Formal training of the initial bullet screen recognition model, including this initial video processing module, can also improve the effectiveness of formal training, reduce the amount of training samples required during formal training, and increase the model training speed. The pre-training process of this initial video processing module will be described below through another method embodiment; details can be found in the relevant content below.

[0122] Step 604: Based on the prediction results and the annotation results, train the initial bullet screen recognition model; once the initial bullet screen recognition model meets the first training termination condition, the bullet screen recognition model is obtained.

[0123] After determining the prediction result corresponding to the training bullet screen based on the first training text and the first training video segment in the first training sample using the initial bullet screen recognition model, the server can construct a loss function for training the initial bullet screen recognition model based on the annotation result and prediction result of the training bullet screen included in the first training sample. Then, with the goal of minimizing the loss function, the server trains the initial bullet screen recognition model and adjusts the model parameters of the initial bullet screen recognition model.

[0124] Once the initial trained bullet screen recognition model meets the first training termination condition, it can be used as a bullet screen recognition model that can be applied in practice. Figure 2 The illustrated embodiment shows a bullet screen recognition model. The first training termination condition here is used to measure whether the initial bullet screen recognition model has been successfully trained. As an example, the first training termination condition could be that the number of iterations for training the initial bullet screen recognition model reaches a preset number of training iterations. As another example, the first training termination condition could be that the performance of the initial bullet screen recognition model reaches a preset performance standard. For example, the initial bullet screen recognition model can be tested using a test sample set to obtain its current recognition accuracy. If the recognition accuracy exceeds a preset accuracy threshold, it can be determined that the initial bullet screen recognition model has met the first training termination condition. Of course, in practical applications, other conditions can also be set as the first training termination condition; this application does not impose any limitations on the first training termination condition.

[0125] The bullet comment recognition model trained using the initial training method described above can comprehensively consider both the textual information of the bullet comment itself and the visual information of the corresponding video clip when identifying whether a bullet comment is inappropriate. It utilizes the visual information of the video clip to assist in identifying whether a bullet comment is inappropriate. This improves the accuracy of identifying inappropriate bullet comments and avoids missed or false identifications.

[0126] Figure 6 As mentioned in the embodiments shown, the initial bullet screen recognition model can be obtained by pre-training a matching relationship recognition model used to identify whether there is a matching relationship between video and text. The pre-training method of the initial bullet screen recognition model will be described in detail below through method embodiments.

[0127] See Figure 7 , Figure 7 This is a flowchart illustrating the pre-training method for the initial bullet screen recognition model provided in this application embodiment. For ease of description, the following embodiment uses a server as the execution entity of the pre-training method for the initial bullet screen recognition model as an example. Figure 7 As shown, the pre-training method for this initial bullet screen recognition model includes the following steps:

[0128] Step 701: Obtain the second training sample; the second training sample includes positive training samples and negative training samples, wherein the positive training sample includes a second training video segment and a second training text that have a matching relationship; and the negative training sample includes a second training video segment and a second training text that do not have a matching relationship.

[0129] Before the server trains the matching relationship recognition model, it needs to acquire a large number of second training samples. The matching relationship recognition model is then trained based on these second training samples to obtain... Figure 6 The initial bullet screen recognition model in the illustrated embodiment.

[0130] It should be noted that the second training samples are used to train the matching relationship recognition model. The obtained second training samples include positive training samples and negative training samples; the positive training samples include second training video clips and second training text with matching relationships, and the negative training samples include second training video clips and second training text without matching relationships. The second training video clips are the video clips used when training the matching relationship recognition model, and the second training text is the text used when training this matching relationship recognition model. Both the second training video clips and second training text with matching relationships, and the second training video clips and second training text without matching relationships, can be randomly crawled from the internet.

[0131] As an example, the second training video clips and second training texts included in the positive training samples that have a matching relationship can be video clips and their corresponding title texts, or video clips and their subtitle texts. The second training video clips and second training texts included in the negative training samples that do not have a matching relationship can be the title texts of the first video clips and the second video clips, or the subtitle texts of the first video clips and the second video clips. The first video clips and the second video clips are completely unrelated video clips, such as two video clips with completely different content.

[0132] Of course, in practical applications, the second training video segment and the second training text included in the positive training samples can also be other video segments and texts with matching relationships, and the second training video segment and the second training text included in the negative training samples can also be other video segments and texts without matching relationships. This application does not impose any limitations on the second training video segment and the second training text included in the above-mentioned positive training samples and negative training samples.

[0133] Step 702: Using the initial text processing module in the matching relationship recognition model, perform feature extraction processing on the second training text in the target second training sample to obtain the second training text feature vector; using the initial video processing module in the matching relationship recognition model, perform feature extraction processing on the second training video segment in the target second training sample to obtain the second training video feature vector.

[0134] Step 703: Using the initial fusion feature processing module in the matching relationship recognition model, determine the predicted matching relationship between the second training text and the second training video segment in the target second training sample based on the second training text feature vector and the second training video feature vector.

[0135] Since steps 702 and 703 are highly correlated and both belong to the internal working process of the matching relationship recognition model, the implementation process of steps 702 and 703 will be introduced in general below.

[0136] After the server obtains a large number of second training samples, it can select any second training sample as the target second training sample. The matching relationship recognition model to be trained is then used to process the second training text and the second training video segment in the target second training sample to predict whether there is a matching relationship between the second training text and the second training video segment in the target second training sample.

[0137] Specifically, the initial text processing module in the matching relationship recognition model can be used to extract features from the second training text in the second training sample of the target, resulting in a feature vector of the second training text. The initial video processing module in the matching relationship recognition model can be used to extract features from the second training video segment in the second training sample of the target, resulting in a feature vector of the second training video. Then, the initial fusion feature processing module in the matching relationship recognition model can be used to determine the predicted matching relationship between the second training text and the second training video segment in the second training sample of the target, based on the feature vector of the second training text and the feature vector of the second training video.

[0138] It should be noted that the matching relationship recognition model described above is used to identify whether there is a matching relationship between text and video, and it differs from the model described above. Figure 6 The initial bullet screen recognition models in the illustrated embodiments have the same model structure and are assigned initial model parameters. This embodiment aims to train the matching relationship recognition model using the large number of second training samples obtained in step 701, adjust the model parameters of the matching relationship recognition model to achieve better matching relationship recognition performance, and then use the matching relationship recognition model with better matching relationship recognition performance as the model described above. Figure 6The initial bullet screen recognition model in the illustrated embodiment provides a good foundation for the formal training of the bullet screen recognition model.

[0139] Figure 8 The diagram shown illustrates the working principle of an exemplary matching relationship recognition model provided in this application. Figure 8 As shown, the matching relationship recognition model includes an initial text processing module 801, an initial video processing module 802, and an initial fusion feature processing module 803. The working principle of the initial text processing module 801 is similar to that described above. Figure 3 The text processing module in the illustrated bullet screen recognition model works similarly to extract the feature vector of the second training text from the second training text in the target second training sample. The initial video processing module 802 works in the same way as described above. Figure 4 The video processing module shown works similarly, extracting the second training video feature vector from the second training video segment in the target second training sample. The initial fusion feature processing module 803 works in the same way as described above. Figure 5 The working principle of the fusion feature processing module in the bullet screen recognition model is similar to that of the fusion feature processing module in the bullet screen recognition model. It is used to fuse the feature vector of the second training text and the feature vector of the second training video. However, the initial fusion feature processing module 803 determines whether there is a matching relationship between the second training text and the second video segment, while the fusion feature processing module 503 determines whether the target bullet screen is an inappropriate bullet screen. That is, although the working principle of the initial fusion feature processing module 803 in the matching relationship recognition model and the fusion feature processing module in the bullet screen recognition model are similar, the tasks they need to perform are different.

[0140] In one possible implementation, the initial text processing module in the matching relationship recognition model can be a partial structure of the model obtained through pre-training, for example, a partial structure of the model obtained by pre-training a text classification model. It should be understood that the pre-trained initial text processing module already possesses strong text feature extraction capabilities. Training the matching relationship recognition model including this initial text processing module can improve training effectiveness, reduce the amount of training samples required during training, and increase model training speed. The pre-training process of this initial text processing module will be described below through another method embodiment; details can be found in the relevant content below.

[0141] In one possible implementation, the initial video processing module in the matching relationship recognition model can also be a partial structure of the model obtained through pre-training. For example, it can be a partial structure of the model obtained by pre-training a video image reconstruction model. It should be understood that the pre-trained initial video processing module already possesses strong video feature extraction capabilities. Training the matching relationship recognition model including this initial video processing module can improve training effectiveness, reduce the amount of training samples required during training, and increase model training speed. The pre-training process of this initial video processing module will be described below through another method embodiment; details can be found in the relevant content below.

[0142] Step 704: Based on the type of the target second training sample and the matching relationship prediction result, train the matching relationship recognition model; when the matching relationship recognition model meets the second training termination condition, obtain the initial bullet screen recognition model.

[0143] The server, through a matching relationship recognition model, determines the predicted matching relationship between the second training text and the second training video segment in the target second training sample. Then, based on the type of the target second training sample (i.e., whether it is a positive or negative training sample) and the predicted matching relationship between the second training text and the second training video segment in the target second training sample, it constructs a loss function for training the matching relationship recognition model. Finally, with the goal of minimizing this loss function, the matching relationship recognition model is trained, and its model parameters are adjusted.

[0144] Once the trained matching relationship recognition model satisfies the second training termination condition, it can be used as the initial bullet screen recognition model, i.e., the training basis model for the bullet screen recognition model. The second training termination condition here is used to measure whether the matching relationship recognition model has been successfully trained. As an example, this second training termination condition could be that the number of iterations for training the matching relationship recognition model reaches a preset number of training iterations. As another example, the second training termination condition could be that the performance of the matching relationship recognition model reaches a preset performance standard. For example, the matching relationship recognition model can be tested using a test sample set to obtain its current recognition accuracy. If the recognition accuracy exceeds a preset accuracy threshold, it can be determined that the matching relationship recognition model has met the second training termination condition. Of course, in practical applications, other conditions can also be set as the second training termination condition; this application does not impose any limitations on the second training termination condition here.

[0145] By pre-training the matching relationship recognition model using the above method, we obtain the training base model of the bullet screen recognition model, namely the initial bullet screen recognition model. Since the acquisition of second training samples and second training video segments with matching relationships (i.e., positive training samples) and second training samples and second training video segments without matching relationships (i.e., negative training samples) is very easy, a large number of second training samples can be easily obtained. Accordingly, training the matching relationship recognition model with a large number of second training samples helps to train a matching relationship recognition model with better model performance, that is, it can train an initial bullet screen recognition model with better model performance. This provides a better training foundation for the formal training of the bullet screen recognition model, improves the training effect and efficiency of formal training, and reduces the training samples required for formal training, thereby reducing the cost of sample labeling.

[0146] Figure 6 and Figure 7 The embodiments shown mention the initial text processing module in the initial bullet screen recognition model and the matching relationship recognition model, which can be obtained by pre-training the text classification model. The pre-training method of the initial text processing module will be described in detail below through method embodiments.

[0147] See Figure 9 , Figure 9 This is a flowchart illustrating the pre-training method for the initial text processing module provided in this application embodiment. For ease of description, the following embodiment uses a server as the execution entity of the pre-training method for the initial text processing module as an example. Figure 9 As shown, the pre-training method for this initial text processing module includes the following steps:

[0148] Step 901: Obtain the third training sample; the third training sample includes the third training text and the labeled text category corresponding to the third training text.

[0149] Before the server trains the text classification model, it needs to obtain a large number of labeled third training samples. Based on the obtained third training samples, a supervised task is performed to train the text classification model. Then, some structures are extracted from the text classification model as the initial text processing module.

[0150] It should be noted that the third training sample is a labeled sample used to train the text classification model; this includes the third training text and the corresponding labeled text category. The third training text is the text used when training the text classification model. In practical applications, this third training text can be obtained in any way. For example, a paragraph or sentence can be randomly selected from an article as the third training text, and the corresponding category of the article (such as sentiment, science, suspense, etc.) is the labeled text category of the third training text. Of course, the third training sample can also be obtained from a general training dataset used to train the text classification model. This application does not impose any restrictions on the method of obtaining the third training sample.

[0151] Step 902: Determine the predicted text category corresponding to the third training text using the text classification model.

[0152] After the server obtains the third training sample, it can analyze and process the third training text in the third training sample using the text classification model to be trained, thereby determining the predicted text category corresponding to the third training text.

[0153] It should be noted that the above text classification model is used to identify the category to which text belongs, and some of its structure is similar to... Figure 8 The structure of the initial text processing module in the matching relationship recognition model shown is the same as that of the matching relationship recognition model. Figure 3 The text processing modules shown have the same structure, and the text classification model is given initial model parameters. This embodiment aims to train the text classification model using the large number of third training samples obtained in step 901, adjust the model parameters to achieve better text recognition performance, and then use a portion of the structure of this text recognition model with better performance as the basis for the above text processing module. Figure 7 The initial text processing module in the matching relationship recognition model shown in the embodiment, or the above... Figure 6 The initial text processing module in the initial bullet screen recognition model in the illustrated embodiment provides a good model training foundation for the pre-training of the initial bullet screen recognition model or the formal training of the bullet screen recognition model.

[0154] Figure 10 The diagram shown illustrates the working principle of an exemplary text recognition model provided in this application. Figure 10 As shown, the text recognition model includes an initial word feature extraction structure 1001 and an initial word feature fusion structure 1002. The working principle of the initial word feature extraction structure 1001 is similar to... Figure 3The working principle of the word feature extraction structure in the text processing module shown is similar to that of the word feature extraction structure. It is used to extract features from each word segment in the third training text to obtain the training word feature vector corresponding to each word segment. The working principle of the initial word feature fusion structure 1002 is the same as that of the word feature extraction structure in the text processing module shown. Figure 3 The working principle of the word feature fusion structure in the text processing module shown is similar. It is used to fuse the training word feature vectors corresponding to each word segment in the third training text to obtain the third training text feature vector. Then, through the classification layer in the text recognition model, the predicted text category corresponding to the third training text is determined based on the third training text feature vector.

[0155] Step 903: Based on the predicted text category and the labeled text category, train the text classification model; once the text classification model meets the third training termination condition, use the structure of the text classification model other than the classification layer as the initial text processing module.

[0156] After the server determines the predicted text category corresponding to the third training text in the third training sample through the text classification model, it can construct a loss function for training the text classification model based on the labeled text type and the predicted text category corresponding to the third training text in the third training sample. Then, with the goal of minimizing the loss function, the server trains the text classification model and adjusts the model parameters of the text classification model.

[0157] Once the trained text classification model satisfies the third training termination condition, the other structures in the text classification model other than the classification layer (i.e., the initial word feature extraction structure and the initial word feature fusion structure) can be used as the initial text processing module in the initial bullet screen recognition model or the initial text processing module in the matching relationship recognition model.

[0158] The aforementioned third training termination condition is used to measure whether the text classification model has been successfully trained. As an example, the third training termination condition could be that the number of iterations of training the text classification model reaches a preset number of training iterations. As another example, the third training termination condition could be that the performance of the text classification model reaches a preset performance standard. For example, the text classification model can be tested using a test sample set to obtain the current recognition accuracy of the text classification model. If the recognition accuracy exceeds a preset accuracy threshold, it can be determined that the text classification model has met the third training termination condition. Of course, in practical applications, other conditions can also be set as the third training termination condition. This application does not impose any limitations on the third training termination condition here.

[0159] The text classification model is pre-trained using the above method to obtain the initial text processing module in the initial bullet screen recognition model or the initial text processing module in the matching relationship recognition model. Since obtaining training data for text classification is relatively easy, a large number of third training samples can be readily acquired. Accordingly, using a large number of third training samples to train the text classification model helps to obtain a text classification model with superior performance. Consequently, using a portion of the structure of this text classification model as the initial text processing module in the initial bullet screen recognition model or the initial text processing module in the matching relationship recognition model provides a better training foundation for the initial bullet screen recognition model or the matching relationship recognition model, helps to improve training effectiveness and efficiency, and reduces the number of training samples required during training.

[0160] Figure 6 and Figure 7 The embodiments shown mention the initial video processing module in the initial bullet screen recognition model and the matching relationship recognition model, which can be obtained by pre-training the video image reconstruction model. The pre-training method of the initial video processing module will be described in detail below through method embodiments.

[0161] See Figure 11 , Figure 11 This is a flowchart illustrating the pre-training method for the initial video processing module provided in this application embodiment. For ease of description, the following embodiment uses a server as the execution entity of the pre-training method for the initial video processing module as an example. Figure 11 As shown, the pre-training method for this initial video processing module includes the following steps:

[0162] Step 1101: Obtain the fourth training sample; the fourth training sample includes the third training video segment.

[0163] Before the server trains the video image reconstruction model, it needs to acquire a large number of fourth training samples to train the video image reconstruction model based on the acquired fourth training samples, and then extract some structures from the video image reconstruction model as the initial video processing module.

[0164] It should be noted that the fourth training sample is a sample used to train the video image reconstruction model; it includes the third training video segment, which is a video segment used when training the video image reconstruction model. In practical applications, the third training video segment can be obtained in any way. For example, any video segment can be randomly crawled from the Internet as the third training video segment. This application does not impose any restrictions on the way the third training video segment is obtained.

[0165] Step 1102: Using the initial video feature extraction structure in the video image reconstruction model, perform feature extraction processing on the third training video segment to obtain the third training video feature vector.

[0166] Step 1103: Using the initial image feature extraction structure in the video image reconstruction model, perform feature extraction processing on multiple frames of training video images collected from the third training video segment to obtain the training image feature vectors corresponding to each of the multiple training video images; determine the training hidden layer vectors corresponding to each of the multiple training video images based on the third training video feature vectors and the training image feature vectors corresponding to each of the multiple training video images.

[0167] Step 1104: Using the image reconstruction structure in the video image reconstruction model, generate the reconstructed images corresponding to each of the multi-frame training video images according to the training hidden layer vectors corresponding to each of the multi-frame training video images.

[0168] Since steps 1102, 1103, and 1104 are highly related and all belong to the internal working process of the video image reconstruction model, the implementation process of steps 1102, 1103, and 1104 will be introduced in general below.

[0169] After the server obtains the fourth training sample, it can analyze and process the third training video segment in the fourth training sample using the video image reconstruction model to be trained, and reconstruct multiple frames of training video images in the third training video segment to obtain the reconstructed images corresponding to each of these multiple frames of training video images.

[0170] Specifically, the initial video feature extraction structure in the video image reconstruction model can be used to extract features from the third training video segment, resulting in a third training video feature vector. This initial image feature extraction structure can then be used to extract features from multiple frames of training video images captured from the third training video segment, yielding training image feature vectors for each frame. Based on the third training video feature vector and the corresponding training image feature vectors for each frame, the corresponding training hidden layer vectors for each frame are determined. Finally, using the image reconstruction structure in the video image reconstruction model, and based on the corresponding training hidden layer vectors, reconstructed images for each of the training video frames are generated.

[0171] It should be noted that the above video image reconstruction model is a model used to reconstruct video images in a video, and some of its structures are similar to those in other videos. Figure 8 The structure of the initial video processing module in the matching relationship recognition model shown is the same as that of the matching relationship recognition model. Figure 4The video processing modules shown have the same structure, and the video image reconstruction model is given initial model parameters. This application embodiment aims to train the video image reconstruction model using the large number of fourth training samples obtained in step 1101, adjust the model parameters of the video image reconstruction model to achieve better image reconstruction performance, and then use a portion of the structure of the video image reconstruction model as described above. Figure 7 The initial video processing module in the matching relationship recognition model shown in the embodiment, or the above Figure 6 The initial video processing module in the initial bullet screen recognition model in the illustrated embodiment provides a good model training foundation for the pre-training of the initial bullet screen recognition model or the formal training of the bullet screen recognition model.

[0172] Figure 12 The diagram shown illustrates the working principle of an exemplary video image reconstruction model provided in this application. Figure 12 As shown, the video image reconstruction model includes an initial video feature extraction structure 1201, an initial image feature extraction structure 1202, and an image reconstruction structure 1203. The working principle of the initial video feature extraction structure 1201 is similar to... Figure 4 The working principle of the video feature extraction structure in the video processing module shown is similar to that of the third training video segment, used to extract features from the third training video segment to obtain the feature vector of the third training video; the working principle of the initial image feature extraction structure 1202 is the same as that of the third training video segment. Figure 4 The working principle of the image feature extraction structure in the video processing module shown is similar, used to determine the training hidden layer vectors corresponding to each of the multiple training video images collected from the third training video; the image reconstruction structure 1203 is used to reconstruct the multiple training video images based on the training hidden layer vectors corresponding to each of the multiple training video images, and obtain the reconstructed images corresponding to each of the multiple training video images.

[0173] Step 1105: Train the video image reconstruction model based on the multi-frame training video images and the reconstructed images corresponding to each of the multi-frame training video images; once the video image reconstruction model meets the fourth training termination condition, determine the initial video processing module based on the initial video feature extraction structure and the initial image feature extraction structure in the image reconstruction model.

[0174] After obtaining the reconstructed images corresponding to each of the multiple training video images in the third training video segment through the video image reconstruction model, the server can construct a loss function for training the video image reconstruction model based on these multiple training video images (i.e., the original images) and their corresponding reconstructed images in the third training video segment. Then, with the goal of minimizing the loss function, the server trains the video image reconstruction model and adjusts its model parameters.

[0175] Once the trained video image reconstruction model satisfies the fourth training termination condition, the initial video feature extraction structure and the initial image feature extraction structure in the video image reconstruction model can be used as the initial video processing module in the initial bullet screen recognition model or the initial video processing module in the matching relationship recognition model.

[0176] The aforementioned fourth training termination condition is used to measure whether the video image reconstruction model has been successfully trained. As an example, this fourth training termination condition could be that the number of iterations for training the video image reconstruction model reaches a preset number of training iterations. As another example, the fourth training termination condition could be that the performance of the video image reconstruction model reaches a preset performance standard. For example, the video image reconstruction model can be tested using a test sample set to obtain its current reconstruction accuracy. If the reconstruction accuracy exceeds a preset accuracy threshold, it can be determined that the video image reconstruction model has met the fourth training termination condition. Of course, in practical applications, other conditions can also be set as the fourth training termination condition. This application does not impose any limitations on the fourth training termination condition.

[0177] The video image reconstruction model is pre-trained using the above method to obtain the initial video processing module in the initial bullet screen recognition model or the initial video processing module in the matching relationship recognition model. Since the third training video segment is relatively easy to obtain and requires no annotation, a large number of fourth training samples can be easily acquired. Accordingly, training the video image reconstruction model using a large number of fourth training samples helps to train a video image reconstruction model with superior performance. Consequently, using a portion of the structure of this video image reconstruction model as the initial video processing module in the initial bullet screen recognition model or the initial video processing module in the matching relationship recognition model provides a better training foundation for the initial bullet screen recognition model or the matching relationship recognition model, helping to improve training effectiveness and efficiency, and reducing the number of training samples required during training.

[0178] To facilitate a further understanding of the training process of the bullet screen recognition model provided in the embodiments of this application, the following is combined with... Figure 13 The diagram shown illustrates the training process of the bullet screen recognition model, providing an overall illustrative description of the training process of the bullet screen recognition model provided in this application embodiment.

[0179] like Figure 13As shown, the training process of the bullet screen recognition model provided in this application embodiment is mainly divided into the following parts: 1. Text classification model pre-training part, that is, training the text classification model based on the third training text; 2. Text processing part, that is, using some structures in the trained text classification model to perform text feature extraction processing in the matching relationship recognition model; 3. Video image reconstruction model pre-training part, that is, training the video image reconstruction model based on the third training video segment; 4. Video processing part, that is, using some structures in the trained video image reconstruction model to perform video feature extraction processing in the matching relationship recognition model; 5. Matching relationship recognition model pre-training part, that is, training the matching relationship recognition model for recognizing whether there is a matching relationship between video and text based on the second training video segment and the second training text in the second training sample, so as to obtain the initial bullet screen recognition model; 6. Bullet screen recognition model formal training part, that is, formally training the initial bullet screen recognition model based on the first training text and the first training video segment in the first training sample, so as to obtain the bullet screen recognition model.

[0180] The following sections will introduce each of the six parts mentioned above.

[0181] 1. Text classification model pre-training part

[0182] The specific structure of the text classification model is as follows: Figure 10 As shown, firstly, the third training text from the third training sample is input into an RNN structure (a GRU or LSTM structure can also be used, depending on the specific application scenario) to obtain the primary word feature vectors corresponding to each word segment in the third training text. Then, each primary word feature vector output by the RNN structure is passed through two fully connected layers, then through a convolutional layer and a pooling layer, and finally through another fully connected layer to output a score value. This score value is used to represent the probability that the third training text belongs to a certain text category.

[0183] This application uses general text classification data, such as sentiment classification text data, to train a text classification model. Although this text classification task is functionally unrelated to the malicious comment recognition task, both are essentially text classification tasks. Training a text classification model based on this task will be helpful for the subsequent malicious comment recognition task. This text classification task uses supervised data, and the supervision signal is included in the training dataset. This training dataset can be a publicly available dataset from academia.

[0184] This section pertains to the pre-training process for subsequent parts. The model parameters of the text classification model trained in this section will be copied to the initial text processing module in the matching relationship recognition model that needs to be trained later.

[0185] 2. Text Processing Section

[0186] The structure of this part is similar to that of the text classification model in the first part, except that this part is only for obtaining the feature vector representation of the text, rather than performing a classification task. Therefore, the classification layer in the text classification model in the first part is removed. By removing the classification layer, the text processing structure can convert a text into a feature vector representation. After completing the pre-training of the text classification model in the first part, the model parameters of the other structures in the text classification model, except for the classification layer, can be directly used as the initialization parameters of the initial text processing module.

[0187] 3. Video image reconstruction model pre-training part

[0188] This section can train a video image reconstruction model (the input is a video clip and multiple frames of video images sampled from the video clip, and the output is a video image that is the same as the input video image). The data used in this section is unsupervised data, which can be easily obtained from the Internet.

[0189] The specific structure of the video image reconstruction model is as follows: Figure 12 As shown. Given a third training video segment, a 3D-CNN model can be used in video image reconstruction to model the third training video segment and obtain its feature vector.

[0190] Secondly, multiple frames of training video images can be sampled from the third training video segment (e.g., one frame of video image is collected per second) and input into the video image reconstruction model. In this application, n frames of training video images are collected. Each frame of training video image is fused through a CNN and an RNN. The CNN represents the training video image as a training image feature vector. Then, the RNN generates the hidden layer vectors corresponding to each of the n training video images based on the feature vectors of the third training video and the corresponding training image feature vectors of the n training video images. The hidden layer vectors corresponding to each training video image will be used to reconstruct the image through a fully connected layer.

[0191] Once the video image reconstruction model is trained, it means that the model can accurately understand and characterize the input video. The pre-training in this section uses unlabeled video data, thus obtaining a large-scale training dataset. Iteratively training the video image reconstruction model can improve its understanding of image and video information. The model parameters trained in this section can be used as the initialization parameters for the initial video processing module in Part Four.

[0192] 4. Video Processing Section

[0193] The structure of this part is similar to that of the video image reconstruction model in Part 3 above. The difference is that this part is only for obtaining the feature vector representation of the video, rather than performing image reconstruction. Therefore, the fully connected layer used for image reconstruction in Part 3 is removed, and a new fully connected layer is added to fuse the video feature vector and the last hidden layer vector (which fuses the video feature vector and the features corresponding to each of the n video images). Through this video processing structure, a video segment can be converted into a feature vector representation. After pre-training the video image reconstruction model through Part 3, the model parameters of some structures in this video image reconstruction model can be directly used as the initialization parameters of the initial video processing module.

[0194] 5. Pre-training part of the matching relationship recognition model

[0195] To enable full interaction between video and text and to better facilitate model understanding of both text and video information, this application designs a text-video joint pre-training scheme, specifically training a matching relationship recognition model to identify whether a matching relationship exists between video and text. The structure of this matching relationship recognition model is as follows: Figure 8 As shown, this includes the initial text processing module and the initial video processing module mentioned in Part III above.

[0196] This application can collect "video-text" clustering data from the Internet as a second training sample. It can mainly collect "video-video title" and "video-video subtitle" data from video websites as the second training data. This application can input the collected videos and texts into the matching relationship recognition model to predict whether the video and text have a matching relationship.

[0197] When training the matching relationship recognition model, positive training samples and negative training samples can be constructed. The positive training samples include videos and texts with matching relationships, and the negative training samples include videos and texts without matching relationships. In training the matching relationship recognition model, this application aims to give the videos and texts in the positive training samples as high a score as possible and the videos and texts in the negative training samples as low a score as possible.

[0198] 6. Formal Training of the Bullet Screen Recognition Model

[0199] This section focuses on supervised training, using the matching relationship recognition model trained in Part 5 as the initial bullet screen recognition model. A large number of first training samples are manually labeled. These first training samples include the first training text of the training bullet screen, the first training video segment corresponding to the training bullet screen, and the labeling result of the first training bullet screen (used to characterize whether the first training bullet screen is inappropriate). Then, the initial bullet screen recognition model is trained using the manually labeled first training samples to obtain a bullet screen recognition model ready for practical application.

[0200] It should be noted that the data processing method provided in this application embodiment can be applied to various application scenarios with bullet comments, such as recognizing bullet comments in various live video streams (such as game live streams, shopping live streams, etc.), recognizing bullet comments in video resources (such as TV series, movies, variety shows, etc.), etc. This application does not limit the application scenarios of this data processing method in any way.

[0201] To facilitate understanding of the data processing method provided in the embodiments of this application, the following example illustrates the overall implementation process of the data processing method by taking its application in identifying inappropriate comments in game live streaming.

[0202] During game live streaming, the backend server of the live streaming platform can receive the bullet comments sent by the viewers in the game live streaming room and regard them as target bullet comments to be identified. The target text is extracted from the target bullet comments, and the target video segment corresponding to the target bullet comments is obtained during the game live streaming. For example, based on the sending time of the target bullet comments, a 5-second video segment is extracted from the game live streaming video according to specific video segment extraction rules as the target video segment.

[0203] Then, the pre-trained bullet screen recognition model is invoked to identify whether the target bullet screen belongs to inappropriate bullet screens. Specifically, the server can use the text processing module in the bullet screen recognition model to perform feature extraction processing on the target text to obtain the target text feature vector; for example, it can first use the word feature extraction structure in the text processing module to extract the word feature vectors corresponding to each word in the target text; then, it can use the word feature fusion structure in the text processing module to determine the target text feature vector based on the word feature vectors corresponding to each word in the target text. The server can use the video processing module in the bullet screen recognition model to perform feature extraction processing on the target video segment to obtain the target video feature vector; for example, it can first use the video feature processing structure in the video processing module to perform feature extraction processing on the target video segment to obtain the reference video feature vector; then, it can use the graph feature extraction structure in the video processing module to perform feature extraction processing on multiple frames of video images captured from the target video segment to obtain the reference image feature vectors corresponding to each of the multiple frames of video images; then, it can use the feature fusion layer in the video processing module to fuse the reference video feature vector and the reference image feature vectors corresponding to each of the multiple frames of video images to obtain the target video feature vector. Furthermore, the server can use the fusion feature processing module in the bullet screen recognition model to determine whether the target bullet screen belongs to inappropriate bullet screen based on the target text feature vector and the target video feature vector.

[0204] If the target comment is inappropriate, the server will filter it out, meaning viewers in the game's live stream will not see it. If the target comment is not inappropriate, the server can play it normally, allowing viewers in the game's live stream to see it.

[0205] In response to the data processing method described above, this application also provides a corresponding data processing apparatus so that the above data processing method can be applied and implemented in practice.

[0206] See Figure 14 , Figure 14 This is consistent with the above text Figure 2 The diagram shows the structure of a data processing device 1400 corresponding to the data processing method illustrated. Figure 14 As shown, the data processing apparatus 1400 includes:

[0207] Data acquisition unit 1401 is used to acquire the target text of the target barrage and the target video segment corresponding to the target barrage for the target barrage to be identified;

[0208] The text feature extraction unit 1402 is used to perform feature extraction processing on the target text through the text processing module in the bullet screen recognition model to obtain the target text feature vector;

[0209] The video feature extraction unit 1403 is used to perform feature extraction processing on the target video segment to obtain a reference video feature vector through the video processing module in the bullet screen recognition model, perform feature extraction processing on the video image captured from the target video segment to obtain a reference image feature vector, and generate a target video feature vector based on the reference video feature vector and the reference image feature vector.

[0210] The bullet screen recognition unit 1404 is used to determine the recognition result corresponding to the target bullet screen based on the target text feature vector and the target video feature vector through the fusion feature processing module in the bullet screen recognition model; the recognition result corresponding to the target bullet screen is used to characterize the type of the target bullet screen.

[0211] Optional, in Figure 14 Based on the data processing device shown, the video feature extraction unit 1403 is specifically used for:

[0212] The target video segment is processed by the video feature extraction structure in the video processing module to obtain a reference video feature vector.

[0213] The image feature extraction structure in the video processing module is used to perform feature extraction processing on multiple frames of video images acquired from the target video segment, so as to obtain the reference image feature vectors corresponding to each of the multiple frames of video images.

[0214] The target video feature vector is obtained by fusing the reference video feature vector and the reference image feature vectors corresponding to each of the multiple video frames through the feature fusion layer in the video processing module.

[0215] Optional, in Figure 14 Based on the data processing device shown, the video feature extraction unit 1403 is specifically used for:

[0216] The n convolutional neural networks in the image feature extraction structure are used to perform feature extraction processing on the n video images captured from the target video segment, respectively, to obtain the primary image feature vectors corresponding to each of the n video images; where n is an integer greater than 1.

[0217] The reference image feature vector is determined by the recurrent neural network in the image feature extraction structure based on the reference video feature vector and the primary image feature vectors corresponding to the n video images.

[0218] Optional, in Figure 14 Based on the data processing device shown, the video feature extraction unit 1403 is specifically used for:

[0219] The recurrent neural network is used to determine the hidden layer vector corresponding to the i-th video frame; where i is an integer greater than or equal to 1 and less than or equal to n; when i equals 1, the recurrent neural network is used to determine the hidden layer vector corresponding to the i-th video frame based on the reference video feature vector and the primary image feature vector corresponding to the i-th video frame; when i is not equal to 1, the recurrent neural network is used to determine the hidden layer vector corresponding to the (i-1)-th video frame and the primary image feature vector corresponding to the i-th video frame.

[0220] The hidden layer vector corresponding to the nth frame of the video image is used as the feature vector of the reference image.

[0221] Optional, in Figure 14 Based on the data processing device shown, the text feature extraction unit 1402 is specifically used for:

[0222] The word feature extraction structure in the text processing module is used to perform feature extraction processing on each word segment in the target text to obtain the word feature vector corresponding to each word segment.

[0223] The target text feature vector is determined by the word feature fusion structure in the text processing module based on the word feature vectors corresponding to each word segment.

[0224] Optional, in Figure 14 Based on the data processing device shown, the data acquisition unit 1401 is specifically used for:

[0225] The target video played by the user who posted the target bullet comment is divided into at least one reference video segment; the reference video segment played at the reference time point corresponding to the target bullet comment is determined as the target video segment; the reference time point corresponding to the target bullet comment includes at least one of the following: the creation time point of the target bullet comment, the sending time point of the target bullet comment;

[0226] or,

[0227] According to the preset video segment extraction rules, based on the reference time point corresponding to the target bullet comment and the preset video segment extraction duration, a video segment is extracted from the target video and used as the target video segment.

[0228] Optional, in Figure 14 Based on the data processing device shown, see Figure 15 , Figure 15 This is a schematic diagram of another data processing apparatus 1500 provided in an embodiment of this application. (See attached diagram.) Figure 15 As shown, the data processing device further includes: a model training unit 1501; the model training unit 1501 is used for:

[0229] Obtain the first training sample; the first training sample includes the first training text of the training bullet comments, the first training video segment corresponding to the training bullet comments, and the annotation result corresponding to the training bullet comments;

[0230] The first training text is processed by the initial text processing module in the initial bullet screen recognition model to be trained to obtain a first training text feature vector; the first training video segment is processed by the initial video processing module in the initial bullet screen recognition model to obtain a first training video feature vector.

[0231] The initial fusion feature processing module in the initial bullet screen recognition model determines the prediction result corresponding to the training bullet screen based on the first training text feature vector and the first training video feature vector.

[0232] Based on the prediction results and the annotation results, the initial bullet screen recognition model is trained; once the initial bullet screen recognition model meets the first training termination condition, the bullet screen recognition model is obtained.

[0233] Optional, in Figure 15 Based on the data processing device shown, the model training unit 1501 is further configured to pre-train the matching relationship recognition model to obtain the initial bullet screen recognition model in the following manner:

[0234] Obtain a second training sample; the second training sample includes positive training samples and negative training samples, wherein the positive training samples include a second training video segment and a second training text that have a matching relationship; and the negative training samples include a second training video segment and a second training text that do not have a matching relationship.

[0235] The initial text processing module in the matching relationship recognition model performs feature extraction processing on the second training text in the target second training sample to obtain the second training text feature vector; the initial video processing module in the matching relationship recognition model performs feature extraction processing on the second training video segment in the target second training sample to obtain the second training video feature vector.

[0236] The initial fusion feature processing module in the matching relationship recognition model determines the predicted matching relationship between the second training text and the second training video segment in the target second training sample based on the second training text feature vector and the second training video feature vector.

[0237] Based on the type of the target second training sample and the matching relationship prediction result, the matching relationship recognition model is trained; once the matching relationship recognition model meets the second training termination condition, the initial bullet screen recognition model is obtained.

[0238] Optional, in Figure 15 Based on the data processing device shown, the model training unit 1501 is further used to pre-train the text classification model to obtain the initial text processing module in the following manner:

[0239] Obtain a third training sample; the third training sample includes a third training text and the corresponding labeled text category of the third training text;

[0240] The predicted text category corresponding to the third training text is determined using the text classification model.

[0241] Based on the predicted text category and the labeled text category, the text classification model is trained; once the text classification model meets the third training termination condition, the structure of the text classification model other than the classification layer is used as the initial text processing module.

[0242] Optional, in Figure 15Based on the data processing device shown, the model training unit 1501 is further configured to pre-train the video image reconstruction model to obtain the initial video processing module in the following manner:

[0243] Obtain a fourth training sample; the fourth training sample includes the third training video segment.

[0244] The third training video segment is processed by feature extraction using the initial video feature extraction structure in the video image reconstruction model to obtain the third training video feature vector.

[0245] The initial image feature extraction structure in the video image reconstruction model is used to perform feature extraction processing on multiple training video images collected from the third training video segment to obtain the training image feature vectors corresponding to each of the multiple training video images; and the training hidden layer vectors corresponding to each of the multiple training video images are determined based on the third training video feature vectors and the training image feature vectors corresponding to each of the multiple training video images.

[0246] Based on the image reconstruction structure in the video image reconstruction model, the reconstructed images corresponding to each of the multi-frame training video images are generated according to the training hidden layer vectors corresponding to each of the multi-frame training video images.

[0247] The video image reconstruction model is trained based on the multi-frame training video images and the corresponding reconstructed images of each multi-frame training video image; once the video image reconstruction model meets the fourth training termination condition, the initial video processing module is determined based on the initial video feature extraction structure and the initial image feature extraction structure in the image reconstruction model.

[0248] The data processing apparatus provided in this application takes into account the high reference value of the video information corresponding to the bullet comments when determining whether they are inappropriate. Based on this, when identifying whether a target bullet comment is inappropriate using the bullet comment recognition model, it comprehensively considers both the text information of the target bullet comment itself and the visual information of the target video segment corresponding to the target bullet comment. The visual information of the target video segment is used to assist in identifying whether the target bullet comment is inappropriate. This improves the accuracy of inappropriate bullet comment recognition, avoids missed or misidentified inappropriate bullet comments, and thus helps to effectively filter inappropriate bullet comments, preventing them from being displayed to video viewers and improving the user's video viewing experience.

[0249] This application also provides a device for recognizing bullet comments. This device may specifically be a terminal device or a server. The terminal device and server provided in this application will be described below from the perspective of hardware implementation.

[0250] See Figure 16 , Figure 16 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. For example... Figure 16 As shown, for ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. The terminal can be any terminal device including mobile phones, tablets, personal digital assistants (PDAs), point-of-sales terminals (POS), in-vehicle computers, etc. Taking a computer as an example:

[0251] Figure 16 This is a block diagram illustrating a portion of the structure of a computer associated with the terminal provided in an embodiment of this application. (Reference) Figure 16 The computer includes: a radio frequency (RF) circuit 1610, a memory 1620, an input unit 1630 (including a touch panel 1631 and other input devices 1632), a display unit 1640 (including a display panel 1641), a sensor 1650, an audio circuit 1660 (which can connect to a speaker 1661 and a microphone 1662), a wireless fidelity (WiFi) module 1670, a processor 1680, and a power supply 1690, etc. Those skilled in the art will understand that... Figure 16 The computer architecture shown does not constitute a limitation on the computer and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0252] The memory 1620 can be used to store software programs and modules. The processor 1680 executes various computer functions and data processing by running the software programs and modules stored in the memory 1620. The memory 1620 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer (such as audio data, telephone directory, etc.). In addition, the memory 1620 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0253] The processor 1680 is the control center of the computer, connecting various parts of the computer through various interfaces and lines. It performs various computer functions and processes data by running or executing software programs and / or modules stored in the memory 1620, and by calling data stored in the memory 1620. Optionally, the processor 1680 may include one or more processing units; preferably, the processor 1680 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 1680.

[0254] In this embodiment of the application, the processor 1680 included in the terminal also has the following functions:

[0255] For the target bullet screen to be identified, obtain the target text of the target bullet screen and the target video segment corresponding to the target bullet screen;

[0256] The text processing module in the bullet screen recognition model performs feature extraction processing on the target text to obtain the target text feature vector;

[0257] The video processing module in the bullet screen recognition model performs feature extraction processing on the target video segment to obtain a reference video feature vector, performs feature extraction processing on the video image captured from the target video segment to obtain a reference image feature vector, and generates a target video feature vector based on the reference video feature vector and the reference image feature vector.

[0258] The fusion feature processing module in the bullet screen recognition model determines the recognition result corresponding to the target bullet screen based on the target text feature vector and the target video feature vector; the recognition result corresponding to the target bullet screen is used to characterize the type of the target bullet screen.

[0259] Optionally, the processor 1680 is further configured to execute steps of any implementation of the data processing method provided in the embodiments of this application.

[0260] See Figure 17 , Figure 17This is a schematic diagram of the structure of a server 1700 provided in an embodiment of this application. The server 1700 can vary significantly due to different configurations or performance, and may include one or more central processing units (CPUs) 1722 (e.g., one or more processors) and memory 1732, and one or more storage media 1730 (e.g., one or more mass storage devices) for storing application programs 1742 or data 1744. The memory 1732 and storage media 1730 can be temporary or persistent storage. The program stored in the storage media 1730 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 1722 may be configured to communicate with the storage media 1730 and execute the series of instruction operations in the storage media 1730 on the server 1700.

[0261] Server 1700 may also include one or more power supplies 1726, one or more wired or wireless network interfaces 1750, one or more input / output interfaces 1758, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0262] The steps performed by the server in the above embodiments can be based on this Figure 17 The server structure shown.

[0263] The CPU 1722 is used to perform the following steps:

[0264] For the target bullet screen to be identified, obtain the target text of the target bullet screen and the target video segment corresponding to the target bullet screen;

[0265] The text processing module in the bullet screen recognition model performs feature extraction processing on the target text to obtain the target text feature vector;

[0266] The video processing module in the bullet screen recognition model performs feature extraction processing on the target video segment to obtain a reference video feature vector, performs feature extraction processing on the video image captured from the target video segment to obtain a reference image feature vector, and generates a target video feature vector based on the reference video feature vector and the reference image feature vector.

[0267] The fusion feature processing module in the bullet screen recognition model determines the recognition result corresponding to the target bullet screen based on the target text feature vector and the target video feature vector; the recognition result corresponding to the target bullet screen is used to characterize the type of the target bullet screen.

[0268] Optionally, the CPU 1722 can also be used to execute any implementation of the data processing method provided in the embodiments of this application.

[0269] This application also provides a computer-readable storage medium for storing a computer program that executes any one of the data processing methods described in the foregoing embodiments.

[0270] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the implementation methods of the data processing method described in the foregoing embodiments.

[0271] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0272] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0273] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0274] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0275] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing computer programs.

[0276] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0277] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method, characterized in that, The method includes: For the target bullet screen to be identified, obtain the target text of the target bullet screen and the target video segment corresponding to the target bullet screen; The text processing module in the bullet screen recognition model performs feature extraction processing on the target text to obtain the target text feature vector; The target video segment is processed by the video feature extraction structure in the video processing module of the bullet screen recognition model to obtain a reference video feature vector; the video feature extraction structure includes a three-dimensional convolutional neural network and two fully connected layers. The n convolutional neural networks in the image feature extraction structure of the video processing module are used to perform feature extraction processing on the n frames of video images acquired from the target video segment, respectively, to obtain the primary image feature vectors corresponding to each of the n frames of video images; the image feature extraction structure includes n convolutional neural networks and recurrent neural networks; where n is an integer greater than 1; The reference image feature vector is determined by the recurrent neural network in the image feature extraction structure based on the reference video feature vector and the primary image feature vectors corresponding to the n frames of video images. The feature fusion layer in the video processing module fuses the reference video feature vector and the reference image feature vectors corresponding to each of the multiple video frames to obtain the target video feature vector; the feature fusion layer includes a fully connected layer, which is used to merge and splice the reference video feature vector and the feature vectors of the multiple images. The fusion feature processing module in the bullet screen recognition model determines the recognition result corresponding to the target bullet screen based on the target text feature vector and the target video feature vector. The recognition result corresponding to the target bullet screen is used to characterize the type of the target bullet screen. The fusion feature processing module includes a first fully connected layer, a convolutional layer, a pooling layer, and a second fully connected layer. The first fully connected layer in the fusion feature processing module performs a primary fusion process on the target text feature vector and the target video feature vector to obtain a primary fusion feature vector. The convolutional layer, pooling layer, and second fully connected layer in the fusion feature processing module process the primary fusion feature vector step by step to output the recognition result corresponding to the target bullet screen.

2. The method according to claim 1, characterized in that, The step of determining the reference image feature vector using the recurrent neural network in the image feature extraction structure, based on the reference video feature vector and the primary image feature vectors corresponding to each of the n video frames, includes: The recurrent neural network is used to determine the hidden layer vector corresponding to the i-th video frame; where i is an integer greater than or equal to 1 and less than or equal to n; when i equals 1, the recurrent neural network is used to determine the hidden layer vector corresponding to the i-th video frame based on the reference video feature vector and the primary image feature vector corresponding to the i-th video frame; when i is not equal to 1, the recurrent neural network is used to determine the hidden layer vector corresponding to the (i-1)-th video frame and the primary image feature vector corresponding to the i-th video frame. The hidden layer vector corresponding to the nth frame of the video image is used as the feature vector of the reference image.

3. The method according to claim 1, characterized in that, The text processing module in the bullet screen recognition model performs feature extraction processing on the target text to obtain a target text feature vector, including: The word feature extraction structure in the text processing module is used to perform feature extraction processing on each word segment in the target text to obtain the word feature vector corresponding to each word segment. The target text feature vector is determined by the word feature fusion structure in the text processing module based on the word feature vectors corresponding to each word segment.

4. The method according to claim 1, characterized in that, The step of obtaining the target video segment corresponding to the target bullet comment includes: The target video played by the user who posted the target bullet comment is divided into at least one reference video segment; the reference video segment played at the reference time point corresponding to the target bullet comment is determined as the target video segment; the reference time point corresponding to the target bullet comment includes at least one of the following: the creation time point of the target bullet comment, the sending time point of the target bullet comment; or, According to the preset video segment extraction rules, based on the reference time point corresponding to the target bullet comment and the preset video segment extraction duration, a video segment is extracted from the target video and used as the target video segment.

5. The method according to claim 1, characterized in that, The bullet screen recognition model is trained using the following method: Obtain the first training sample; the first training sample includes the first training text of the training bullet comments, the first training video segment corresponding to the training bullet comments, and the annotation result corresponding to the training bullet comments; The first training text is processed by the initial text processing module in the initial bullet screen recognition model to be trained to obtain a first training text feature vector; the first training video segment is processed by the initial video processing module in the initial bullet screen recognition model to obtain a first training video feature vector. The initial fusion feature processing module in the initial bullet screen recognition model determines the prediction result corresponding to the training bullet screen based on the first training text feature vector and the first training video feature vector. Based on the prediction results and the annotation results, the initial bullet screen recognition model is trained; Once the initial bullet screen recognition model meets the first training termination condition, the bullet screen recognition model is obtained.

6. The method according to claim 5, characterized in that, The initial bullet screen recognition model was obtained by pre-training the matching relationship recognition model in the following way: Obtain a second training sample; the second training sample includes positive training samples and negative training samples, wherein the positive training samples include a second training video segment and a second training text that have a matching relationship; and the negative training samples include a second training video segment and a second training text that do not have a matching relationship. The initial text processing module in the matching relationship recognition model performs feature extraction processing on the second training text in the target second training sample to obtain the second training text feature vector; the initial video processing module in the matching relationship recognition model performs feature extraction processing on the second training video segment in the target second training sample to obtain the second training video feature vector. The initial fusion feature processing module in the matching relationship recognition model determines the predicted matching relationship between the second training text and the second training video segment in the target second training sample based on the second training text feature vector and the second training video feature vector. The matching relationship recognition model is trained based on the type of the target second training sample and the matching relationship prediction result; Once the matching relationship recognition model meets the second training termination condition, the initial bullet screen recognition model is obtained.

7. The method according to claim 5 or 6, characterized in that, The initial text processing module is obtained by pre-training the text classification model in the following way: Obtain a third training sample; the third training sample includes a third training text and the corresponding labeled text category of the third training text; The predicted text category corresponding to the third training text is determined using the text classification model. The text classification model is trained based on the predicted text category and the labeled text category; Once the text classification model meets the third training termination condition, the structure of the text classification model other than the classification layer is used as the initial text processing module.

8. The method according to claim 5 or 6, characterized in that, The initial video processing module is obtained by pre-training the video image reconstruction model in the following way: Obtain a fourth training sample; the fourth training sample includes the third training video segment. The third training video segment is processed by feature extraction using the initial video feature extraction structure in the video image reconstruction model to obtain the third training video feature vector. The initial image feature extraction structure in the video image reconstruction model is used to perform feature extraction processing on multiple training video images collected from the third training video segment to obtain the training image feature vectors corresponding to each of the multiple training video images; and the training hidden layer vectors corresponding to each of the multiple training video images are determined based on the third training video feature vectors and the training image feature vectors corresponding to each of the multiple training video images. Based on the image reconstruction structure in the video image reconstruction model, the reconstructed images corresponding to each of the multi-frame training video images are generated according to the training hidden layer vectors corresponding to each of the multi-frame training video images. The video image reconstruction model is trained based on the multi-frame training video images and the corresponding reconstructed images of each multi-frame training video image; once the video image reconstruction model meets the fourth training termination condition, the initial video processing module is determined based on the initial video feature extraction structure and the initial image feature extraction structure in the image reconstruction model.

9. A data processing apparatus, characterized in that, The device includes: The data acquisition unit is used to acquire the target text of the target bullet screen and the target video segment corresponding to the target bullet screen for the target bullet screen to be identified; The text feature extraction unit is used to perform feature extraction processing on the target text through the text processing module in the bullet screen recognition model to obtain the target text feature vector; The video feature extraction unit is used to perform feature extraction processing on the target video segment to obtain a reference video feature vector through the video processing module in the bullet screen recognition model, perform feature extraction processing on the video image captured from the target video segment to obtain a reference image feature vector, and generate a target video feature vector based on the reference video feature vector and the reference image feature vector. The bullet screen recognition unit is used to determine the recognition result corresponding to the target bullet screen based on the target text feature vector and the target video feature vector through the fusion feature processing module in the bullet screen recognition model. The recognition result corresponding to the target bullet screen is used to characterize the type of the target bullet screen. The fusion feature processing module includes a first fully connected layer, a convolutional layer, a pooling layer, and a second fully connected layer. The first fully connected layer in the fusion feature processing module performs a primary fusion processing on the target text feature vector and the target video feature vector to obtain a primary fusion feature vector. The convolutional layer, pooling layer, and second fully connected layer in the fusion feature processing module process the primary fusion feature vector step by step to output the recognition result corresponding to the target bullet screen. The video feature extraction unit is specifically used for: The target video segment is processed by the video feature extraction structure in the video processing module to obtain a reference video feature vector; the video feature extraction structure includes a three-dimensional convolutional neural network and two fully connected layers. The image feature extraction structure in the video processing module performs feature extraction processing on multiple frames of video images acquired from the target video segment to obtain reference image feature vectors corresponding to each of the multiple frames of video images; the image feature extraction structure includes n convolutional neural networks and recurrent neural networks, where n is greater than 1; The feature fusion layer in the video processing module fuses the reference video feature vector and the reference image feature vectors corresponding to each of the multiple video frames to obtain the target video feature vector; the feature fusion layer includes a fully connected layer, which is used to merge and splice the reference video feature vector and the feature vectors of the multiple images. The video feature extraction unit is specifically used for: The n convolutional neural networks in the image feature extraction structure are used to perform feature extraction processing on the n frames of video images captured from the target video segment, respectively, to obtain the primary image feature vectors corresponding to each of the n frames of video images; where n is an integer greater than 1. The reference image feature vector is determined by the recurrent neural network in the image feature extraction structure based on the reference video feature vector and the primary image feature vectors corresponding to the n frames of video images.

10. The apparatus according to claim 9, characterized in that, The video feature extraction unit is specifically used for: The recurrent neural network is used to determine the hidden layer vector corresponding to the i-th video frame; where i is an integer greater than or equal to 1 and less than or equal to n; when i equals 1, the recurrent neural network is used to determine the hidden layer vector corresponding to the i-th video frame based on the reference video feature vector and the primary image feature vector corresponding to the i-th video frame; when i is not equal to 1, the recurrent neural network is used to determine the hidden layer vector corresponding to the (i-1)-th video frame and the primary image feature vector corresponding to the i-th video frame. The hidden layer vector corresponding to the nth frame of the video image is used as the feature vector of the reference image.

11. The apparatus according to claim 9, characterized in that, The text feature extraction unit is specifically used for: The word feature extraction structure in the text processing module is used to perform feature extraction processing on each word segment in the target text to obtain the word feature vector corresponding to each word segment. The target text feature vector is determined by the word feature fusion structure in the text processing module based on the word feature vectors corresponding to each word segment.

12. The apparatus according to claim 9, characterized in that, The data acquisition unit is specifically used for: The target video played by the user who posted the target bullet comment is divided into at least one reference video segment; the reference video segment played at the reference time point corresponding to the target bullet comment is determined as the target video segment; The reference time point corresponding to the target bullet screen includes at least one of the following: the creation time point of the target bullet screen, and the sending time point of the target bullet screen; or, According to the preset video segment extraction rules, based on the reference time point corresponding to the target bullet comment and the preset video segment extraction duration, a video segment is extracted from the target video and used as the target video segment.

13. The apparatus according to claim 9, characterized in that, The device further includes: a model training unit; the model training unit is used for: Obtain the first training sample; the first training sample includes the first training text of the training bullet comments, the first training video segment corresponding to the training bullet comments, and the annotation result corresponding to the training bullet comments; The first training text is processed by the initial text processing module in the initial bullet screen recognition model to be trained to obtain a first training text feature vector; the first training video segment is processed by the initial video processing module in the initial bullet screen recognition model to obtain a first training video feature vector. The initial fusion feature processing module in the initial bullet screen recognition model determines the prediction result corresponding to the training bullet screen based on the first training text feature vector and the first training video feature vector. Based on the prediction results and the annotation results, the initial bullet screen recognition model is trained; once the initial bullet screen recognition model meets the first training termination condition, the bullet screen recognition model is obtained.

14. The apparatus according to claim 13, characterized in that, The model training unit is also used to pre-train the matching relationship recognition model to obtain the initial bullet screen recognition model in the following manner: Obtain a second training sample; the second training sample includes positive training samples and negative training samples, wherein the positive training samples include a second training video segment and a second training text that have a matching relationship; and the negative training samples include a second training video segment and a second training text that do not have a matching relationship. The initial text processing module in the matching relationship recognition model performs feature extraction processing on the second training text in the target second training sample to obtain the second training text feature vector; the initial video processing module in the matching relationship recognition model performs feature extraction processing on the second training video segment in the target second training sample to obtain the second training video feature vector. The initial fusion feature processing module in the matching relationship recognition model determines the predicted matching relationship between the second training text and the second training video segment in the target second training sample based on the second training text feature vector and the second training video feature vector. The matching relationship recognition model is trained based on the type of the target second training sample and the matching relationship prediction result; Once the matching relationship recognition model meets the second training termination condition, the initial bullet screen recognition model is obtained.

15. The apparatus according to claim 13 or 14, characterized in that, The model training unit is also used to pre-train the text classification model to obtain the initial text processing module in the following manner: Obtain a third training sample; the third training sample includes a third training text and the corresponding labeled text category of the third training text; The predicted text category corresponding to the third training text is determined using the text classification model. The text classification model is trained based on the predicted text category and the labeled text category; Once the text classification model meets the third training termination condition, the structure of the text classification model other than the classification layer is used as the initial text processing module.

16. The apparatus according to claim 13 or 14, characterized in that, The model training unit is also used to pre-train the video image reconstruction model to obtain the initial video processing module in the following manner: Obtain a fourth training sample; the fourth training sample includes the third training video segment. The third training video segment is processed by feature extraction using the initial video feature extraction structure in the video image reconstruction model to obtain the third training video feature vector. The initial image feature extraction structure in the video image reconstruction model is used to perform feature extraction processing on multiple training video images collected from the third training video segment to obtain the training image feature vectors corresponding to each of the multiple training video images; and the training hidden layer vectors corresponding to each of the multiple training video images are determined based on the third training video feature vectors and the training image feature vectors corresponding to each of the multiple training video images. Based on the image reconstruction structure in the video image reconstruction model, the reconstructed images corresponding to each of the multi-frame training video images are generated according to the training hidden layer vectors corresponding to each of the multi-frame training video images. The video image reconstruction model is trained based on the multi-frame training video images and the corresponding reconstructed images of each multi-frame training video image; once the video image reconstruction model meets the fourth training termination condition, the initial video processing module is determined based on the initial video feature extraction structure and the initial image feature extraction structure in the image reconstruction model.

17. A device, characterized in that, The device includes a processor and a memory; The memory is used to store computer programs; The processor is configured to execute the data processing method according to any one of claims 1 to 8 according to the computer program.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the data processing method according to any one of claims 1 to 8.

19. A computer program product, characterized in that, The computer program product includes instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 8.

Images