Video recognition model training method, video recognition method, and related device

Abstract

This disclosure relates to a training method for a video recognition model, a video recognition method, and related equipment. The method includes: acquiring a video sample set, wherein the video samples in the video sample set include display data, audio data, and correlation annotation data between the display data and the audio data; extracting display text data from the display data; performing speech recognition on the audio data to obtain playback text data; inputting the display text data and the playback text data into a first network structure of the video recognition model to obtain a text relation vector; inputting the text relation vector into a second network structure of the video recognition model to obtain correlation prediction data; and training the network parameters corresponding to the second network structure based on the target loss determined by the correlation prediction data and the correlation annotation data to obtain a video recognition model that meets preset conditions. The obtained trained video recognition model can quickly and accurately recognize videos.

Description

Technical Field

[0001] This disclosure relates to the field of Internet technology, and in particular to training methods for video recognition models, video recognition methods, and related equipment. Background Technology

[0002] With the development of internet technology and the widespread use of media applications, various videos permeate people's daily lives. Users decide whether to click to play a video based on its title. Therefore, the video title has a significant impact on the video's click-through rate. In order to obtain more clicks, video uploaders often deliberately use a popular or controversial topic unrelated to the actual content of the video as the title to lure users into clicking and playing. After clicking, users find that the actual video content does not match the title, resulting in a poor user experience.

[0003] In related technologies, deep structural semantic models are used to identify videos and determine whether the actual content of the video matches the title. However, deep structural semantic models are a typical dual-tower structure, which makes it difficult to capture effective information between the actual content of the video and the title, resulting in low accuracy of the identification results. Summary of the Invention

[0004] This disclosure provides a training method for a video recognition model, a video recognition method, and related equipment, to at least solve the problem of low accuracy in video recognition results when using deep structural semantic models in related technologies. The technical solution of this disclosure is as follows:

[0005] According to a first aspect of the present disclosure, a method for training a video recognition model is provided, comprising:

[0006] Obtain a video sample set, wherein the video samples in the video sample set include display data, audio data, and correlation annotation data between the display data and the audio data;

[0007] Extract the display text data from the display data;

[0008] Perform speech recognition on the audio data to obtain the playback text data;

[0009] The displayed text data and the played text data are input into the first network structure of the video recognition model. The first network structure is used to capture the dependency relationship between the displayed text data and the played text data and obtain a text relationship vector.

[0010] The text relationship vector is input into the second network structure of the video recognition model to obtain relevance prediction data;

[0011] Based on the target loss determined by the correlation prediction data and the correlation annotation data, the network parameters corresponding to the second network structure are trained to obtain a video recognition model that meets preset conditions.

[0012] Optionally, if the display data includes a cover image and descriptive text data, then extracting the display text data from the display data includes:

[0013] The cover image is identified to obtain the cover text data;

[0014] Extract the first sub-text data from the cover text data;

[0015] Extract the second sub-text data from the descriptive text data;

[0016] Based on the first sub-text data and the second sub-text data, the text data to be displayed is determined.

[0017] Optionally, extracting the second sub-text data from the descriptive text data includes:

[0018] Extract the first valid text data after the first preset character from the description text data;

[0019] The descriptive text data is compared with the data in the first preset list to determine the second valid text data;

[0020] Filter the first valid text data into the second valid text data to determine the second sub-text data;

[0021] The step of determining the display text data based on the first sub-text data and the second sub-text data includes:

[0022] Based on the first sub-text data, the second sub-text data, and the first valid text data, the text data to be displayed is determined.

[0023] Optionally, extracting the first sub-text data from the cover text data includes:

[0024] Extract the third valid text data after the second preset character from the cover text data;

[0025] The cover text data is compared with the data in the second preset list to determine the fourth valid text data;

[0026] Filter the third valid text data from the fourth valid text data to determine the first sub-text data;

[0027] The step of determining the display text data based on the first sub-text data, the second sub-text data, and the first valid text data includes:

[0028] Based on the first sub-text data, the second sub-text data, the first valid text data, and the third valid text data, the text data to be displayed is determined.

[0029] Optionally, determining the display text data based on the first sub-text data, the second sub-text data, and the first valid text data includes:

[0030] Determine the similarity value between the first sub-text data and the second sub-text data;

[0031] If the similarity value is greater than a preset threshold, then the first sub-text data is determined as the basic text data, or the second sub-text data is determined as the basic text data; or the basic text data is determined based on the number of characters corresponding to the first sub-text data and the second sub-text data respectively.

[0032] If the similarity value is less than the preset threshold, the concatenation result of the first sub-text data and the second sub-text data is determined as the basic text data;

[0033] The basic text data and the first valid text data are concatenated to determine the text data to be displayed.

[0034] Optionally, the acquisition of the video sample set includes:

[0035] Retrieve video samples from the video sample set;

[0036] If the video sample does not have a cover image, then the cover image of the video sample without a cover image is extracted.

[0037] Optionally, the step of performing speech recognition on the audio data to obtain playback text data includes:

[0038] The audio data is segmented to obtain at least two audio segment data;

[0039] Extract the target sound data from each of the at least two audio segment data;

[0040] Speech recognition is performed on the target sound data to obtain the playback text data.

[0041] Optionally, the second network structure includes at least two one-dimensional convolutional layers, at least two pooling layers, and a regression layer; wherein the kernel sizes of the at least two one-dimensional convolutional layers are different; then, inputting the text relation vector into the second network structure of the video recognition model to obtain relevance prediction data includes:

[0042] The text relation vectors are respectively input into the at least two one-dimensional convolutional layers to obtain at least two feature extraction results;

[0043] The at least two feature extraction results are respectively input into at least two pooling layers to obtain at least two pooling results;

[0044] The at least two pooling results are input into the regression layer, and the correlation prediction data is obtained through the concatenation, linear mapping and regression processing of the regression layer.

[0045] According to a second aspect of the present disclosure, a video recognition method is provided, comprising:

[0046] Acquire the video to be identified, which includes display data and audio data;

[0047] Extract the display text data from the display data;

[0048] Perform speech recognition on the audio data to obtain the playback text data;

[0049] The displayed text data and the played text data are input into the video recognition model to determine the video recognition result;

[0050] The video recognition model is trained according to the training method of the video recognition model described in the first aspect.

[0051] According to a third aspect of the present disclosure, a training apparatus for a video recognition model is provided, comprising:

[0052] The sample set acquisition module is configured to acquire a video sample set, wherein the video samples in the video sample set include display data, audio data, and correlation annotation data between the display data and the audio data.

[0053] The text data extraction module is configured to extract the display text data from the display data;

[0054] The speech recognition processing module is configured to perform speech recognition on the audio data and obtain the playback text data;

[0055] The text input processing module is configured to input the displayed text data and the played text data into the first network structure of the video recognition model. The first network structure is used to capture the dependency relationship between the displayed text data and the played text data and obtain a text relationship vector.

[0056] The prediction data acquisition module is configured to input the text relationship vector into the second network structure of the video recognition model to acquire correlation prediction data.

[0057] The parameter training processing module is configured to train the network parameters corresponding to the second network structure based on the target loss determined by the correlation prediction data and the correlation annotation data, so as to obtain a video recognition model that meets preset conditions.

[0058] Optionally, if the displayed data includes a cover image and descriptive text data, then the text data extraction module includes:

[0059] The cover image recognition unit is configured to recognize the cover image and obtain cover text data;

[0060] The first extraction processing unit is configured to extract the first sub-text data from the cover text data;

[0061] The second extraction processing unit is configured to extract the second sub-text data from the descriptive text data;

[0062] The text data acquisition unit is configured to determine the text data to be displayed based on the first sub-text data and the second sub-text data.

[0063] Optionally, the second extraction processing unit includes:

[0064] The first extraction subunit is configured to extract the first valid text data after extracting the first preset character from the description text data;

[0065] The first determining subunit is configured to compare the descriptive text data with data in a first preset list to determine the second valid text data;

[0066] The second determining subunit is configured to filter the first valid text data in the second valid text data to determine the second subtext data;

[0067] The text data acquisition unit is then configured to determine the text data to be displayed based on the first sub-text data, the second sub-text data, and the first valid text data.

[0068] Optionally, the first extraction processing unit includes:

[0069] The second extraction subunit is configured to extract the third valid text data after the second preset character from the cover text data;

[0070] The third determining subunit is configured to compare the cover text data with the data in the second preset list to determine the fourth valid text data;

[0071] The fourth determining subunit is configured to filter the third valid text data in the fourth valid text data to determine the first subtext data;

[0072] The text data acquisition unit is configured to determine the text data to be displayed based on the first sub-text data, the second sub-text data, the first valid text data, and the third valid text data.

[0073] Optionally, the text data acquisition unit includes:

[0074] The similarity value determination subunit is configured to determine the similarity value between the first sub-text data and the second sub-text data;

[0075] The first judgment subunit is configured to determine the first sub-text data as basic text data, or the second sub-text data as basic text data, or to determine basic text data based on the number of characters corresponding to the first sub-text data and the second sub-text data respectively, if the similarity value is greater than a preset threshold.

[0076] The second judgment subunit is configured to determine the concatenation result of the first sub-text data and the second sub-text data as the basic text data if the similarity value is less than the preset threshold.

[0077] The fifth determining subunit is configured to concatenate the basic text data and the first valid text data to determine the text data to be displayed.

[0078] Optionally, the sample set acquisition module includes:

[0079] The sample set acquisition unit is configured to acquire video samples from the video sample set;

[0080] The cover image determination unit is configured to extract an image from the video sample that does not have a cover if the video sample does not have a cover, and determine the cover image of the video sample.

[0081] Optionally, the speech recognition processing module includes:

[0082] An audio segmentation processing unit is configured to segment the audio data to obtain at least two audio segment data.

[0083] The sound extraction processing unit is configured to extract target sound data from the at least two audio segment data respectively;

[0084] The speech recognition processing unit is configured to perform speech recognition on the target sound data and obtain the playback text data.

[0085] Optionally, the second network structure includes at least two one-dimensional convolutional layers, at least two pooling layers, and a regression layer; wherein the kernel sizes of the at least two one-dimensional convolutional layers are different; then the prediction data acquisition module includes:

[0086] The first feature extraction unit is configured to input the text relationship vector into the at least two one-dimensional convolutional layers respectively to obtain at least two feature extraction results;

[0087] The second feature extraction unit is configured to input the at least two feature extraction results into at least two pooling layers respectively to obtain at least two pooling results;

[0088] The prediction data acquisition unit is configured to input the at least two pooling results into the regression layer, and obtain correlation prediction data through the concatenation, linear mapping and regression processing of the regression layer.

[0089] According to a fourth aspect of the present disclosure, a video recognition device is provided, comprising:

[0090] The video acquisition module is configured to acquire a video to be identified, the video to be identified including display data and audio data;

[0091] The data extraction module is configured to extract the display text data from the display data;

[0092] The speech recognition module is configured to perform speech recognition on the audio data and obtain the playback text data;

[0093] The result determination module is configured to input the displayed text data and the played text data into the video recognition model to determine the video recognition result;

[0094] The video recognition model is trained according to the training method of the video recognition model described in the first aspect.

[0095] According to a fifth aspect of the present disclosure, an electronic device is provided, comprising:

[0096] processor;

[0097] Memory used to store the processor's executable instructions;

[0098] The processor is configured to execute the instructions to implement the training method for the video recognition model as described in the first aspect, or the video recognition method as described in the second aspect.

[0099] According to a sixth aspect of the present disclosure, a storage medium is provided that, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform a training method for a video recognition model as described in the first aspect, or a video recognition method as described in the second aspect.

[0100] According to a seventh aspect of the present disclosure, a computer program product is provided, the computer program product including readable program code, which, when executed by a processor of an electronic device, enables the electronic device to perform a training method for a video recognition model as described in the first aspect, or a video recognition method as described in the second aspect.

[0101] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0102] This disclosure first obtains a video sample set, which includes display data, audio data, and correlation annotation data between the display data and audio data. Display data refers to the data presented to the user before the video is clicked and played, used to provide the user with a preliminary understanding of the video. Data extraction is performed on the display data to determine the display text data. The actual content of the video is usually represented by audio data; therefore, speech recognition is performed on the audio data of the video samples to determine the corresponding playback text data. The displayed text data and the played text data are input into a video recognition model, which includes a first network structure and a second network structure. The first network structure is used to capture the dependency relationship between the displayed text data and the played text data, effectively capturing grammatical, syntactic, and word order information between them. This results in a text relationship vector with rich features of the displayed text data, the played text data, and the features between the displayed and played text data. This text relationship vector is used as the input to the second network structure, which then generates relevance prediction data. Using the relevance prediction data as the predicted value and the relevance annotation data as the learning objective, a target loss is determined. Based on this target loss, the network parameters corresponding to the second network structure are trained to obtain a video recognition model that meets preset conditions. The obtained trained video recognition model can effectively capture the features between the displayed text data and the played text data, resulting in high accuracy of the recognition results. Furthermore, this video recognition model is used to identify videos before distribution, effectively filtering out videos with low correlation between display data and audio data. This avoids the distribution of such videos or reduces their exposure, minimizing the chances of users encountering them, thus addressing the problem at its root and improving user experience.

[0103] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. Attached Figure Description

[0104] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0105] Figure 1 This is a flowchart illustrating a training method for a video recognition model according to an exemplary embodiment. Figure 1 ;

[0106] Figure 2 This is a flowchart illustrating a method for extracting display text data during the training of a video recognition model, according to an exemplary embodiment.

[0107] Figure 3 This is an example image of the cover image and descriptive text in this disclosure;

[0108] Figure 4 This is a flowchart illustrating a training method for a video recognition model according to an exemplary embodiment. Figure 2 ;

[0109] Figure 5 This is a flowchart illustrating a video recognition method according to an exemplary embodiment;

[0110] Figure 6 This is a block diagram illustrating a training apparatus for a video recognition model according to an exemplary embodiment;

[0111] Figure 7 This is a block diagram illustrating a video recognition device according to an exemplary embodiment;

[0112] Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment. Detailed Implementation

[0113] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0114] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0115] Figure 1 This is a flowchart illustrating a training method for a video recognition model according to an exemplary embodiment, including the following steps.

[0116] In step S11, a video sample set is obtained, wherein the video samples in the video sample set include display data, audio data, and correlation annotation data between the display data and the audio data.

[0117] In one embodiment, each video sample includes display data, audio data, and correlation annotation data between the display data and audio data. Display data refers to the data presented to the user before the video is played, providing the user with an initial understanding of the video. Specifically, it includes a cover image and descriptive text, where the descriptive text is a textual description of the video content written by the uploader. Various types of videos often rely on audio data to convey content to viewers, thus the audio data reflects the actual content of the video. Correlation annotation data refers to the actual data corresponding to the correlation between the display data and audio data of the video sample. This can be determined manually, with a range of 0 to 1. A value closer to 0 indicates a weaker correlation between the display data and audio data, meaning the display data and audio data are inconsistent, i.e., the display data and the actual video content are inconsistent. A value closer to 1 indicates a stronger correlation between the display data and audio data, meaning the display data and audio data are consistent, i.e., the display data and the actual video content are consistent.

[0118] In step S12, the display text data is extracted from the display data.

[0119] In one embodiment, display text data refers to the text data presented to the user before the video is clicked to play. Display data includes display text data as well as other data besides display text data, such as non-text data in the cover image. Normally, display text data is a high-level summary of the video content, directly reflecting the main content of the video. Video uploaders often exaggerate or fabricate display text data to induce users to click to play the video. That is, display text data is the key data in the display data. Extracting display text data can ensure the accuracy of subsequent training results while improving data processing speed.

[0120] In step S13, speech recognition is performed on the audio data to obtain the playback text data.

[0121] In one embodiment, the text data being played is the result of speech recognition of the audio data, reflecting the content of the audio data.

[0122] In one possible implementation, the audio data is segmented to obtain at least two audio segment data; target sound data is extracted from each of the at least two audio segment data; speech recognition is performed on the target sound data to obtain playback text data. When segmenting the audio data, a preset duration can be used. Since audio data corresponding to videos is typically long, segmenting the audio data into at least two audio segment data according to the preset duration allows for simultaneous processing of at least two audio segment data, improving processing speed. Because audio information in audio data is complex, often including human voices, background music, instrumental music, and noise, to ensure the validity and accuracy of determining the playback text data, human voices are used as the target. Human voices are extracted from at least two audio segment data to determine at least two target sound data. Speech recognition is then performed on the at least two target sound data to determine at least two recognition results. These at least two recognition results are then concatenated to determine the playback text data.

[0123] In step S14, the displayed text data and the played text data are input into the first network structure of the video recognition model. The first network structure is used to capture the dependency relationship between the displayed text data and the played text data to obtain a text relationship vector.

[0124] In one embodiment, the video recognition model includes a first network structure and a second network structure. The first network structure is used to capture the dependency relationship between the displayed text data and the played text data. After the displayed text data and the played text data are input into the first network structure, the output of the first network structure is a text relationship vector, which effectively reflects the dependency relationship between the displayed text data and the played text data. Specifically, the first network structure is built based on a transformer network, which uses a self-attention mechanism. This allows the first network structure to effectively learn the features of the displayed text data, the features of the played text data, and the features between the displayed text data and the played text data, thereby obtaining a text relationship vector that carries relatively rich information.

[0125] Specifically, the first network structure can use the pre-trained BERT model to initialize parameters, or it can use modified versions of BERT, such as BERT-wwm and ALBERT. The main change of BERT-wwm compared to BERT is that it changes the mask of a single character to the mask of the entire word, making the model more capable of learning. One of the main changes of ALBERT compared to BERT is that it shares the parameters of each layer of the transformer, effectively reducing the number of model parameters.

[0126] In step S15, the text relationship vector is input into the second network structure of the video recognition model to obtain relevance prediction data.

[0127] In one embodiment, the relevance prediction data is the output of the second network structure, representing the prediction result obtained by the video recognition model. Specifically, the relevance prediction data corresponds to the relevance annotation data, and the range of the relevance prediction data is 0 to 1. The closer the prediction result is to 0, the smaller the relevance between the predicted display data and the audio data, meaning that the displayed data and the actual video content are inconsistent; the closer the prediction result is to 1, the greater the relevance between the predicted display data and the audio data, meaning that the displayed data and the actual video content are consistent.

[0128] In step S16, the network parameters corresponding to the second network structure are trained based on the target loss determined by the correlation prediction data and the correlation annotation data to obtain a video recognition model that meets the preset conditions.

[0129] In one embodiment, relevance prediction data is used as the predicted value, and relevance labeled data is used as the learning target. The relevance prediction data and relevance labeled data are substituted into a preset loss function to obtain the target loss. The target loss is then used to update the network parameters of the second network structure using an error backpropagation algorithm until a video recognition model that meets preset conditions is obtained. The preset conditions may be that the number of iterations is greater than a set value, or the prediction accuracy is greater than a set accuracy value. Specifically, the preset loss function may be: (y – y_pred) 2 , where y represents the correlation-labeled data and y_pred represents the correlation-predicted data.

[0130] This exemplary embodiment first obtains a video sample set. The video samples in the video sample set include display data, audio data, and correlation annotation data between the display data and the audio data. The display data refers to the data presented to the user before the video is clicked and played, which is used to give the user a preliminary understanding of the video. Data extraction is performed on the display data to determine the display text data. The actual content of the video is usually represented by audio data. Therefore, speech recognition is performed on the audio data of the video samples to determine the playback text data corresponding to the audio data. The displayed text data and the played text data are input into a video recognition model, which includes a first network structure and a second network structure. The first network structure is used to capture the dependency relationship between the displayed text data and the played text data, effectively capturing grammatical, syntactic, and word order information between them. This results in a text relationship vector with rich features of the displayed text data, the played text data, and the features between the displayed and played text data. This text relationship vector is used as the input to the second network structure, which then generates relevance prediction data. Using the relevance prediction data as the predicted value and the relevance annotation data as the learning objective, a target loss is determined. Based on this target loss, the network parameters corresponding to the second network structure are trained to obtain a video recognition model that meets preset conditions. The obtained trained video recognition model can effectively capture the features between the displayed text data and the played text data, resulting in high accuracy of the recognition results. Furthermore, this video recognition model is used to identify videos before distribution, effectively filtering out videos with low correlation between display data and audio data. This avoids the distribution of such videos or reduces their exposure, minimizing the chances of users encountering them, thus addressing the problem at its root and improving user experience.

[0131] like Figure 2 As shown, in an exemplary embodiment of this disclosure, the display data includes a cover image and descriptive text data. Step 12, extracting the display text data from the display data, includes:

[0132] In step 121, the cover image is identified to obtain cover text data.

[0133] In one embodiment, the cover image is subjected to OCR (Optical Character Recognition) to determine the text content in the cover image and obtain the cover text data.

[0134] In one possible implementation, video samples are obtained from a set of video samples. After obtaining the set of video samples, it is determined whether the video samples have a cover image. If the video samples do not have a cover image, image extraction is performed on the video samples without a cover image to determine the cover image of the video samples. When extracting images from the video samples without a cover image, the first frame image can be extracted, that is, the first frame image is determined as the cover image of the video samples without a cover image.

[0135] In step 122, the first sub-text data is extracted from the cover text data.

[0136] In one embodiment, data extraction is performed on the cover text data, and the valid content in the cover text data is extracted as the first sub-text data. The first sub-text data is a part of the cover text data.

[0137] In one possible implementation, extracting the first sub-text data from the cover text data using a first preset rule includes: extracting the third valid text data after a second preset character from the cover text data; comparing the cover text data with data in a second preset list to determine the fourth valid text data; and filtering the third valid text data from the fourth valid text data to determine the first sub-text data. Specifically, the second preset character can be the "#" character. The third valid text data extracted using the second preset character is the key content in the cover text data, and therefore, the third valid text data will be used as the content in the display text data. By maintaining the second preset list to filter meaningless data in the cover text data, the valid content in the cover text data, i.e., the fourth valid text data, is obtained. Furthermore, to avoid data duplication, the third valid text data is filtered from the fourth valid text data to obtain the first sub-text data. If the cover text data does not contain the second preset character or data in the second preset list, then the cover text data is the first sub-text data.

[0138] In step 123, the second sub-text data is extracted from the descriptive text data.

[0139] In one embodiment, data extraction is performed on the descriptive text data, and the valid content in the descriptive text data is extracted as the second sub-text data. The second sub-text data is a part of the descriptive text data.

[0140] In one possible implementation, the step of extracting the second sub-text data from the descriptive text data using a second preset rule includes: extracting the first valid text data after the first preset character from the descriptive text data; comparing the descriptive text data with data in a first preset list to determine the second valid text data; and filtering the first valid text data from the second valid text data to determine the second sub-text data.

[0141] Specifically, the first preset character can be the "#" character. The "#" character is often followed by topic information, which can be extracted using regular expressions to obtain the topic information after the "#" character in the descriptive text data as the first valid text data. The descriptive text data also contains some meaningless content, such as meaningless sentences like "Thank you, A official" or "I want to be trending," as well as content after the "@" character. The "@" character is usually followed by other users' account or username information, meaning it often represents social relationships. Therefore, these meaningless sentences and "@" characters are added to the first preset list. By comparing the descriptive text data with the data in the first preset list, meaningless sentences and social relationship content after the "@" character are filtered out, obtaining the valid content in the descriptive text data, i.e., the second valid text data. The first valid text data extracted using the first preset character is the key content in the descriptive text data; therefore, the first valid text data will be used as the content in the display text data. By maintaining a first preset list to filter meaningless data within the descriptive text data, the valid content within the descriptive text data is obtained, namely the second valid text data. Furthermore, to avoid data duplication, the first valid text data is filtered from the second valid text data to obtain the second sub-text data.

[0142] It should be noted that because the descriptive text data and the cover text data differ significantly—for example, the descriptive text data may contain the characters "#" and "@", while the cover text data typically contains fewer of these characters—different extraction rules can be used to extract the cover text data and the descriptive text data. Furthermore, in this exemplary embodiment, the order of steps 122 and 123 is not specifically limited; that is, step 122 can be executed first, followed by step 123; or step 123 can be executed first, followed by step 122; or steps 122 and 123 can be executed simultaneously. Similarly, the order of extracting the third valid text data and determining the fourth valid text data in step 122, and the order of extracting the first valid text data and determining the second valid text data in step 123, are not specifically limited.

[0143] In step 124, display text data is determined based on the first sub-text data and the second sub-text data.

[0144] In one embodiment, after obtaining the first sub-text data and the second sub-text data, the first sub-text data and the second sub-text data can be concatenated to determine the display text data.

[0145] In one possible implementation, if first valid text data is extracted based on a first preset character, then determining the display text data based on the first sub-text data and the second sub-text data specifically involves determining the display text data based on the first sub-text data, the second sub-text data, and the first valid text data. Specifically, determining the display text data based on the first sub-text data, the second sub-text data, and the first valid text data includes: determining the similarity value between the first sub-text data and the second sub-text data; if the similarity value is greater than a preset threshold, then determining the first sub-text data as basic text data, or determining the second sub-text data as basic text data, or determining basic text data based on the number of characters corresponding to the first sub-text data and the second sub-text data respectively; if the similarity value is less than the preset threshold, then determining the concatenation result of the first sub-text data and the second sub-text data as basic text data, and concatenating the basic text data and the first valid text data to determine the display text data.

[0146] The first sub-text data is the valid content in the cover text data, and the second sub-text data is the valid content in the description text data. Since both cover and description text data often display the actual content of the video, they may be quite similar. To determine a concise base text data, we first determine the similarity value between the first and second sub-text data and check if this similarity value is greater than a preset threshold. If the similarity value is greater than the preset threshold, the content of the first and second sub-text data is quite similar and highly overlapping. Therefore, either the first or second sub-text data can be determined as the base text data, or the one with more characters can be selected as the base text data. If the similarity value is less than the preset threshold, the overlap between the first and second sub-text data is small. Therefore, the first and second sub-text data are concatenated to determine the base text data. The first valid text data is the key data in the description text data. Therefore, after determining the base text data, the first valid text data is concatenated after the base text data to determine the display text data.

[0147] If not only is a first valid text data extracted based on a first preset character, but also a third valid text data is extracted based on a second preset character, then determining the display text data based on the first sub-text data, the second sub-text data, and the first valid text data includes: determining the display text data based on the first sub-text data, the second sub-text data, the first valid text data, and the third valid text data. Specifically, basic text data is determined based on the first and second sub-text data, and the first and third valid text data are concatenated after the basic text data to determine the display text data.

[0148] For example, such as Figure 3 As shown, OCR recognition was performed on the cover image of the short video to obtain the cover text data "B Phone, still stunning and more practical true full-screen flagship" and the descriptive text data "B Phone [Review]: still stunning true full-screen flagship #B Phone's corresponding brand #B Phone #Full-screen @A Creator Center". The first and second preset characters were both set to the "#" character. Both the first and second preset lists included the "@" character. Since the cover text data did not contain "#" or "@" characters, the first sub-text data was determined to be "B Phone, still stunning and more practical true full-screen flagship". Using the first preset list... The table removes the content after the "@" character from the descriptive text data, i.e., removes @A Creator Center, resulting in the second valid text data "B Phone [Review]: Still a Stunning True Full-Screen Flagship #B Phone's Corresponding Brand #B Phone #Full-Screen". The content after the "#" character in the descriptive text data is then extracted to obtain the first valid text data "#B Phone's Corresponding Brand #B Phone #Full-Screen". The content of the first valid text data is then filtered from the second valid text data to obtain the second sub-text data "B Phone [Review]: Still a Stunning True Full-Screen Flagship". Based on the first valid text data, the first sub-text data, and the second sub-text data, the display data is determined. The edit distance between the first and second sub-text data, i.e., the similarity value, is calculated, and a valid title is synthesized based on the similarity value to determine the display text data. For the first and second sub-text data mentioned above, if the similarity value is greater than a preset threshold, the sub-text data with more characters in the first and second sub-text data will be determined as the basic text data. The first valid text data will be placed in parentheses after the basic text data to obtain the displayed text data: "B mobile phone [review]: still a stunning true full-screen flagship (#B mobile phone corresponding brand #B mobile phone #full-screen)".

[0149] In this embodiment, the effective content in the cover text data is extracted, namely the first sub-text data, and the effective content in the description text data is extracted, namely the second sub-text data. The display text data determined based on the first and second sub-text data can reflect the key content in the display data, and the data quality is high, which helps to speed up subsequent processing.

[0150] In an exemplary embodiment of this disclosure, the second network structure includes at least two one-dimensional convolutional layers, at least two pooling layers, and a regression layer; wherein the kernel sizes of the at least two one-dimensional convolutional layers are different; then, the step of inputting the text relationship vector into the second network structure of the video recognition model to obtain relevance prediction data includes: inputting the text relationship vector into the at least two one-dimensional convolutional layers respectively to obtain at least two feature extraction results; inputting the at least two feature extraction results into the at least two pooling layers respectively to obtain at least two pooling results; inputting the at least two pooling results into the regression layer, and obtaining relevance prediction data through concatenation, linear mapping, and regression processing by the regression layer.

[0151] In this exemplary embodiment, after obtaining the text relation vector of the first network structure, the text relation vector is input into different one-dimensional convolutional layers, and different sizes of convolutional kernels are set for different one-dimensional convolutional layers. Different convolutional kernels are used to extract local semantic information of different dimensions to fully extract the features in the text relation vector and obtain the feature extraction results. A pooling layer is set after each one-dimensional convolutional layer to further extract important information in the feature extraction results and obtain the pooling results. Different pooling results are input into the regression layer. In the regression layer, the pooling results are first concatenated, then linearly mapped, and then regressed to obtain the relevance prediction data. The accuracy of the relevance prediction data is high.

[0152] For example, such as Figure 4 As shown, after obtaining the input short video sample, the cover frame (or cover image) of the short video is obtained and processed as follows:

[0153] For the cover frame, OCR recognition is performed on the cover frame to obtain the cover text data. The third valid text data in the cover text data is extracted using the second preset character "#". The meaningless content in the cover text data is filtered out using the second preset list to obtain the fourth valid text data. After filtering the third valid text data from the fourth valid text data, the first sub-text data is determined.

[0154] For the description text, the first valid text data is extracted using the first preset character "#", and the meaningless content in the description text is filtered out using the first preset list to obtain the second valid text data. After filtering the first valid text data from the second valid text data, the second sub-text data is determined.

[0155] Calculate the similarity value between the first and second sub-text data, determine the basic text data based on the similarity value, and concatenate the third and first valid text data after the basic text data to determine the valid title, i.e., the display text data.

[0156] For audio, the audio is segmented to obtain audio segment 1, segment 2, ... segment n. For each audio segment, the human voice is extracted from the audio segment to determine n target sound data. Speech recognition is performed on each of the n target sound data, and the content after speech recognition is spliced ​​to obtain the effective text, i.e., the playback text data.

[0157] The obtained valid titles and valid copy are concatenated and then input into the first network structure. Specifically, the first network structure includes multiple transformer layers. The output of the last transformer layer is a text relation vector. The obtained text relation vector is then input into different one-dimensional convolutional layers (Conv-1d) to obtain multiple first-level extracted features. These one-dimensional convolutional layers have different kernel sizes, such as 3x3, 4x4, ..., nxn. Each one-dimensional convolutional layer is followed by a global max-pooling layer. Each feature extraction result is input into the corresponding max-pooling layer to obtain multiple pooling results. These multiple pooling results are then input into a regression layer. Within the regression layer, the multiple pooling results are first concatenated, then linearly mapped, and finally activated by a function such as the sigmoid activation function to output the relevance score, i.e., the relevance prediction data. (y – y_pred) 2 The predetermined loss function is used to input the predicted relevance data and the labeled relevance data into the predetermined loss function to determine the target loss. The target loss is then used to update the network parameters of the second network structure using the error backpropagation algorithm until a video recognition model that meets the predetermined conditions is trained. It should be noted that the network parameters of the first network structure remain unchanged during the training process.

[0158] Figure 5 This is a flowchart illustrating a video recognition method according to an exemplary embodiment, including the following steps.

[0159] In step 51, the video to be identified is obtained, which includes display data and audio data.

[0160] In step 52, the display text data is extracted from the display data.

[0161] In step 53, speech recognition is performed on the audio data to obtain the playback text data.

[0162] In step 54, the displayed text data and the played text data are input into the video recognition model to determine the video recognition result.

[0163] The video recognition model is trained according to the above-described video recognition model training method.

[0164] This exemplary embodiment, after acquiring the video to be identified, determines the display data and audio data of the video. The display data refers to the data presented to the user before the video is clicked and played, used to provide the user with an initial understanding of the video. Data extraction is performed on the display data to determine the display text data. The actual content of the video is usually represented by audio data; therefore, speech recognition is performed on the audio data of the video sample to determine the corresponding playback text data. The display text data and playback text data are input into the video recognition model. Relevance prediction data is obtained based on the video recognition model, and the video recognition result is determined based on the correlation prediction data. If the video recognition result shows a low correlation between the display data and audio data of the video to be identified, then the distribution of the video to be identified is avoided or the exposure of the video to be identified is reduced, minimizing the opportunity for the user to encounter the video, thus addressing the problem at its root and improving the user experience.

[0165] Figure 6 This is a block diagram illustrating a training apparatus for a video recognition model according to an exemplary embodiment. The apparatus includes a sample set acquisition module 61, a text data extraction module 62, a speech recognition processing module 63, a text input processing module 64, a prediction data acquisition module 65, and a parameter training processing module 66.

[0166] The sample set acquisition module 61 is configured to acquire a video sample set, wherein the video samples in the video sample set include display data, audio data, and correlation annotation data between the display data and the audio data.

[0167] The text data extraction module 62 is configured to extract the display text data from the display data;

[0168] The speech recognition processing module 63 is configured to perform speech recognition on the audio data and obtain the playback text data;

[0169] The text input processing module 64 is configured to input the displayed text data and the played text data into the first network structure of the video recognition model. The first network structure is used to capture the dependency relationship between the displayed text data and the played text data and obtain a text relationship vector.

[0170] The prediction data acquisition module 65 is configured to input the text relationship vector into the second network structure of the video recognition model to acquire correlation prediction data.

[0171] The parameter training processing module 66 is configured to train the network parameters corresponding to the second network structure based on the target loss determined by the correlation prediction data and the correlation annotation data, so as to obtain a video recognition model that meets preset conditions.

[0172] In an exemplary embodiment of this disclosure, the display data includes a cover image and descriptive text data; therefore, the text data extraction module includes:

[0173] The cover image recognition unit is configured to recognize the cover image and obtain cover text data;

[0174] The first extraction processing unit is configured to extract the first sub-text data from the cover text data;

[0175] The second extraction processing unit is configured to extract the second sub-text data from the descriptive text data;

[0176] The text data acquisition unit is configured to determine the text data to be displayed based on the first sub-text data and the second sub-text data.

[0177] In an exemplary embodiment of this disclosure, the second extraction processing unit includes:

[0178] The first extraction subunit is configured to extract the first valid text data after extracting the first preset character from the description text data;

[0179] The first determining subunit is configured to compare the descriptive text data with data in a first preset list to determine the second valid text data;

[0180] The second determining subunit is configured to filter the first valid text data in the second valid text data to determine the second subtext data;

[0181] The text data acquisition unit is then configured to determine the text data to be displayed based on the first sub-text data, the second sub-text data, and the first valid text data.

[0182] In an exemplary embodiment of this disclosure, the first extraction processing unit includes:

[0183] The second extraction subunit is configured to extract the third valid text data after the second preset character from the cover text data;

[0184] The third determining subunit is configured to compare the cover text data with the data in the second preset list to determine the fourth valid text data;

[0185] The fourth determining subunit is configured to filter the third valid text data in the fourth valid text data to determine the first subtext data;

[0186] The text data acquisition unit is configured to determine the text data to be displayed based on the first sub-text data, the second sub-text data, the first valid text data, and the third valid text data.

[0187] In an exemplary embodiment of this disclosure, the text data acquisition unit includes:

[0188] The similarity value determination subunit is configured to determine the similarity value between the first sub-text data and the second sub-text data;

[0189] The first judgment subunit is configured to determine the first sub-text data as basic text data, or the second sub-text data as basic text data, or to determine basic text data based on the number of characters corresponding to the first sub-text data and the second sub-text data respectively, if the similarity value is greater than a preset threshold.

[0190] The second judgment subunit is configured to determine the concatenation result of the first sub-text data and the second sub-text data as the basic text data if the similarity value is less than the preset threshold.

[0191] The fifth determining subunit is configured to concatenate the basic text data and the first valid text data to determine the text data to be displayed.

[0192] In an exemplary embodiment of this disclosure, the sample set acquisition module includes:

[0193] The sample set acquisition unit is configured to acquire video samples from the video sample set;

[0194] The cover image determination unit is configured to extract an image from the video sample that does not have a cover if the video sample does not have a cover, and determine the cover image of the video sample.

[0195] In an exemplary embodiment of this disclosure, the speech recognition processing module includes:

[0196] An audio segmentation processing unit is configured to segment the audio data to obtain at least two audio segment data.

[0197] The sound extraction processing unit is configured to extract target sound data from the at least two audio segment data respectively;

[0198] The speech recognition processing unit is configured to perform speech recognition on the target sound data and obtain the playback text data.

[0199] In an exemplary embodiment of this disclosure, the second network structure includes at least two one-dimensional convolutional layers, at least two pooling layers, and a regression layer; wherein the kernel sizes of the at least two one-dimensional convolutional layers are different; then the prediction data acquisition module includes:

[0200] The first feature extraction unit is configured to input the text relationship vector into the at least two one-dimensional convolutional layers respectively to obtain at least two feature extraction results;

[0201] The second feature extraction unit is configured to input the at least two feature extraction results into at least two pooling layers respectively to obtain at least two pooling results;

[0202] The prediction data acquisition unit is configured to input the at least two pooling results into the regression layer, and obtain correlation prediction data through the concatenation, linear mapping and regression processing of the regression layer.

[0203] Figure 7 This is a block diagram illustrating a video recognition device according to an exemplary embodiment. The video recognition device includes: a video acquisition module 71, a data extraction module 72, a speech recognition module 73, and a result determination module 74;

[0204] The video acquisition module 71 is configured to acquire a video to be identified, the video to be identified including display data and audio data;

[0205] Data extraction module 72 is configured to extract display text data from the display data;

[0206] The speech recognition module 73 is configured to perform speech recognition on the audio data and obtain the playback text data;

[0207] The result determination module 74 is configured to input the displayed text data and the played text data into the video recognition model to determine the video recognition result;

[0208] The video recognition model is trained according to the above-described video recognition model training method.

[0209] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0210] Figure 8 This is a block diagram illustrating an electronic device according to an exemplary embodiment. The electronic device may be a server or a similar computing device. Taking electronic device 800 as an example, it is described as a server. (Refer to...) Figure 8 The electronic device 800 includes a processing component 822, which further includes one or more processors, and memory resources represented by a memory 832 for storing instructions, such as application programs, that can be executed by the processing component 822. The application programs stored in the memory 832 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 822 is configured to execute instructions to perform the training method or video recognition method of the aforementioned video recognition model.

[0211] Electronic device 800 may also include a power supply component 826 configured to perform power management of electronic device 800, a wired or wireless network interface 850 configured to connect electronic device 800 to a network, and an input / output (I / O) interface 858. Electronic device 800 may operate on an operating system stored in memory 832, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or similar.

[0212] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 832 including instructions, which can be executed by a processing component 822 of an electronic device 800 to complete the above-described method for implementing the relational tag. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, or optical data storage device, etc.

[0213] In an exemplary embodiment, a computer program product is also provided, including a computer program or computer instructions, which, when executed by a processor, implement the above-described training method or video recognition method for the video recognition model.

[0214] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0215] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A training method for a video recognition model, characterized in that, include: Obtain a video sample set, wherein the video samples in the video sample set include display data, audio data, and correlation annotation data between the display data and the audio data. The display data is the data presented to the user before the video is clicked to play, and the correlation annotation data refers to the real data corresponding to the correlation between the display data and the audio data of the video samples. Extract the display text data from the display data; Perform speech recognition on the audio data to obtain the playback text data; The displayed text data and the played text data are input into the first network structure of the video recognition model. The first network structure is used to capture the dependency relationship between the displayed text data and the played text data and obtain a text relationship vector. The text relationship vector is input into the second network structure of the video recognition model to obtain relevance prediction data; Based on the target loss determined by the correlation prediction data and the correlation annotation data, the network parameters corresponding to the second network structure are trained to obtain a video recognition model that meets preset conditions.

2. The method according to claim 1, characterized in that, The display data includes a cover image and descriptive text data. Extracting the display text data from the display data includes: The cover image is identified to obtain the cover text data; Extract the first sub-text data from the cover text data; Extract the second sub-text data from the descriptive text data; Based on the first sub-text data and the second sub-text data, the text data to be displayed is determined.

3. The method according to claim 2, characterized in that, The extraction of the second sub-text data from the descriptive text data includes: Extract the first valid text data after the first preset character from the description text data; The descriptive text data is compared with the data in the first preset list to determine the second valid text data; Filter the first valid text data into the second valid text data to determine the second sub-text data; The step of determining the display text data based on the first sub-text data and the second sub-text data includes: Based on the first sub-text data, the second sub-text data, and the first valid text data, the text data to be displayed is determined.

4. The method according to claim 3, characterized in that The extraction of the first sub-text data from the cover text data includes: Extract the third valid text data after the second preset character from the cover text data; The cover text data is compared with the data in the second preset list to determine the fourth valid text data; Filter the third valid text data from the fourth valid text data to determine the first sub-text data; The step of determining the display text data based on the first sub-text data, the second sub-text data, and the first valid text data includes: Based on the first sub-text data, the second sub-text data, the first valid text data, and the third valid text data, the text data to be displayed is determined.

5. The method according to claim 3, characterized in that, The step of determining the display text data based on the first sub-text data, the second sub-text data, and the first valid text data includes: Determine the similarity value between the first sub-text data and the second sub-text data; If the similarity value is greater than a preset threshold, then the first sub-text data is determined as the basic text data, or the second sub-text data is determined as the basic text data, or the basic text data is determined based on the number of characters corresponding to the first sub-text data and the second sub-text data respectively; If the similarity value is less than the preset threshold, the concatenation result of the first sub-text data and the second sub-text data is determined as the basic text data; The basic text data and the first valid text data are concatenated to determine the text data to be displayed.

6. The method according to claim 2, characterized in that, The acquired video sample set includes: Retrieve video samples from the video sample set; If the video sample does not have a cover image, then the cover image of the video sample without a cover image is extracted.

7. The method according to claim 1, characterized in that, The step of performing speech recognition on the audio data to obtain the playback text data includes: The audio data is segmented to obtain at least two audio segment data; Extract the target sound data from each of the at least two audio segment data; Speech recognition is performed on the target sound data to obtain the playback text data.

8. The method according to any one of claims 1-7, characterized in that, The second network structure includes at least two one-dimensional convolutional layers, at least two pooling layers, and a regression layer; wherein the kernel sizes of the at least two one-dimensional convolutional layers are different; then, inputting the text relation vector into the second network structure of the video recognition model to obtain relevance prediction data includes: The text relationship vector is input into the at least two one-dimensional convolutional layers respectively to obtain at least two feature extraction results; The at least two feature extraction results are respectively input into at least two pooling layers to obtain at least two pooling results; The at least two pooling results are input into the regression layer, and the correlation prediction data is obtained through the concatenation, linear mapping and regression processing of the regression layer.

9. A video recognition method, characterized in that, include: Acquire the video to be identified, which includes display data and audio data; Extract the display text data from the display data; Perform speech recognition on the audio data to obtain the playback text data; The displayed text data and the played text data are input into the video recognition model to determine the video recognition result; The video recognition model is trained using the training method for the video recognition model according to any one of claims 1-8.

10. A training device for a video recognition model, characterized in that, include: The sample set acquisition module is configured to acquire a video sample set, wherein the video samples in the video sample set include display data, audio data, and correlation annotation data between the display data and the audio data. The display data is the data presented to the user before the video is clicked and played, and the correlation annotation data refers to the real data corresponding to the correlation between the display data and the audio data of the video samples. The text data extraction module is configured to extract the display text data from the display data; The speech recognition processing module is configured to perform speech recognition on the audio data and obtain the playback text data; The text input processing module is configured to input the displayed text data and the played text data into the first network structure of the video recognition model. The first network structure is used to capture the dependency relationship between the displayed text data and the played text data and obtain a text relationship vector. The prediction data acquisition module is configured to input the text relationship vector into the second network structure of the video recognition model to acquire correlation prediction data. The parameter training processing module is configured to train the network parameters corresponding to the second network structure based on the target loss determined by the correlation prediction data and the correlation annotation data, so as to obtain a video recognition model that meets preset conditions.

11. The apparatus according to claim 10, characterized in that, The displayed data includes a cover image and descriptive text data. Therefore, the text data extraction module includes: The cover image recognition unit is configured to recognize the cover image and obtain cover text data; The first extraction processing unit is configured to extract the first sub-text data from the cover text data; The second extraction processing unit is configured to extract the second sub-text data from the descriptive text data; The text data acquisition unit is configured to determine the text data to be displayed based on the first sub-text data and the second sub-text data.

12. The apparatus according to claim 11, characterized in that, The second extraction processing unit includes: The first extraction subunit is configured to extract the first valid text data after extracting the first preset character from the description text data; The first determining subunit is configured to compare the descriptive text data with data in a first preset list to determine the second valid text data; The second determining subunit is configured to filter the first valid text data in the second valid text data to determine the second subtext data; The text data acquisition unit is then configured to determine the text data to be displayed based on the first sub-text data, the second sub-text data, and the first valid text data.

13. The apparatus according to claim 12, characterized in that, The first extraction processing unit includes: The second extraction subunit is configured to extract the third valid text data after the second preset character from the cover text data; The third determining subunit is configured to compare the cover text data with the data in the second preset list to determine the fourth valid text data; The fourth determining subunit is configured to filter the third valid text data in the fourth valid text data to determine the first subtext data; The text data acquisition unit is configured to determine the text data to be displayed based on the first sub-text data, the second sub-text data, the first valid text data, and the third valid text data.

14. The apparatus according to claim 12, characterized in that, The text data acquisition unit includes: The similarity value determination subunit is configured to determine the similarity value between the first sub-text data and the second sub-text data; The first judgment subunit is configured to determine the first sub-text data as basic text data, or the second sub-text data as basic text data, or to determine basic text data based on the number of characters corresponding to the first sub-text data and the second sub-text data respectively, if the similarity value is greater than a preset threshold. The second judgment subunit is configured to determine the concatenation result of the first sub-text data and the second sub-text data as the basic text data if the similarity value is less than the preset threshold. The fifth determining subunit is configured to concatenate the basic text data and the first valid text data to determine the text data to be displayed.

15. The apparatus according to claim 11, characterized in that, The sample set acquisition module includes: The sample set acquisition unit is configured to acquire video samples from the video sample set; The cover image determination unit is configured to extract an image from the video sample that does not have a cover if the video sample does not have a cover, and determine the cover image of the video sample.

16. The apparatus according to claim 10, characterized in that, The speech recognition processing module includes: An audio segmentation processing unit is configured to segment the audio data to obtain at least two audio segment data. The sound extraction processing unit is configured to extract target sound data from the at least two audio segment data respectively; The speech recognition processing unit is configured to perform speech recognition on the target sound data and obtain the playback text data.

17. The apparatus according to any one of claims 10-16, characterized in that, The second network structure includes at least two one-dimensional convolutional layers, at least two pooling layers, and a regression layer; wherein the kernel sizes of the at least two one-dimensional convolutional layers are different; then the prediction data acquisition module includes: The first feature extraction unit is configured to input the text relationship vector into the at least two one-dimensional convolutional layers respectively to obtain at least two feature extraction results; The second feature extraction unit is configured to input the at least two feature extraction results into at least two pooling layers respectively to obtain at least two pooling results; The prediction data acquisition unit is configured to input the at least two pooling results into the regression layer, and obtain correlation prediction data through the concatenation, linear mapping and regression processing of the regression layer.

18. A video recognition device, characterized in that, include: The video acquisition module is configured to acquire a video to be identified, the video to be identified including display data and audio data; The data extraction module is configured to extract the display text data from the display data; The speech recognition module is configured to perform speech recognition on the audio data and obtain the playback text data; The result determination module is configured to input the displayed text data and the played text data into the video recognition model to determine the video recognition result; The video recognition model is trained using the training method for the video recognition model according to any one of claims 1-8.

19. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the training method of the video recognition model as described in any one of claims 1 to 8, or the video recognition method as described in claim 9.

20. A storage medium, wherein instructions in the storage medium, when executed by a processor of an electronic device, enable the electronic device to perform a training method for a video recognition model as described in any one of claims 1 to 8, or a video recognition method as described in claim 9.

21. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of the video recognition model according to any one of claims 1 to 8, or the video recognition method according to claim 9.

Images