Text generation method, device, computer equipment and medium

Abstract

The present disclosure relates to a text generation method, device, computer equipment and medium, and belongs to the field of computer technology. In the embodiment of the present disclosure, by constructing a dual-stream architecture of an image feature extraction submodel and a text feature extraction submodel in a video feature extraction model, the image features and text features of the target video can be accurately extracted, and by setting a text generation submodel in the video feature extraction model, the image features and text features of the target video are processed, and multiple characters whose self-attention scores meet the text generation conditions can be output, and then the content description text of the target video can be automatically generated based on the output multiple characters, and a video feature extraction model based on text generation is provided, and the generated content description text contains a rich amount of information, which can better characterize the target video and improve the accuracy of video characterization.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a text generation method, device, computer equipment and medium. Background Art

[0002] With the rapid development of computer technology and Internet technology, video processing technology has gradually become an emerging research hotspot. In video processing technology, it is usually necessary to extract video features that can represent the video content, and then use the video features to perform video recommendation, video classification or video search and other processing processes.

[0003] At present, when extracting video features from a video, a classifier is usually used to classify the images in the video to obtain the category features of the video. However, when extracting video features through video classification, the extracted video features contain less information, which is not conducive to the subsequent video recommendation, video classification or video search process. Summary of the invention

[0004] The present disclosure provides a text generation method, device, computer equipment and medium, and provides a video feature extraction model based on text generation. The generated content description text contains rich information and can better characterize the target video. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a text generation method is provided, the method comprising:

[0006] Obtain image information and text information of the target video;

[0007] The image information and the text information are input into a video feature extraction model, and the image information is subjected to feature extraction by the image feature extraction submodel of the video feature extraction model to obtain image features of the target video, and the text information is subjected to feature extraction by the text feature extraction submodel of the video feature extraction model to obtain text features of the target video;

[0008] The image feature and the text feature are processed through the self-attention layer included in the text generation sub-model of the video feature extraction model, and a plurality of characters whose self-attention scores meet the text generation conditions are output, and a content description text of the target video is generated based on the plurality of characters.

[0009] In the disclosed embodiment, by constructing a dual-stream architecture of an image feature extraction submodel and a text feature extraction submodel in a video feature extraction model, the image features and text features of a target video can be accurately extracted. Moreover, by setting a text generation submodel in the video feature extraction model, the image features and text features of the target video can be processed, and a plurality of characters whose self-attention scores meet text generation conditions can be output. Then, based on the outputted plurality of characters, a content description text of the target video can be automatically generated. A video feature extraction model based on text generation is provided, and the generated content description text contains a rich amount of information, can better characterize the target video, and improves the accuracy of video characterization.

[0010] In some embodiments, the process of acquiring the image information of the target video includes at least one of the following:

[0011] Obtain a cover image of the target video; or obtain at least one frame of image in the target video.

[0012] In the disclosed embodiments, the image information of the target video can be quickly acquired by acquiring the cover image of the target video or the image frames included in the target video. While ensuring the efficiency of acquiring the image information, the types of image information are enriched and the flexibility of acquiring the image information is improved.

[0013] In some embodiments, the process of obtaining the text information of the target video includes at least one of the following:

[0014] Obtain description information of the target video; obtain title information of the target video; obtain subtitle information of the target video; obtain text recognition result of the target video, which is the result obtained by performing text recognition on at least one frame of image in the target video; obtain audio recognition result of the target video, which is the result obtained by performing audio recognition on background audio of the target video.

[0015] In the disclosed embodiments, by acquiring description information, title information, subtitle information, text recognition results or audio recognition results of the target video, the text information of the target video can be quickly acquired. While ensuring the efficiency of acquiring the text information, the types of text information are enriched and the flexibility of acquiring the text information is improved.

[0016] In some embodiments, the content description text includes at least one of content category description text, content form description text, content theme description text, and content detail description text.

[0017] In the disclosed embodiment, based on the video feature extraction model, various types of content description texts can be obtained. On the one hand, more expressive content description texts are generated. On the other hand, the various types of content description texts can describe the content of the target video from different dimensions, enriching the types of generated content description texts and being able to more fully and completely characterize the target video.

[0018] In some embodiments, the content description text includes multiple types of content description text;

[0019] The method further comprises: processing the image feature and the text feature through the self-attention layer included in the text generation sub-model of the video feature extraction model, outputting a plurality of characters whose self-attention scores meet the text generation condition, and generating the content description text of the target video based on the plurality of characters.

[0020] Splicing the image feature with the text feature to obtain a spliced ​​feature;

[0021] On the splicing feature, add the type identification of each type;

[0022] The image feature and the text feature are processed by the self-attention layer included in the text generation sub-model of the video feature extraction model, and a plurality of characters whose self-attention scores meet the text generation condition are outputted. The content description text of the target video is generated based on the plurality of characters, including:

[0023] The splicing features after adding the type identifier are input into the text generation sub-model. The splicing features are processed based on the processing mechanisms corresponding to each type identifier through the self-attention layer included in the text generation sub-model, and a plurality of characters whose self-attention scores meet the text generation conditions are output. Based on the plurality of characters, multiple types of content description texts of the target video are generated.

[0024] In the disclosed embodiment, each type of type identifier is added to the splicing feature so that the text generation sub-model in the video feature extraction model can trigger the generation of multiple types of content description texts of the target video based on the type identifier of each type, thereby ensuring the smooth progress of text generation.

[0025] In some embodiments, the training process of the video feature extraction model includes:

[0026] In the i-th iteration process of model training, the image information and text information of the sample video are input into the video feature extraction model determined in the i-1-th iteration process to obtain the text training result of the i-th iteration process, where i is a positive integer greater than 1;

[0027] Determine a model loss value of the i-th iteration process based on the text training result of the i-th iteration process and the description text label of the sample video;

[0028] Based on the model loss value of the i-th iteration process, the model parameters of the video feature extraction model determined in the i-1-th iteration process are adjusted, and the i+1-th iteration process is performed based on the adjusted model parameters. The above training iteration process is repeated until the training meets the target conditions.

[0029] In the disclosed embodiment, during any iteration of model training, the model loss value of this iteration is used to adjust the model parameters of the video feature extraction model to improve the feature extraction capability of the video feature extraction model, thereby training a video feature extraction model with higher feature extraction capability.

[0030] In some embodiments, the content description text includes multiple types of content description text;

[0031] Based on the text training result of the i-th iteration process and the description text label of the sample video, determining the model loss value of the i-th iteration process includes:

[0032] For any type, based on the text training result of the i-th iteration process on the type and the descriptive text label of the sample video on the type, determine the loss value of the i-th iteration process on the type;

[0033] Based on the loss values ​​of the i-th iteration process on the multiple types and the weight coefficients of the video feature extraction network on the multiple types, a weighted sum is performed to obtain the model loss value of the i-th iteration process.

[0034] In the disclosed embodiment, a weight coefficient corresponding to each type is set for each type involved in text generation, and then the model loss value is determined by using the loss value corresponding to each type and the weight coefficient corresponding to each type, thereby improving the accuracy of determining the model loss value.

[0035] In some embodiments, before performing weighted summation based on the loss values ​​of the i-th iteration process on the multiple types and the weight coefficients of the video feature extraction network on the multiple types to obtain the model loss value of the i-th iteration process, the method further includes:

[0036] For any type, based on the number of correct texts in the i-th iteration process on the type and the total number of texts, determine the correct proportion of the i-th iteration process on the type, where the correct proportion represents the proportion of the number of correct texts to the total number of texts in the i-th iteration process;

[0037] Based on the correct proportion of the i-th iteration process on the type, the weight coefficient of the video feature extraction network on the type is determined, wherein the correct proportion is negatively correlated with the weight coefficient.

[0038] In the disclosed embodiments, for each type involved in text generation, the weight coefficient corresponding to each type is determined according to the correct proportion corresponding to each type. Since the correct proportion represents the ratio of the number of correct texts to the total number of texts, and since the correct proportion is negatively correlated with the weight coefficient, when calculating the model loss value, a smaller weight coefficient is set for the type with a large correct proportion, and a larger weight coefficient is set for the type with a small correct proportion, thereby improving the accuracy of determining the weight coefficient and thus improving the accuracy of determining the model loss value.

[0039] In some embodiments, the number of the text information is multiple;

[0040] Before extracting features from the text information through the text feature extraction submodel of the video feature extraction model to obtain the text features of the target video, the method further includes:

[0041] Splicing multiple pieces of text information to obtain the spliced ​​text information;

[0042] Based on the spliced ​​text information, the step of extracting features from the text information through the text feature extraction submodel of the video feature extraction model is executed to obtain text features of the target video.

[0043] In the embodiment of the present disclosure, when there are multiple pieces of text information, the text information is spliced ​​to obtain spliced ​​text information, and then the spliced ​​text information is used to extract text features. This process refers to multiple types of text information and improves the accuracy of extracting text features.

[0044] In some embodiments, after splicing the plurality of text messages to obtain the spliced ​​text messages, the method further includes:

[0045] Extract the target number of characters from the concatenated text information;

[0046] Based on the extracted characters, the step of extracting features from the text information through the text feature extraction submodel of the video feature extraction model is executed to obtain text features of the target video.

[0047] In the disclosed embodiment, a target number of characters are extracted from the spliced ​​text information so that a subsequent process of extracting text features can be performed based on the extracted number of characters. This reduces the amount of computation of the video feature extraction model and improves the efficiency of extracting text features while ensuring sufficient text information is input.

[0048] According to a second aspect of an embodiment of the present disclosure, a text generation device is provided, the device comprising:

[0049] An acquisition unit is configured to acquire image information and text information of a target video;

[0050] An extraction unit is configured to input the image information and the text information into a video feature extraction model, perform feature extraction on the image information through an image feature extraction submodel of the video feature extraction model to obtain image features of the target video, and perform feature extraction on the text information through a text feature extraction submodel of the video feature extraction model to obtain text features of the target video;

[0051] The processing unit is configured to execute the self-attention layer included in the text generation sub-model of the video feature extraction model, process the image feature and the text feature, output a plurality of characters whose self-attention scores meet the text generation conditions, and generate a content description text of the target video based on the plurality of characters.

[0052] In some embodiments, the acquisition unit is configured to perform at least one of the following:

[0053] Obtain a cover image of the target video; or obtain at least one frame of image in the target video.

[0054] In some embodiments, the acquisition unit is configured to perform at least one of the following:

[0055] Obtain description information of the target video; obtain title information of the target video; obtain subtitle information of the target video; obtain text recognition result of the target video, which is the result obtained by performing text recognition on at least one frame of image in the target video; obtain audio recognition result of the target video, which is the result obtained by performing audio recognition on background audio of the target video.

[0056] In some embodiments, the content description text includes at least one of content category description text, content form description text, content theme description text, and content detail description text.

[0057] In some embodiments, the content description text includes multiple types of content description text;

[0058] The device also includes:

[0059] A first splicing unit is configured to perform splicing of the image feature and the text feature to obtain a splicing feature;

[0060] An adding unit is configured to add type identifiers of various types on the splicing feature;

[0061] The processing unit is configured to perform:

[0062] The splicing features after adding the type identifier are input into the text generation sub-model. The splicing features are processed based on the processing mechanisms corresponding to each type identifier through the self-attention layer included in the text generation sub-model, and a plurality of characters whose self-attention scores meet the text generation conditions are output. Based on the plurality of characters, multiple types of content description texts of the target video are generated.

[0063] In some embodiments, the apparatus further comprises:

[0064] An input unit is configured to input the image information and text information of the sample video into the video feature extraction model determined in the i-1th iteration process during the i-th iteration process of the model training, and obtain the text training result of the i-th iteration process, where i is a positive integer greater than 1;

[0065] A determination unit is configured to determine a model loss value of the i-th iteration process based on the text training result of the i-th iteration process and the description text label of the sample video;

[0066] The adjustment unit is configured to adjust the model parameters of the video feature extraction model determined in the i-1th iteration process based on the model loss value of the i-th iteration process, perform the i+1th iteration process based on the adjusted model parameters, and repeat the above training iteration process until the training meets the target conditions.

[0067] In some embodiments, the content description text includes multiple types of content description text;

[0068] The determination unit is configured to execute:

[0069] For any type, based on the text training result of the i-th iteration process on the type and the descriptive text label of the sample video on the type, determine the loss value of the i-th iteration process on the type;

[0070] Based on the loss values ​​of the i-th iteration process on the multiple types and the weight coefficients of the video feature extraction network on the multiple types, a weighted sum is performed to obtain the model loss value of the i-th iteration process.

[0071] In some embodiments, the determining unit is further configured to execute:

[0072] For any type, based on the number of correct texts in the i-th iteration process on the type and the total number of texts, determine the correct proportion of the i-th iteration process on the type, where the correct proportion represents the proportion of the number of correct texts to the total number of texts in the i-th iteration process;

[0073] Based on the correct proportion of the i-th iteration process on the type, the weight coefficient of the video feature extraction network on the type is determined, wherein the correct proportion is negatively correlated with the weight coefficient.

[0074] In some embodiments, the number of the text information is multiple;

[0075] The device also includes a second splicing unit configured to splice the plurality of text messages to obtain the spliced ​​text messages;

[0076] The extraction unit is further configured to execute, based on the spliced ​​text information, a step of extracting features from the text information through a text feature extraction sub-model of the video feature extraction model to obtain text features of the target video.

[0077] In some embodiments, the extraction unit is further configured to perform:

[0078] Extract the target number of characters from the concatenated text information;

[0079] Based on the extracted characters, the step of extracting features from the text information through the text feature extraction submodel of the video feature extraction model is executed to obtain text features of the target video.

[0080] According to a third aspect of an embodiment of the present disclosure, a computer device is provided, the computer device comprising:

[0081] one or more processors;

[0082] A memory for storing program codes executable by the processor;

[0083] The processor is configured to execute the program code to implement the above-mentioned text generation method.

[0084] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, which includes: when the program code in the computer-readable storage medium is executed by a processor of a computer device, the computer device is enabled to execute the above-mentioned text generation method.

[0085] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, which implements the above-mentioned text generation method when executed by a processor.

[0086] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0088] Figure 1 is a schematic diagram of an implementation environment of a text generation method according to an exemplary embodiment;

[0089] Figure 2 is a flowchart of a text generation method according to an exemplary embodiment;

[0090] Figure 3 is a flowchart of a text generation method according to an exemplary embodiment;

[0091] Figure 4 A framework of a video feature extraction model is shown according to an exemplary embodiment;

[0092] Figure 5 is a flow chart of a method for training a video feature extraction model according to an exemplary embodiment;

[0093] Figure 6 is a block diagram of a text generation device according to an exemplary embodiment;

[0094] Figure 7 is a block diagram of a terminal according to an exemplary embodiment;

[0095] Figure 8 It is a block diagram of a server according to an exemplary embodiment. DETAILED DESCRIPTION

[0096] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0097] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0098] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the embodiments of the present disclosure are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the image information and text information involved in the embodiments of the present disclosure are obtained with full authorization. In some embodiments, the terminal provides a permission inquiry page, which is used to ask the user whether to grant permission to obtain the image information and text information of the target video. In the permission inquiry page, an authorization consent control and a authorization rejection control are displayed. When the user triggers the authorization consent control, the text generation method provided in the embodiments of the present disclosure is used to obtain the image information and text information of the target video, and then the content description text of the target video is obtained based on the image information and text information of the target video.

[0099] Figure 1 is a schematic diagram of an implementation environment of a text generation method according to an exemplary embodiment, see Figure 1 , the implementation environment includes: a terminal 101 and a server 102.

[0100] The terminal 101 may be at least one of a smart phone, a smart watch, a desktop computer, a laptop, a virtual reality terminal, an augmented reality terminal, a wireless terminal, and a laptop computer. The terminal 101 has a communication function and can access a wired network or a wireless network. The terminal 101 may generally refer to one of a plurality of terminals. This embodiment is only illustrated by taking the terminal 101 as an example. Those skilled in the art may know that the number of the above terminals may be more or less.

[0101] The server 102 can be an independent physical server, or a server cluster or distributed file system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some embodiments, the server 102 is directly or indirectly connected to the terminal 101 via wired or wireless communication, which is not limited in the embodiments of the present disclosure. Optionally, the number of the above-mentioned servers 102 can be more or less, which is not limited in the embodiments of the present disclosure. Of course, the server 102 can also include other functional servers to provide more comprehensive and diversified services.

[0102] In some embodiments, the text generation method provided by the embodiment of the present disclosure is executed by the terminal 101. For example, for the target video stored locally by the terminal 101 or the target video downloaded by the terminal 101, the terminal 101 responds to the text generation operation of the target video and uses the text generation method provided by the embodiment of the present disclosure to generate the content description text of the target video; in some other embodiments, the text generation method provided by the embodiment of the present disclosure is executed by the server 102. For example, for the target video in the video database associated with the server 102, the server 102 uses the text generation method provided by the embodiment of the present disclosure to periodically generate the content description text of the target video; in other embodiments, the text generation method provided by the embodiment of the present disclosure is jointly executed by the terminal 101 and the server 102. For example, the terminal 101 responds to the text generation operation of the target video and uploads the target video to the server 102. The server 102 receives the target video uploaded by the terminal 101 and uses the text generation method provided by the embodiment of the present disclosure to generate the content description text of the target video. In the embodiment of the present disclosure, the target video is used to refer to the video to be described, that is, the video to be generated with the content description text.

[0103] It should be noted that the text generation method provided in the embodiments of the present disclosure can be applied in the scenarios of video recommendation, video classification or video search. In some embodiments, the text generation method provided in the embodiments of the present disclosure is used to generate the content description text of the target video, and then the content description text of the target video is used to characterize the target video to achieve subsequent video recommendation, video classification or video search.

[0104] Figure 2 is a flowchart of a text generation method according to an exemplary embodiment. Figure 2 As shown, the method is executed by a computer device, which can be provided as the above Figure 1 The terminal or server shown schematically comprises the following steps:

[0105] In step 201, a computer device obtains image information and text information of a target video.

[0106] In step 202, the computer device inputs the image information and the text information into a video feature extraction model, performs feature extraction on the image information through the image feature extraction submodel of the video feature extraction model to obtain image features of the target video, and performs feature extraction on the text information through the text feature extraction submodel of the video feature extraction model to obtain text features of the target video.

[0107] In step 203, the computer device processes the image feature and the text feature through the self-attention layer included in the text generation sub-model of the video feature extraction model, outputs a plurality of characters whose self-attention scores meet the text generation conditions, and generates a content description text of the target video based on the plurality of characters.

[0108] The technical solution provided by the embodiments of the present disclosure can accurately extract the image features and text features of the target video by constructing a dual-stream architecture of an image feature extraction submodel and a text feature extraction submodel in a video feature extraction model, and can process the image features and text features of the target video by setting a text generation submodel in the video feature extraction model, and can output multiple characters whose self-attention scores meet the text generation conditions, and then can automatically generate a content description text of the target video based on the output multiple characters, thereby providing a video feature extraction model based on text generation, and the generated content description text contains a rich amount of information, can better characterize the target video, and improves the accuracy of video characterization.

[0109] In some embodiments, the process of acquiring the image information of the target video includes at least one of the following:

[0110] Obtain a cover image of the target video; or obtain at least one frame of image in the target video.

[0111] In some embodiments, the process of obtaining the text information of the target video includes at least one of the following:

[0112] Obtain description information of the target video; obtain title information of the target video; obtain subtitle information of the target video; obtain text recognition result of the target video, which is the result obtained by performing text recognition on at least one frame of image in the target video; obtain audio recognition result of the target video, which is the result obtained by performing audio recognition on background audio of the target video.

[0113] In some embodiments, the content description text includes at least one of content category description text, content form description text, content theme description text, and content detail description text.

[0114] In some embodiments, the content description text includes multiple types of content description text;

[0115] The method further comprises: processing the image feature and the text feature through the self-attention layer included in the text generation sub-model of the video feature extraction model, outputting a plurality of characters whose self-attention scores meet the text generation condition, and generating the content description text of the target video based on the plurality of characters.

[0116] Splicing the image feature with the text feature to obtain a spliced ​​feature;

[0117] On the splicing feature, add the type identification of each type;

[0118] The image feature and the text feature are processed by the self-attention layer included in the text generation sub-model of the video feature extraction model, and a plurality of characters whose self-attention scores meet the text generation condition are outputted. The content description text of the target video is generated based on the plurality of characters, including:

[0119] The splicing features after adding the type identifier are input into the text generation sub-model. The splicing features are processed based on the processing mechanisms corresponding to each type identifier through the self-attention layer included in the text generation sub-model, and a plurality of characters whose self-attention scores meet the text generation conditions are output. Based on the plurality of characters, multiple types of content description texts of the target video are generated.

[0120] In some embodiments, the training process of the video feature extraction model includes:

[0121] In the i-th iteration process of model training, the image information and text information of the sample video are input into the video feature extraction model determined in the i-1-th iteration process to obtain the text training result of the i-th iteration process, where i is a positive integer greater than 1;

[0122] Determine a model loss value of the i-th iteration process based on the text training result of the i-th iteration process and the description text label of the sample video;

[0123] Based on the model loss value of the i-th iteration process, the model parameters of the video feature extraction model determined in the i-1-th iteration process are adjusted, and the i+1-th iteration process is performed based on the adjusted model parameters. The above training iteration process is repeated until the training meets the target conditions.

[0124] In some embodiments, the content description text includes multiple types of content description text;

[0125] Based on the text training result of the i-th iteration process and the description text label of the sample video, determining the model loss value of the i-th iteration process includes:

[0126] For any type, based on the text training result of the i-th iteration process on the type and the descriptive text label of the sample video on the type, determine the loss value of the i-th iteration process on the type;

[0127] Based on the loss values ​​of the i-th iteration process on the multiple types and the weight coefficients of the video feature extraction network on the multiple types, a weighted sum is performed to obtain the model loss value of the i-th iteration process.

[0128] In some embodiments, before performing weighted summation based on the loss values ​​of the i-th iteration process on the multiple types and the weight coefficients of the video feature extraction network on the multiple types to obtain the model loss value of the i-th iteration process, the method further includes:

[0129] For any type, based on the number of correct texts in the i-th iteration process on the type and the total number of texts, determine the correct proportion of the i-th iteration process on the type, where the correct proportion represents the proportion of the number of correct texts to the total number of texts in the i-th iteration process;

[0130] Based on the correct proportion of the i-th iteration process on the type, the weight coefficient of the video feature extraction network on the type is determined, wherein the correct proportion is negatively correlated with the weight coefficient.

[0131] In some embodiments, the number of the text information is multiple;

[0132] Before extracting features from the text information through the text feature extraction submodel of the video feature extraction model to obtain text features of the target video, the method further includes:

[0133] Splicing multiple pieces of text information to obtain the spliced ​​text information;

[0134] Based on the spliced ​​text information, the step of extracting features from the text information through the text feature extraction submodel of the video feature extraction model is executed to obtain text features of the target video.

[0135] In some embodiments, after splicing the plurality of text messages to obtain the spliced ​​text messages, the method further includes:

[0136] Extract the target number of characters from the concatenated text information;

[0137] Based on the extracted characters, the step of extracting features from the text information through the text feature extraction submodel of the video feature extraction model is executed to obtain text features of the target video.

[0138] Above Figure 2 The following is only a basic process of the present disclosure. The solution provided by the present disclosure is further described based on a specific implementation method. Figure 3 is a flowchart of a text generation method according to an exemplary embodiment. Figure 3 , the method comprising:

[0139] In step 301, a computer device obtains image information and text information of a target video.

[0140] In the embodiments of the present disclosure, a target video is used to refer to a video to be described, that is, a video for which a content description text is to be generated. The content description text refers to a sentence used to describe the content of the target video. In some embodiments, taking the computer device as a terminal as an example, the target video is a video stored locally in the terminal, or a video downloaded by the terminal, etc.; in other embodiments, taking the computer device as a server as an example, the target video is a video in a video database associated with the server, or a video uploaded by the terminal, etc. The embodiments of the present disclosure do not limit the source of the target video.

[0141] In some embodiments, the process of a computer device acquiring image information of a target video includes at least one of the following: acquiring a cover image of the target video; or acquiring at least one frame of image within the target video.

[0142] The number of the at least one frame of images may be one frame, two frames, or more than two frames, such as three frames. In some embodiments, the process of the computer device acquiring the at least one frame of images is: randomly extracting a preset number of images from the multiple frames of images included in the target video, such as randomly extracting three frames of images; or uniformly extracting a preset number of images from the multiple frames of images included in the target video, such as extracting three frames of images at equal intervals; extracting key frame images from the multiple frames of images included in the target video. Of course, the computer device can also acquire at least one frame of images in the target video based on other methods, which is not limited in the embodiments of the present disclosure.

[0143] In the above embodiments, by acquiring the cover image of the target video or the image frames included in the target video, the image information of the target video can be quickly acquired. While ensuring the efficiency of acquiring the image information, the types of image information are enriched and the flexibility of acquiring the image information is improved.

[0144] In some embodiments, the process of a computer device obtaining text information of a target video includes at least one of the following: obtaining description information of the target video; obtaining title information of the target video; obtaining subtitle information of the target video; obtaining text recognition results of the target video; and obtaining audio recognition results of the target video.

[0145] Among them, the description information is information used to describe the video theme of the target video, such as theme description information or topic description information (caption); or, the description information is information used to describe the video content of the target video, such as content description information (hashtag). Title information refers to the video title (title) of the target video. In some embodiments, the description information and title information are information set by the publisher of the target video. For example, when the publisher of the target video publishes the target video, the terminal corresponding to the publisher provides a description information entry box and a title information entry box. Through the description information entry box and the title information entry box, the publisher can set the description information and title information of the target video. Then, when the computer device obtains the target video, it can also obtain the description information and title information of the target video.

[0146] Subtitle information refers to the subtitles (text) included in the image in the target video. In some embodiments, the computer device uses a subtitle extraction tool to extract the subtitle information of the target video. The text recognition result is the result obtained by performing text recognition on at least one frame of the target video. In some embodiments, the computer device uses OCR (Optical Character Recognition) technology to perform text recognition on multiple frames of the target video to obtain the text recognition result of the target video. The audio recognition result is the result obtained by performing audio recognition on the background audio of the target video. In some embodiments, the computer device uses ASR (Automatic Speech Recognition) technology to perform audio recognition on the background audio of the target video to obtain the audio recognition result of the target video.

[0147] In the above embodiments, by obtaining the description information, title information, subtitle information, text recognition results or audio recognition results of the target video, the text information of the target video can be quickly obtained. While ensuring the efficiency of obtaining the text information, the types of text information are enriched and the flexibility of obtaining text information is improved.

[0148] In step 302, the computer device inputs the image information and the text information into a video feature extraction model.

[0149] In the disclosed embodiment, the video feature extraction model provides a function of generating a content description text of the video. In some embodiments, the video feature extraction model is an encoder-decoder architecture based on a self-attention mechanism. Among them, the self-attention mechanism is a mechanism for learning the meaning of features based on the dependency relationship between features. In the self-attention mechanism, for each input feature, the similarity or correlation between the feature and its neighboring features is calculated according to the feature, such as calculating the vector dot product of the two, calculating the vector similarity of the two, or evaluating by introducing an additional neural network, etc., to obtain the calculated score of the feature and its neighboring features, and then using a calculation method such as a softmax function to perform a numerical conversion on the calculated score. In this way, on the one hand, the calculated score can be converted into a probability distribution with a sum of element weights of 1, and normalization is achieved. On the other hand, through the inherent mechanism of the softmax function, the weight of important elements can be more highlighted, and then, the weight of each element is used for weighted summation to output the self-attention score.

[0150] For example, Figure 4 is a block diagram of a video feature extraction model according to an exemplary embodiment. Figure 4 The video feature extraction model includes an image feature extraction sub-model, a text feature extraction sub-model and a text generation sub-model, wherein the encoder of the video feature extraction model includes an image feature extraction sub-model and a text feature extraction sub-model, and the decoder of the video feature extraction model includes a text generation sub-model. Figure 4 The video feature extraction model shown illustrates the text generation method provided by the embodiment of the present disclosure.

[0151] In step 303, feature extraction is performed on the image information using the image feature extraction submodel of the video feature extraction model to obtain image features of the target video.

[0152] In the disclosed embodiment, the image feature extraction submodel is used to extract the image features of the target video. In some embodiments, the image feature extraction submodel is Resnet (residual network) or ViT (Vision Transformer).

[0153] In some embodiments, after the computer device inputs the image information and the text information into a video feature extraction model, the image information is input into an image feature extraction sub-model of the video feature extraction model through the video feature extraction model. The image information of the target video is extracted through the image feature extraction sub-model, and image features of a predetermined dimension can be obtained, such as image features of 512 dimensions (or other number of dimensions).

[0154] In step 304, the text information is subjected to feature extraction by the text feature extraction sub-model of the video feature extraction model to obtain text features of the target video.

[0155] In the disclosed embodiment, the text feature extraction sub-model is used to extract the text features of the target video. In some embodiments, the text feature extraction sub-model is Bert (Bidirectional Encoder Representations from Transformers, language representation model) and the like.

[0156] In some embodiments, after the computer device inputs the image information and the text information into the video feature extraction model, the computer device inputs the text information into the text feature extraction submodel of the video feature extraction model through the video feature extraction model, and extracts the text information of the target video through the text feature extraction submodel, so as to obtain text features of a predetermined dimension, such as text features of 512 dimensions (or other number of dimensions). It should be noted that the dimension of the text feature is the same as the dimension of the image feature.

[0157] In some embodiments, the number of text information is multiple (such as two or more), and accordingly, before extracting features from the text information, the method further includes: splicing the multiple text information to obtain spliced ​​text information, and executing the above step 304 based on the spliced ​​text information. By way of example, taking the five text information shown in step 301 as an example, the description information, title information, subtitle information, text recognition results, and audio recognition results of the target video are spliced ​​to obtain spliced ​​text information, and then the spliced ​​text information is input into the above text feature extraction submodel to execute the above step 304. In this embodiment, when the number of text information is multiple, the text information is spliced ​​to obtain the spliced ​​text information, and then the spliced ​​text information is used to extract text features, which refers to multiple types of text information and improves the accuracy of extracting text features.

[0158] Furthermore, in some embodiments, after obtaining the spliced ​​text information, the method further includes: extracting a target number of characters from the spliced ​​text information, and executing the above step 304 based on the extracted characters. The target number is a preset fixed number, such as 200. In this embodiment, the target number of characters is extracted from the spliced ​​text information, so that the subsequent process of extracting text features is performed based on the extracted certain number of characters. On the basis of ensuring that sufficient text information is input, the amount of computation of the video feature extraction model is reduced, and the efficiency of extracting text features is improved.

[0159] It should be noted that the above steps 303 to 304 take the extraction of image features first and then text features as an example. In other embodiments, the computer device can also extract text features first and then image features, or the computer device can also execute the process of extracting image features and extracting text features at the same time. The embodiments of the present disclosure do not limit the execution order of steps 303 and 304.

[0160] In step 305, the image feature and the text feature are processed through the self-attention layer included in the text generation sub-model of the video feature extraction model, and a plurality of characters whose self-attention scores meet the text generation conditions are output, and the content description text of the target video is generated based on the plurality of characters.

[0161] The text generation sub-model is used to generate a content description text of the target video. In some embodiments, the content description text includes multiple types of content description texts, for example, the content description text includes at least one of content category description text, content form description text, content theme description text, and content detail description text.

[0162] In some embodiments, the content category description text includes multiple levels of category description text, for example, a primary category description text and a secondary category description text, wherein the primary category description text refers to a sentence for describing the primary category of the target video, and the secondary category description text refers to a sentence for describing the secondary category of the target video. It should be understood that the primary category refers to the general classification of the target video, and the secondary category refers to the subclassification of the target video based on the primary category, wherein the secondary category is a tree structure relative to the primary category, that is, the primary category will include multiple secondary categories. For example, the primary category can be a life category, and the secondary category can be a life record, good things sharing, or health and wellness, etc., or the primary category can be a beauty category, and the secondary category can be a beauty tutorial, beauty evaluation, skin care, etc.

[0163] The content form description text refers to a sentence used to describe the content form of the target video, and the content form indicates the presentation form (or shooting form) of the target video. For example, the content form can be a documentary form, a situational skit form, or a street interview form, etc. The content theme description text refers to a sentence used to describe the content theme of the target video, for example, the content theme can be a video topic. The content details description text refers to a sentence used to describe the content details of the target video, for example, a sentence used to describe the screen content of the target video.

[0164] In the above embodiment, based on the video feature extraction model, multiple types of content description texts can be obtained. On the one hand, more expressive content description texts are generated. On the other hand, the multiple types of content description texts can describe the content of the target video from different dimensions, enriching the types of generated content description texts and being able to more fully and completely characterize the target video.

[0165] In some embodiments, the text generation submodel includes a multi-layer self-attention layer (Transformer layer). In some embodiments, the image feature and the text feature are input into the text generation submodel of the video feature extraction model through the video feature extraction model, and the image feature and the text feature are processed through the multi-layer self-attention layer included in the text generation submodel, and a plurality of characters whose self-attention scores reach the text generation condition are output, and the content description text of the target video is generated based on the outputted plurality of characters. By way of example, the text generation condition may be that the self-attention score reaches a score threshold.

[0166] In an optional embodiment, the text generation sub-model is used to perform multiple types of text generation tasks. Through the multi-layer self-attention layers included in the text generation sub-model, the image features and the text features are processed according to the processing mechanisms of the text generation tasks corresponding to the multiple types, so as to generate multiple types of content description texts for the target video. Further, for any type, in an optional embodiment, the first feature sequence of image features and text features is processed through the multi-layer self-attention layer included in the text generation sub-model, and the characters whose self-attention scores in the first feature sequence meet the text generation conditions are output; based on the characters output in the first feature sequence, the second feature sequence of image features and text features is continued to be processed, and the characters whose self-attention scores in the second feature sequence meet the text generation conditions are output; based on the characters output in the first feature sequence and the characters output in the second feature sequence, the third feature sequence of image features and text features is continued to be processed, and the characters whose self-attention scores in the third feature sequence meet the text generation conditions are output, and then, based on the output characters, the above processing and output processes are repeated, and the characters whose self-attention scores in the next feature sequence meet the text generation conditions are output, so as to obtain multiple characters output by the text generation sub-model, and the multiple characters are spliced ​​according to the output timing of the multiple characters to obtain the content description text of the target video.

[0167] In some embodiments, the text generation sub-model also provides a mask mechanism, which refers to masking features that do not need to be paid attention to in a feature sequence to avoid affecting the video feature extraction model, thereby improving the accuracy of the self-attention mechanism.

[0168] In some embodiments, before the image feature and the text feature are input into the text generation sub-model, the image feature and the text feature are spliced ​​to obtain a spliced ​​feature, and the above step 305 is performed based on the spliced ​​feature. In some embodiments, the image feature and the text feature are spliced ​​through the feature splicing layer (concat) of the video feature extraction model, and the spliced ​​feature is output. In this embodiment, by splicing the image feature and the text feature to obtain the spliced ​​feature, and then using the spliced ​​feature to perform the text generation process, the text generation sub-model in the video feature extraction model not only refers to the image features of the target video, but also refers to the text features of the target video, thereby increasing the amount of information referenced by the video feature extraction model and improving the accuracy of text generation.

[0169] Further, in some embodiments, when the text generation submodel is used to perform multiple types of text generation tasks, after obtaining the splicing feature, the method further includes: adding each type of type identification to the splicing feature, and performing the above step 305 based on the splicing feature after adding the type identification. Wherein, the type identification is used to indicate the text generation task of the corresponding type. In some embodiments, the splicing feature after adding the type identification is input into the text generation submodel through the video feature extraction model, and the splicing feature is processed based on the processing mechanism corresponding to each type identification through the self-attention layer included in the text generation submodel, and the multiple characters whose self-attention scores reach the text generation condition are output, and the multiple types of content description texts of the target video are generated based on the multiple characters. In this embodiment, by adding each type of type identification to the splicing feature, the text generation submodel in the video feature extraction model can trigger the generation of multiple types of content description texts of the target video based on the type identification of each type, so as to ensure the smooth progress of text generation.

[0170] The technical solution provided by the embodiments of the present disclosure can accurately extract the image features and text features of the target video by constructing a dual-stream architecture of an image feature extraction submodel and a text feature extraction submodel in a video feature extraction model, and can process the image features and text features of the target video by setting a text generation submodel in the video feature extraction model, and can output multiple characters whose self-attention scores meet the text generation conditions, and then can automatically generate a content description text of the target video based on the output multiple characters, thereby providing a video feature extraction model based on text generation, and the generated content description text contains a rich amount of information, can better characterize the target video, and improves the accuracy of video characterization.

[0171] In the above Figure 3In the scheme shown, a model for generating a content description text of a video based on image information and text information is provided, which refers to the multimodal information of the image modality and the text model, and increases the amount of information referenced by the video feature extraction model. On the one hand, it enriches the video features extracted by the video feature extraction model, and on the other hand, it improves the accuracy of the video feature extraction model in extracting video features. Among them, modality refers to the way or form of representation of information. It should be understood that each medium or form of information can be called a modality, for example, the medium of information, such as image, text, audio, etc. In other embodiments, the computer device can also use information of other modalities to perform the above-mentioned text generation method, for example, the release information of the target video, etc.

[0172] Regarding the above Figure 3 In the illustrated embodiment, the video feature extraction model, in some embodiments, before implementing the above-mentioned scheme of steps 301 to 305, the computer device performs model training based on the image information of the sample video, the text information of the sample video, and the descriptive text label of the sample video to obtain the video feature extraction model. In the disclosed embodiment, the sample video is used to refer to the training video used to train the video feature extraction model. In some embodiments, the number of sample videos is multiple. It should be understood that the image information of the sample video, the text information of the sample video, and the descriptive text label of the sample video are obtained as a training data set. Among them, the image information of the sample video includes at least one of the cover image of the sample video or at least one frame of the sample video. The text information of the sample video includes at least one of the description information, title information, subtitle information, text recognition results, and audio recognition results of the sample video. It should be noted that the process of obtaining the image information and text information of the sample video refers to the process of obtaining the image information and text information of the target video in step 301, which will not be repeated. The descriptive text label is used to indicate the content description text of the sample video. In some embodiments, the descriptive text tag refers to a content description text of a sample video that is manually generated (eg, tagged).

[0173] Figure 5 is a flowchart of a method for training a video feature extraction model according to an exemplary embodiment. Figure 5 , the method comprising:

[0174] In step 501, during the i-th iteration of model training, the computer device inputs the image information and text information of the sample video into the video feature extraction model determined by the i-1-th iteration to obtain the text training result of the i-th iteration, where i is a positive integer greater than 1.

[0175] The text training result is the content description text generated during the model training process. In some embodiments, during the i-th iteration of the model training, the computer device inputs the image information and text information of the sample video into the video feature extraction model determined during the i-1-th iteration, performs feature extraction on the image information through the image feature extraction submodel of the video feature extraction model to obtain the image features of the sample video, performs feature extraction on the text information through the text feature extraction submodel of the video feature extraction model to obtain the text features of the sample video, and processes the image features and the text features through the self-attention layer included in the text generation submodel of the video feature extraction model to obtain the text training result of the sample video.

[0176] In some embodiments, during the first iteration of model training, the image information and text information of the sample video are input into the initial video feature extraction model to obtain the text training result of the first iteration, wherein the processing process of the initial video feature extraction model is as follows: the image information is subjected to feature extraction through the image feature extraction submodel of the initial video feature extraction model to obtain the image features of the sample video; the text information is subjected to feature extraction through the text feature extraction submodel of the initial video feature extraction model to obtain the text features of the sample video; the image features and the text features are processed through the self-attention layer included in the text generation submodel of the initial video feature extraction model to obtain the text training result of the sample video.

[0177] In step 502, for any type, the computer device determines the loss value of the i-th iteration process on the type based on the text training result of the i-th iteration process on the type and the descriptive text label of the sample video on the type.

[0178] In some embodiments, for any type, the computer device determines the cross entropy loss value of the i-th iterative process on the type based on the text training results of the i-th iterative process on the type and the descriptive text label of the sample video on the type, and then performs subsequent steps based on the cross entropy loss value of the i-th iterative process on the type.

[0179] In some embodiments, for any type, the process of obtaining the cross entropy loss value of the type is: the computer device determines the cross entropy loss value of the i-th iterative process on the type based on the text training results of the i-th iterative process on the type, the descriptive text label of the sample video on the type, the number of sample videos and the CEloss (Cross Entropy Loss) formula (1).

[0180]

[0181] In the formula, CEloss represents the cross entropy loss value; m represents the number of sample videos in the training dataset; y k represents the text training result of the video feature extraction model for the kth sample video; p represents the correct probability of the text training result of the video feature extraction model for the kth sample video. For example, the correct probability can be the similarity between the text training result and the descriptive text label.

[0182] In step 503, for any type, the computer device determines the correct proportion of the i-th iteration process on the type based on the number of correct texts in the i-th iteration process on the type and the total number of texts, and the correct proportion represents the ratio of the number of correct texts to the total number of texts in the i-th iteration process.

[0183] Among them, the number of correct texts refers to the number of correct text training results generated during the model training process, for example, the number of text training results whose correct probability reaches the probability threshold. The total number of texts refers to the total number of text training results generated during the model training process. In some embodiments, for any type, the computer device determines the ratio of the number of correct texts to the total number of texts in the i-th iteration process on the type, and determines the determined ratio as the correct proportion of the i-th iteration process on the type.

[0184] In step 504, the computer device determines the weight coefficient of the video feature extraction network on the type based on the correct proportion of the i-th iteration process on the type, wherein the correct proportion is negatively correlated with the weight coefficient.

[0185] Regarding the process of determining the weight coefficients corresponding to each type in the above steps 503 to 504, in some embodiments, the computer device determines the weight coefficient of the video feature extraction network on the type based on the number of correct texts on the type in the i-th iteration process, the total number of texts on the type in the i-th iteration process, and the following weight coefficient formula (2).

[0186] W=1-(correct / total) (2)

[0187] In the formula, W represents the weight coefficient of the video feature extraction network on this type; correct represents the number of correct texts on this type; and total represents the total number of texts on this type.

[0188] In the above embodiment, for each type involved in text generation, the weight coefficient corresponding to each type is determined according to the correct proportion corresponding to each type. Since the correct proportion represents the ratio of the number of correct texts to the total number of texts, and since the correct proportion is negatively correlated with the weight coefficient, when calculating the model loss value, a smaller weight coefficient is set for the type with a large correct proportion, and a larger weight coefficient is set for the type with a small correct proportion, thereby improving the accuracy of determining the weight coefficient and thus improving the accuracy of determining the model loss value.

[0189] In step 505, the computer device performs weighted summation based on the loss values ​​of the i-th iteration process on the multiple types and the weight coefficients of the video feature extraction network on the multiple types to obtain the model loss value of the i-th iteration process.

[0190] For the i-th iteration process, in some embodiments, based on the above CEloss formula (1), the cross entropy loss value corresponding to the at least one type is calculated, and based on the above weight coefficient formula (2), the weight coefficient corresponding to the at least one type is calculated. Then, based on the cross entropy loss value corresponding to the at least one type, the weight coefficient corresponding to the at least one type and the following loss value formula (3), a weighted sum is performed to obtain the model loss value of the i-th iteration process.

[0191]

[0192] In the formula, Totalloss represents the model loss value of the i-th iteration process; n represents the number of multiple types; W s Indicates the weight coefficient corresponding to type s; CEloss s Represents the cross entropy loss value corresponding to type s.

[0193] In the above steps 502 to 505, the computer device determines the model loss value of the i-th iteration process based on the text training results of the i-th iteration process on multiple types and the descriptive text labels of the sample video on multiple types. In this embodiment, for each type involved in text generation, a weight coefficient corresponding to each type is set respectively, and then the model loss value is determined by using the loss value corresponding to each type and the weight coefficient corresponding to each type, thereby improving the accuracy of determining the model loss value.

[0194] In step 506, the computer device adjusts the model parameters of the video feature extraction model determined in the i-1th iteration process based on the model loss value of the i-th iteration process, performs the i+1th iteration process based on the adjusted model parameters, and repeats the above training iteration process until the training meets the target conditions.

[0195] In some embodiments, the target condition satisfied by the training is that the number of training iterations of the video feature extraction model reaches the target number, which is a preset number of training iterations, such as 1000 times; or, the target condition satisfied by the training is that the model loss value meets the target threshold condition, such as the loss value is less than 0.00001. The present disclosure embodiment does not limit the setting of the target condition.

[0196] In the above embodiment, in any iteration process of model training, the model loss value of this iteration process is used to adjust the model parameters of the video feature extraction model to improve the feature extraction capability of the video feature extraction model, thereby training a video feature extraction model with higher feature extraction capability.

[0197] The technical solution provided by the embodiment of the present disclosure utilizes the image information of a sample video, the text information of a sample video, and the descriptive text label of the sample video to perform model training, so as to obtain a video feature extraction model with a text generation function through training, and then utilizes the trained video feature extraction model to generate a content description text of a target video. A video feature extraction model based on text generation is provided, and the generated content description text contains a rich amount of information, which can better characterize the target video and improve the accuracy of video characterization.

[0198] Figure 6 is a block diagram of a text generation device according to an exemplary embodiment. Figure 6 The device includes an acquisition unit 601, an extraction unit 602 and a processing unit 603.

[0199] An acquisition unit 601 is configured to acquire image information and text information of a target video;

[0200] The extraction unit 602 is configured to input the image information and the text information into a video feature extraction model, perform feature extraction on the image information through an image feature extraction submodel of the video feature extraction model to obtain image features of the target video, and perform feature extraction on the text information through a text feature extraction submodel of the video feature extraction model to obtain text features of the target video;

[0201] The processing unit 603 is configured to execute the self-attention layer included in the text generation sub-model of the video feature extraction model, process the image feature and the text feature, output a plurality of characters whose self-attention scores meet the text generation conditions, and generate the content description text of the target video based on the plurality of characters.

[0202] The technical solution provided by the embodiments of the present disclosure can accurately extract the image features and text features of the target video by constructing a dual-stream architecture of an image feature extraction submodel and a text feature extraction submodel in a video feature extraction model, and can process the image features and text features of the target video by setting a text generation submodel in the video feature extraction model, and can output multiple characters whose self-attention scores meet the text generation conditions, and then can automatically generate a content description text of the target video based on the output multiple characters, thereby providing a video feature extraction model based on text generation, and the generated content description text contains a rich amount of information, can better characterize the target video, and improves the accuracy of video characterization.

[0203] In some embodiments, the acquisition unit 601 is configured to perform at least one of the following:

[0204] Obtain a cover image of the target video; or obtain at least one frame of image in the target video.

[0205] In some embodiments, the acquisition unit 601 is configured to perform at least one of the following:

[0206] Obtain description information of the target video; obtain title information of the target video; obtain subtitle information of the target video; obtain text recognition result of the target video, which is the result obtained by performing text recognition on at least one frame of image in the target video; obtain audio recognition result of the target video, which is the result obtained by performing audio recognition on background audio of the target video.

[0207] In some embodiments, the content description text includes at least one of content category description text, content form description text, content theme description text, and content detail description text.

[0208] In some embodiments, the content description text includes multiple types of content description text;

[0209] The device also includes:

[0210] A first splicing unit is configured to perform splicing of the image feature and the text feature to obtain a splicing feature;

[0211] An adding unit is configured to add type identifiers of various types to the splicing feature;

[0212] The processing unit 603 is configured to execute:

[0213] The splicing features after adding the type identifier are input into the text generation sub-model. The splicing features are processed based on the processing mechanisms corresponding to each type identifier through the self-attention layer included in the text generation sub-model, and a plurality of characters whose self-attention scores meet the text generation conditions are output. Based on the plurality of characters, multiple types of content description texts of the target video are generated.

[0214] In some embodiments, the apparatus further comprises:

[0215] An input unit is configured to input the image information and text information of the sample video into the video feature extraction model determined in the i-1th iteration process during the i-th iteration process of the model training, and obtain the text training result of the i-th iteration process, where i is a positive integer greater than 1;

[0216] A determination unit is configured to determine a model loss value of the i-th iteration process based on the text training result of the i-th iteration process and the description text label of the sample video;

[0217] The adjustment unit is configured to adjust the model parameters of the video feature extraction model determined in the i-1th iteration process based on the model loss value of the i-th iteration process, perform the i+1th iteration process based on the adjusted model parameters, and repeat the above training iteration process until the training meets the target conditions.

[0218] In some embodiments, the content description text includes multiple types of content description text;

[0219] The determination unit is configured to execute:

[0220] For any type, based on the text training result of the i-th iteration process on the type and the descriptive text label of the sample video on the type, determine the loss value of the i-th iteration process on the type;

[0221] Based on the loss values ​​of the i-th iteration process on the multiple types and the weight coefficients of the video feature extraction network on the multiple types, a weighted sum is performed to obtain the model loss value of the i-th iteration process.

[0222] In some embodiments, the determining unit is further configured to execute:

[0223] For any type, based on the number of correct texts in the i-th iteration process on the type and the total number of texts, determine the correct proportion of the i-th iteration process on the type, where the correct proportion represents the proportion of the number of correct texts to the total number of texts in the i-th iteration process;

[0224] Based on the correct proportion of the i-th iteration process on the type, the weight coefficient of the video feature extraction network on the type is determined, wherein the correct proportion is negatively correlated with the weight coefficient.

[0225] In some embodiments, the number of the text information is multiple;

[0226] The device also includes a second splicing unit configured to splice the plurality of text messages to obtain the spliced ​​text messages;

[0227] The extraction unit 602 is further configured to execute, based on the spliced ​​text information, a step of extracting features from the text information through a text feature extraction sub-model of the video feature extraction model to obtain text features of the target video.

[0228] In some embodiments, the extracting unit 602 is further configured to perform:

[0229] Extract the target number of characters from the concatenated text information;

[0230] Based on the extracted characters, the step of extracting features from the text information through the text feature extraction submodel of the video feature extraction model is executed to obtain text features of the target video.

[0231] It should be noted that: the text generation device provided in the above embodiment only uses the division of the above functional modules as an example when generating text. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the text generation device provided in the above embodiment and the text generation method embodiment belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0232] The computer device mentioned in the embodiments of the present disclosure may be provided as a terminal. Figure 7 The structure block diagram of a terminal 700 provided by an exemplary embodiment of the present disclosure is shown. The terminal 700 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer or a desktop computer. The terminal 700 may also be called a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0233] Typically, the terminal 700 includes a processor 701 and a memory 702 .

[0234] The processor 701 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 701 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 701 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 701 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0235] The memory 702 may include one or more computer-readable storage media, which may be non-transitory. The memory 702 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 702 is used to store at least one program code, which is used to be executed by the processor 701 to implement the process executed by the terminal in the text generation method provided in the method embodiment of the present disclosure.

[0236] In some embodiments, the terminal 700 may further optionally include: a peripheral device interface 703 and at least one peripheral device. The processor 701, the memory 702 and the peripheral device interface 703 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 703 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 704, a display screen 705, a camera assembly 706, an audio circuit 707, a positioning assembly 708 and a power supply 709.

[0237] The peripheral device interface 703 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 701 and the memory 702. In some embodiments, the processor 701, the memory 702, and the peripheral device interface 703 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 701, the memory 702, and the peripheral device interface 703 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0238] The radio frequency circuit 704 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 704 communicates with communication networks and other communication devices through electromagnetic signals. The radio frequency circuit 704 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 704 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 704 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 704 may also include circuits related to NFC (Near Field Communication), which is not limited in the present disclosure.

[0239] The display screen 705 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 705 is a touch display screen, the display screen 705 also has the ability to collect touch signals on the surface or above the surface of the display screen 705. The touch signal can be input to the processor 701 as a control signal for processing. At this time, the display screen 705 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 705 can be one, set on the front panel of the terminal 700; in other embodiments, the display screen 705 can be at least two, respectively set on different surfaces of the terminal 700 or in a folding design; in other embodiments, the display screen 705 can be a flexible display screen, set on a curved surface or a folding surface of the terminal 700. Even, the display screen 705 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 705 can be made of materials such as LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0240] The camera assembly 706 is used to capture images or videos. Optionally, the camera assembly 706 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 706 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0241] The audio circuit 707 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 701 for processing, or input them into the radio frequency circuit 704 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 700. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 701 or the radio frequency circuit 704 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 707 may also include a headphone jack.

[0242] The positioning component 708 is used to locate the current geographical location of the terminal 700 to implement navigation or LBS (Location Based Service).

[0243] The power supply 709 is used to power various components in the terminal 700. The power supply 709 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 709 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0244] In some embodiments, the terminal 700 further includes one or more sensors 710 , including but not limited to: an acceleration sensor 711 , a gyroscope sensor 712 , a pressure sensor 713 , a fingerprint sensor 714 , an optical sensor 715 , and a proximity sensor 716 .

[0245] The acceleration sensor 711 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 700. For example, the acceleration sensor 711 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 701 can control the display screen 705 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 711. The acceleration sensor 711 can also be used to collect motion data of games or users.

[0246] The gyro sensor 712 can detect the body direction and rotation angle of the terminal 700, and the gyro sensor 712 can cooperate with the acceleration sensor 711 to collect the user's 3D actions on the terminal 700. The processor 701 can implement the following functions based on the data collected by the gyro sensor 712: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0247] The pressure sensor 713 can be set on the side frame of the terminal 700 and / or the lower layer of the display screen 705. When the pressure sensor 713 is set on the side frame of the terminal 700, it can detect the user's holding signal of the terminal 700, and the processor 701 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 713. When the pressure sensor 713 is set on the lower layer of the display screen 705, the processor 701 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 705. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0248] The optical sensor 714 is used to collect the ambient light intensity. In one embodiment, the processor 701 can control the display brightness of the display screen 705 according to the ambient light intensity collected by the optical sensor 714. Specifically, when the ambient light intensity is high, the display brightness of the display screen 705 is increased; when the ambient light intensity is low, the display brightness of the display screen 705 is reduced. In another embodiment, the processor 701 can also dynamically adjust the shooting parameters of the camera assembly 706 according to the ambient light intensity collected by the optical sensor 714.

[0249] The proximity sensor 715, also called a distance sensor, is usually arranged on the front panel of the terminal 700. The proximity sensor 715 is used to collect the distance between the user and the front of the terminal 700. In one embodiment, when the proximity sensor 715 detects that the distance between the user and the front of the terminal 700 is gradually decreasing, the processor 701 controls the display screen 705 to switch from the screen-on state to the screen-off state; when the proximity sensor 715 detects that the distance between the user and the front of the terminal 700 is gradually increasing, the processor 701 controls the display screen 705 to switch from the screen-off state to the screen-on state.

[0250] Those skilled in the art will understand that Figure 7 The structure shown in the figure does not constitute a limitation on the terminal 700, and the terminal 700 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0251] The computer device mentioned in the embodiments of the present disclosure may be provided as a server. Figure 8It is a block diagram of a server according to an exemplary embodiment. The server 800 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 801 and one or more memories 802, wherein the one or more memories 802 store at least one program code, and the at least one program code is loaded and executed by the one or more processors 801 to implement the process executed by the server in the text generation method provided by the above-mentioned various method embodiments. Of course, the server 800 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 800 may also include other components for implementing device functions, which will not be repeated here.

[0252] In an exemplary embodiment, a computer-readable storage medium including a program code is also provided, such as a memory 802 including the program code, and the program code can be executed by a processor 801 of a server 800 to complete the above-mentioned text generation method. Optionally, the computer-readable storage medium can be a ROM (Read-Only Memory), a RAM (Random Access Memory), a CD-ROM (Compact-Disc Read-Only Memory), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0253] In an exemplary embodiment, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the above-mentioned text generation method is implemented.

[0254] In some embodiments, the computer program involved in the embodiments of the present disclosure may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected through a communication network. Multiple computer devices distributed at multiple locations and interconnected through a communication network may constitute a blockchain system.

[0255] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0256] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A text generation method, characterized in that: The method comprises: Obtain image information and text information of the target video; Inputting the image information and the text information into a video feature extraction model, performing feature extraction on the image information through the image feature extraction submodel of the video feature extraction model to obtain image features of the target video, and performing feature extraction on the text information through the text feature extraction submodel of the video feature extraction model to obtain text features of the target video; Splicing the image feature with the text feature to obtain a spliced ​​feature; Adding each type of type identification to the splicing feature, wherein the type identification indicates a text generation task of a corresponding type; Input the splicing features after adding the type identifier into the text generation sub-model of the video feature extraction model, process the splicing features based on the processing mechanisms corresponding to the various type identifiers through the self-attention layer included in the text generation sub-model, output a plurality of characters whose self-attention scores meet the text generation conditions, and generate a plurality of types of content description texts of the target video based on the plurality of characters, wherein the text generation sub-model includes a multi-layer attention layer for performing a plurality of types of text generation tasks, and the content description text includes at least one of a content category description text, a content form description text, a content theme description text, and a content detail description text; The training process of the video feature extraction model includes: In the i-th iteration process of model training, the image information and text information of the sample video are input into the video feature extraction model determined in the i-1-th iteration process to obtain the text training result of the i-th iteration process, where i is a positive integer greater than 1; Based on the text training results of the i-th iteration process and the descriptive text labels of the sample video, determine the model loss value of the i-th iteration process; wherein, for any type, based on the text training results of the i-th iteration process on the type and the descriptive text labels of the sample video on the type, determine the loss value of the i-th iteration process on the type; based on the number of correct texts and the total number of texts on the type in the i-th iteration process, determine the correct proportion of the i-th iteration process on the type, the correct proportion represents the proportion of the number of correct texts to the total number of texts in the i-th iteration process; based on the correct proportion of the i-th iteration process on the type, determine the weight coefficient of the video feature extraction network on the type, wherein the correct proportion is negatively correlated with the weight coefficient; based on the loss values ​​of the i-th iteration process on the multiple types and the weight coefficients of the video feature extraction network on the multiple types, perform weighted summation to obtain the model loss value of the i-th iteration process; Based on the model loss value of the i-th iteration process, the model parameters of the video feature extraction model determined by the i-1-th iteration process are adjusted, and the i+1-th iteration process is performed based on the adjusted model parameters, and the iterative process of the above training is repeated until the training meets the target conditions.

2. The text generation method according to claim 1, characterized in that: The number of the text information is multiple; Before extracting features from the text information through the text feature extraction submodel of the video feature extraction model to obtain text features of the target video, the method further includes: splicing a plurality of the text information to obtain the spliced ​​text information; Based on the spliced ​​text information, the step of extracting features from the text information through the text feature extraction sub-model of the video feature extraction model is performed to obtain text features of the target video.

3. The text generation method according to claim 2, characterized in that: After the plurality of text messages are spliced ​​together to obtain the spliced ​​text message, the method further includes: Extracting a target number of characters from the concatenated text information; Based on the extracted characters, the step of extracting features from the text information through the text feature extraction sub-model of the video feature extraction model is performed to obtain text features of the target video.

4. A text generation device, characterized in that: The device comprises: An acquisition unit is configured to acquire image information and text information of a target video; an extraction unit configured to input the image information and the text information into a video feature extraction model, perform feature extraction on the image information through an image feature extraction submodel of the video feature extraction model to obtain image features of the target video, and perform feature extraction on the text information through a text feature extraction submodel of the video feature extraction model to obtain text features of the target video; A first splicing unit is configured to perform splicing on the image feature and the text feature to obtain a splicing feature; An adding unit is configured to add type identifiers of various types to the splicing features, wherein the type identifiers indicate text generation tasks of corresponding types; The processing unit is configured to execute inputting the splicing features after adding the type identifier into the text generation sub-model of the video feature extraction model, processing the splicing features based on the processing mechanisms corresponding to the respective type identifiers through the self-attention layer included in the text generation sub-model, outputting a plurality of characters whose self-attention scores meet the text generation conditions, and generating a plurality of types of content description texts of the target video based on the plurality of characters, wherein the text generation sub-model includes a multi-layer attention layer for performing a plurality of types of text generation tasks, and the content description text includes at least one of a content category description text, a content form description text, a content theme description text, and a content detail description text; The device also includes: An input unit is configured to input the image information and text information of the sample video into the video feature extraction model determined in the i-1th iteration process during the i-th iteration process of the model training, and obtain the text training result of the i-th iteration process, where i is a positive integer greater than 1; A determination unit is configured to determine the model loss value of the i-th iteration process based on the text training result of the i-th iteration process and the descriptive text label of the sample video; the determination unit is configured to execute: For any type, based on the text training result of the i-th iteration process on the type and the descriptive text label of the sample video on the type, determine the loss value of the i-th iteration process on the type; based on the number of correct texts and the total number of texts on the type in the i-th iteration process, determine the correct proportion of the i-th iteration process on the type, the correct proportion represents the proportion of the number of correct texts to the total number of texts in the i-th iteration process; based on the correct proportion of the i-th iteration process on the type, determine the weight coefficient of the video feature extraction network on the type, wherein the correct proportion is negatively correlated with the weight coefficient; Based on the loss values ​​of the i-th iteration process on the multiple types and the weight coefficients of the video feature extraction network on the multiple types, a weighted sum is performed to obtain a model loss value of the i-th iteration process; The adjustment unit is configured to adjust the model parameters of the video feature extraction model determined by the i-1th iteration process based on the model loss value of the i-th iteration process, perform the i+1th iteration process based on the adjusted model parameters, and repeat the iterative process of the above training until the training meets the target conditions.

5. The text generation device according to claim 4, characterized in that: The number of the text information is multiple; The device further includes a second splicing unit configured to splice the plurality of text messages to obtain the spliced ​​text messages; The extraction unit is further configured to execute, based on the spliced ​​text information, the step of extracting features from the text information through the text feature extraction sub-model of the video feature extraction model to obtain text features of the target video.

6. The text generation device according to claim 5, characterized in that: The extraction unit is further configured to execute: Extracting a target number of characters from the concatenated text information; Based on the extracted characters, the step of extracting features from the text information through the text feature extraction sub-model of the video feature extraction model is performed to obtain text features of the target video.

7. A computer device, characterized in that: The computer device comprises: one or more processors; a memory for storing program code executable by the processor; The processor is configured to execute the program code to implement the text generation method as described in any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that: When the program code in the computer-readable storage medium is executed by a processor of a computer device, the computer device is enabled to execute the text generation method according to any one of claims 1 to 3.

Images