Video-assisted text reconstruction model training method and device

Abstract

The application discloses a video auxiliary-based text reconstruction model training method and device, which comprises the following steps: training a to-be-trained text model according to all determined training samples, obtaining a trained text model, and judging whether the trained text model converges; wherein, when the to-be-trained text model is trained, the input content of the feature fusion layer included in the to-be-trained text model comprises a text feature vector corresponding to a sample text in any training sample and a video feature vector corresponding to the sample text and generated in advance; and if yes, the trained text model is determined as a text reconstruction model. It can be seen that the text reconstruction model can be trained with the aid of the video feature vector, which is not only beneficial to the user to quickly produce a video text through the text reconstruction model without repeatedly correcting the video text, but also beneficial to improving the matching degree between the generated video text and the video, thereby meeting the production demand of the user for the video text.

Description

Technical Field

[0001] This invention relates to the field of model training technology, and in particular to a method and apparatus for training a video-assisted text reconstruction model. Background Technology

[0002] With the rapid development of the video production industry, video production is playing an increasingly important role in the domestic advertising market, becoming one of the most commonly used and effective advertising methods for various enterprises. The design of video scripts (such as video text) is indispensable in the video production process. Vivid and engaging video scripts can make the produced videos more creative, thereby creating better marketing results for businesses.

[0003] Currently, video text generation primarily relies on creators editing video text using fixed templates. However, practical experience shows that this method, dependent on manual editing, requires creators to repeatedly revise the text based on their experience, resulting in excessively long production cycles and low matching rates between the generated text and the video. Therefore, providing a method for quickly generating video text that perfectly matches the video is crucial. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and apparatus for training a video-assisted text reconstruction model, which not only helps users to quickly create video text through the text reconstruction model, but also helps to improve the matching degree between the generated video text and the video, thereby meeting the user's needs for video text production.

[0005] To address the aforementioned technical problems, the first aspect of this invention discloses a method for training a video-assisted text reconstruction model, the method comprising:

[0006] Determine a target training sample set; the target training sample set includes a plurality of target training samples, and each target training sample includes at least sample text;

[0007] Based on all the target training samples, a model training operation is performed on the text model to be trained to obtain a trained text model, and it is determined whether the trained text model has converged; wherein, when performing the model training operation on the text model to be trained, for any target training sample, the input content of the feature fusion layer of the text model to be trained includes the text feature vector corresponding to the sample text in the target training sample and the pre-generated video feature vector corresponding to the sample text.

[0008] When the judgment result is yes, the trained text model is determined as the text reconstruction model; the text reconstruction model is used to supplement the target text material to be generated text in order to generate text that matches the target text material.

[0009] As an optional implementation, in the first aspect of the present invention, the step of performing a model training operation on the text model to be trained based on all the target training samples to obtain a trained text model includes:

[0010] All the target training samples are input into the text model to be trained, and a text vector transformation operation is performed on the sample text included in each target training sample through the embedding layer of the text model to be trained to obtain the text feature vector corresponding to each target training sample; each text feature vector corresponding to each target training sample contains the corresponding masked vector content;

[0011] For each of the input target training samples, the feature fusion layer of the text model to be trained performs a fusion operation on the text feature vector corresponding to the target training sample and the pre-generated video feature vector to obtain the fused feature vector corresponding to the target training sample.

[0012] For each of the input target training samples, the prediction and reconstruction layer of the text model to be trained is used to predict and reconstruct the masked vector content in the fused feature vector corresponding to the target training sample, so as to obtain the prediction and reconstruction vector content corresponding to the target training sample.

[0013] As an optional implementation, in the first aspect of the present invention, the step of performing a text vector transformation operation on the sample text included in each target training sample through the embedding layer of the text model to be trained, to obtain a text feature vector corresponding to each target training sample, includes:

[0014] For each of the input target training samples, the embedding layer of the text model to be trained performs a character splitting operation on the sample text in the target training sample to obtain all the target characters of the sample text, and performs a character vector transformation operation on all the target characters of the sample text to obtain all the character feature vectors corresponding to the target training sample.

[0015] For each of the input target training samples, a concatenation operation is performed on all the character feature vectors corresponding to the target training sample to obtain all the sentence feature vectors corresponding to the target training sample, and the undetermined text feature vector corresponding to the target training sample is determined based on all the sentence feature vectors corresponding to the target training sample.

[0016] For each input target training sample, according to the preset mask parameters, the vector content in the undetermined text feature vector corresponding to the target training sample that matches the mask parameters is masked to obtain the text feature vector corresponding to the target training sample.

[0017] As an optional implementation, in the first aspect of the present invention, for each input target training sample, the step of performing a fusion operation between the text feature vector corresponding to the target training sample and the pre-generated video feature vector through the feature fusion layer of the text model to be trained, to obtain the fused feature vector corresponding to the target training sample, includes:

[0018] For each input target training sample, the text feature vector corresponding to the target training sample is concatenated with the pre-generated video feature vector through the feature fusion layer of the text model to be trained, to obtain the concatenated feature vector corresponding to the target training sample. Then, a first-dimensional transformation operation is performed on the concatenated feature vector corresponding to the target training sample to obtain the transformed feature vector corresponding to the target training sample. Based on the pre-determined video feature parameters, a vector averaging operation is performed on the transformed feature vector corresponding to the target training sample to obtain the averaged feature vector corresponding to the target training sample, which is used as the fused feature vector corresponding to the target training sample.

[0019] As an optional implementation, in the first aspect of the present invention, before performing a fusion operation between the text feature vector corresponding to the target training sample and the pre-generated video feature vector through the feature fusion layer of the text model to be trained for each input target training sample to obtain the fused feature vector corresponding to the target training sample, the method further includes:

[0020] Obtain the video feature vector corresponding to the sample text in each of the target training samples pre-generated by the embedding layer;

[0021] Based on the first dimension feature information of the text feature vector corresponding to the sample text in all the target training samples and the second dimension feature information of the corresponding video feature vector, it is determined whether the text feature vector corresponding to all the target training samples matches the corresponding video feature vector; the first dimension feature information of the text feature vector corresponding to the sample text in each target training sample and the second dimension feature information of the corresponding video feature vector each include at least one of the following: dimension type, number of dimensions, and dimension order.

[0022] When the judgment result is negative, all video feature vectors to be processed that do not match the corresponding text feature vectors are determined from the video feature vectors corresponding to the sample text in all the target training samples. Then, based on the first dimension feature information of the text feature vectors corresponding to all the video feature vectors to be processed, a second dimension transformation operation is performed on all the video feature vectors to be processed to obtain all the transformed video feature vectors to be processed. All the transformed video feature vectors to be processed match the corresponding text feature vectors.

[0023] Based on all the transformed video feature vectors to be processed, update the video feature vectors corresponding to the sample text in all the target training samples, and trigger the execution of the operation that, for each of the input target training samples, through the feature fusion layer of the text model to be trained, performs a fusion operation between the text feature vector corresponding to the target training sample and the pre-generated video feature vector to obtain the fused feature vector corresponding to the target training sample.

[0024] As an optional implementation, in the first aspect of the present invention, for each input target training sample, predicting and reconstructing the masked vector content in the fused feature vector corresponding to the target training sample through the prediction and reconstruction layer of the text model to be trained, to obtain the prediction and reconstruction vector content corresponding to the target training sample, includes:

[0025] For each target training sample, a vector order transformation operation is performed on the masked vector content in the fused feature vector to update the masked vector content in the fused feature vector.

[0026] The masked vector content in the fused feature vector corresponding to all the updated target training samples is input into the prediction reconstruction layer of the text model to be trained, so that the prediction reconstruction layer of the text model to be trained performs the following operations: For each target training sample, based on the masked vector content in the fused feature vector corresponding to the target training sample, the semantic feature information of the target training sample is extracted, and based on the semantic feature information of the target training sample, a vector order restoration operation is performed on the masked vector content in the fused feature vector corresponding to the target training sample to update the masked vector content in the fused feature vector corresponding to the target training sample again; based on the semantic feature information of the target training sample, prediction reconstruction is performed on the masked vector content in the fused feature vector corresponding to the target training sample after the second update to obtain the prediction reconstruction vector content corresponding to the target training sample.

[0027] As an optional implementation, in the first aspect of the present invention, determining whether the trained text model has converged includes:

[0028] Obtain the distance regression loss parameter between the predicted reconstruction vector content of each target training sample calculated by the prediction reconstruction layer and the corresponding undetermined text feature vector, and determine the target reconstruction loss value corresponding to the target training sample set based on the distance regression loss parameter corresponding to all target training samples;

[0029] Determine whether the target reconstruction loss value is less than or equal to a preset reconstruction loss threshold;

[0030] When the judgment result is yes, it is determined that the trained text model has converged;

[0031] If the judgment result is negative, it is determined that the trained text model has not converged.

[0032] A second aspect of the present invention discloses a training device for a video-assisted text reconstruction model, the device comprising:

[0033] A determination module is used to determine a target training sample set; the target training sample set includes a plurality of target training samples, and each target training sample includes at least sample text;

[0034] The training module is used to perform model training operations on the text model to be trained based on all the target training samples to obtain the trained text model; wherein, when performing the model training operation on the text model to be trained, for any target training sample, the input content of the feature fusion layer of the text model to be trained includes the text feature vector corresponding to the sample text in the target training sample and the pre-generated video feature vector corresponding to the sample text.

[0035] The judgment module is used to determine whether the trained text model has converged.

[0036] The determining module is further configured to determine the trained text model as a text reconstruction model when the judgment result of the judgment module is yes; the text reconstruction model is used to supplement the text content of the target text material to be generated, so as to generate text that matches the target text material.

[0037] As an optional implementation, in a second aspect of the present invention, the training module includes:

[0038] The input submodule is used to input all the target training samples into the text model to be trained;

[0039] The vector transformation submodule is used to perform text vector transformation operations on the sample text included in each target training sample through the embedding layer of the text model to be trained, so as to obtain the text feature vector corresponding to each target training sample; each text feature vector corresponding to the target training sample contains the corresponding masked vector content;

[0040] The vector fusion submodule is used to perform a fusion operation on the text feature vector corresponding to the target training sample and the pre-generated video feature vector through the feature fusion layer of the text model to be trained for each input target training sample, so as to obtain the fused feature vector corresponding to the target training sample.

[0041] The prediction and reconstruction submodule is used to predict and reconstruct the masked vector content in the fused feature vector corresponding to each target training sample through the prediction and reconstruction layer of the text model to be trained, so as to obtain the prediction and reconstruction vector content corresponding to the target training sample.

[0042] As an optional implementation, in the second aspect of the present invention, the vector transformation submodule performs a text vector transformation operation on the sample text included in each target training sample through the embedding layer of the text model to be trained, and obtains the text feature vector corresponding to each target training sample in the following specific manner:

[0043] For each of the input target training samples, the embedding layer of the text model to be trained performs a character splitting operation on the sample text in the target training sample to obtain all the target characters of the sample text, and performs a character vector transformation operation on all the target characters of the sample text to obtain all the character feature vectors corresponding to the target training sample.

[0044] For each of the input target training samples, a concatenation operation is performed on all the character feature vectors corresponding to the target training sample to obtain all the sentence feature vectors corresponding to the target training sample, and the undetermined text feature vector corresponding to the target training sample is determined based on all the sentence feature vectors corresponding to the target training sample.

[0045] For each input target training sample, according to the preset mask parameters, the vector content in the undetermined text feature vector corresponding to the target training sample that matches the mask parameters is masked to obtain the text feature vector corresponding to the target training sample.

[0046] As an optional implementation, in the second aspect of the present invention, the vector fusion submodule, for each input target training sample, performs a fusion operation between the text feature vector corresponding to the target training sample and the pre-generated video feature vector through the feature fusion layer of the text model to be trained, specifically to obtain the fused feature vector corresponding to the target training sample, as follows:

[0047] For each input target training sample, the text feature vector corresponding to the target training sample is concatenated with the pre-generated video feature vector through the feature fusion layer of the text model to be trained, to obtain the concatenated feature vector corresponding to the target training sample. Then, a first-dimensional transformation operation is performed on the concatenated feature vector corresponding to the target training sample to obtain the transformed feature vector corresponding to the target training sample. Based on the pre-determined video feature parameters, a vector averaging operation is performed on the transformed feature vector corresponding to the target training sample to obtain the averaged feature vector corresponding to the target training sample, which is used as the fused feature vector corresponding to the target training sample.

[0048] As an optional implementation, in a second aspect of the invention, the training module further includes:

[0049] The acquisition submodule is used to acquire the video feature vector corresponding to the sample text in each target training sample pre-generated by the embedding layer before the vector fusion submodule performs a fusion operation on the text feature vector corresponding to the target training sample and the pre-generated video feature vector through the feature fusion layer of the text model to be trained for each target training sample.

[0050] The judgment submodule is used to determine whether the text feature vectors corresponding to all target training samples match the corresponding video feature vectors based on the first dimension feature information of the text feature vectors corresponding to the sample texts in all target training samples and the second dimension feature information of the corresponding video feature vectors. The first dimension feature information of the text feature vectors corresponding to the sample texts in each target training sample and the second dimension feature information of the corresponding video feature vectors each include at least one of the following: dimension type, number of dimensions, and dimension order.

[0051] The determination submodule is used to determine, when the judgment submodule determines the result of the judgment, all video feature vectors to be processed that do not match the corresponding text feature vectors from the video feature vectors corresponding to the sample text in all the target training samples;

[0052] The transformation submodule is used to perform a second-dimensional transformation operation on all the video feature vectors to be processed based on the first-dimensional feature information of the text feature vectors corresponding to all the video feature vectors to be processed, so as to obtain all the transformed video feature vectors to be processed; all the transformed video feature vectors to be processed are matched with the corresponding text feature vectors.

[0053] The update submodule is used to update the video feature vectors corresponding to the sample text in all the target training samples according to all the transformed video feature vectors to be processed, and to trigger the vector fusion submodule to perform the operation of fusing the text feature vector corresponding to the target training sample with the pre-generated video feature vector through the feature fusion layer of the text model to be trained for each input target training sample to obtain the fused feature vector corresponding to the target training sample.

[0054] As an optional implementation, in the second aspect of the present invention, the prediction and reconstruction submodule, for each input target training sample, performs prediction and reconstruction of the masked vector content in the fused feature vector corresponding to the target training sample through the prediction and reconstruction layer of the text model to be trained, specifically obtaining the prediction and reconstruction vector content corresponding to the target training sample in the following manner:

[0055] For each target training sample, a vector order transformation operation is performed on the masked vector content in the fused feature vector to update the masked vector content in the fused feature vector.

[0056] The masked vector content in the fused feature vector corresponding to all the updated target training samples is input into the prediction reconstruction layer of the text model to be trained, so that the prediction reconstruction layer of the text model to be trained performs the following operations: For each target training sample, based on the masked vector content in the fused feature vector corresponding to the target training sample, the semantic feature information of the target training sample is extracted, and based on the semantic feature information of the target training sample, a vector order restoration operation is performed on the masked vector content in the fused feature vector corresponding to the target training sample to update the masked vector content in the fused feature vector corresponding to the target training sample again; based on the semantic feature information of the target training sample, prediction reconstruction is performed on the masked vector content in the fused feature vector corresponding to the target training sample after the second update to obtain the prediction reconstruction vector content corresponding to the target training sample.

[0057] As an optional implementation, in the second aspect of the present invention, the method by which the judging module judges whether the trained text model has converged is specifically as follows:

[0058] Obtain the distance regression loss parameter between the predicted reconstruction vector content of each target training sample calculated by the prediction reconstruction layer and the corresponding undetermined text feature vector, and determine the target reconstruction loss value corresponding to the target training sample set based on the distance regression loss parameter corresponding to all target training samples;

[0059] Determine whether the target reconstruction loss value is less than or equal to a preset reconstruction loss threshold;

[0060] When the judgment result is yes, it is determined that the trained text model has converged;

[0061] If the judgment result is negative, it is determined that the trained text model has not converged.

[0062] A third aspect of the present invention discloses another video-assisted text reconstruction model training device, the device comprising:

[0063] Memory containing executable program code;

[0064] A processor coupled to the memory;

[0065] The processor calls the executable program code stored in the memory to execute the video-assisted text reconstruction model training method disclosed in the first aspect of the present invention.

[0066] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute the video-assisted text reconstruction model training method disclosed in the first aspect of the present invention.

[0067] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0068] In this embodiment of the invention, a target training sample set is determined; based on all target training samples, a model training operation is performed on the text model to be trained to obtain a trained text model, and it is determined whether the trained text model has converged; wherein, when performing the model training operation on the text model to be trained, for any target training sample, the input content of the feature fusion layer of the text model to be trained includes the text feature vector corresponding to the sample text in the target training sample and the pre-generated video feature vector corresponding to the sample text; when the determination result is yes, the trained text model is determined as a text reconstruction model. It can be seen that implementing this invention can train a text reconstruction model with the assistance of video feature vectors, which not only helps users quickly create video text using the text reconstruction model without needing to manually edit and repeatedly correct the video text using a fixed video text production template, greatly shortening the video text production cycle, but also helps improve the matching degree between the generated video text and the video, thereby meeting the user's needs for video text production. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 This is a flowchart illustrating a video-assisted text reconstruction model training method disclosed in an embodiment of the present invention.

[0071] Figure 2 This is a flowchart illustrating another video-assisted text reconstruction model training method disclosed in an embodiment of the present invention.

[0072] Figure 3 This is a schematic diagram of the structure of a video-assisted text reconstruction model training device disclosed in an embodiment of the present invention;

[0073] Figure 4 This is a schematic diagram of another video-assisted text reconstruction model training device disclosed in an embodiment of the present invention;

[0074] Figure 5 This is a schematic diagram of the structure of another video-assisted text reconstruction model training device disclosed in an embodiment of the present invention. Detailed Implementation

[0075] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0076] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0077] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0078] This invention discloses a video-assisted text reconstruction model training method and apparatus, which not only helps users quickly create video text using the text reconstruction model, but also improves the matching degree between the generated video text and the video, thereby meeting users' needs for video text creation. Detailed descriptions follow.

[0079] Example 1

[0080] Please see Figure 1 , Figure 1 This is a flowchart illustrating a video-assisted text reconstruction model training method disclosed in an embodiment of the present invention. Wherein, Figure 1The text reconstruction model trained by the video-assisted text reconstruction model training method described herein can supplement the target text material with additional text content to generate text that matches the target text material. It can also utilize the output model results to perform subsequent classification work on the text to be classified. This embodiment of the invention is not limited in its scope. Optionally, this method can be implemented by a text reconstruction model training system, which can be integrated into a text reconstruction model training device, or it can be a local server or cloud server used to process the text reconstruction model training process, etc. This embodiment of the invention is not limited in its scope. Figure 1 As shown, the training method for this video-assisted text reconstruction model may include the following operations:

[0081] 101. Determine the target training sample set.

[0082] In this embodiment of the invention, the target training sample set includes a plurality of target training samples, and each target training sample includes at least sample text. Optionally, each target training sample may include only sample text, or it may include sample text and a set of video frames corresponding to the sample text, wherein the sample text includes a plurality of sentences.

[0083] 102. Based on all target training samples, perform model training operations on the text model to be trained to obtain the trained text model, and determine whether the trained text model has converged.

[0084] In this embodiment of the invention, when performing model training operations on the text model to be trained, for any target training sample, the input content of the feature fusion layer of the text model to be trained includes the text feature vector corresponding to the sample text in the target training sample and the pre-generated video feature vector corresponding to the sample text. That is, the text feature vector corresponding to the sample text in each target training sample and the pre-generated video feature vector corresponding to the sample text need to be input in pairs into the feature fusion layer of the text model to be trained.

[0085] Specifically, for any target training sample, the pre-generated video feature vector corresponding to the sample text is determined as follows: when the target training sample includes a set of video frames corresponding to the sample text, it is obtained by performing a vector transformation operation on the set of video frames included in the sample text by the embedding layer of the text model to be trained; when the target training sample does not include a set of video frames corresponding to the sample text, the video feature vector corresponding to the sample text is initialized and generated by the embedding layer of the text model to be trained. That is, when the sample text has missing video feature vectors, it can be randomly initialized by the embedding layer to ensure that the feature vectors are input in pairs.

[0086] 103. When the judgment result is yes, the trained text model is determined as the text reconstruction model.

[0087] In this embodiment of the invention, the text reconstruction model is used to supplement the target text material of the text to be generated, so as to generate text that matches the target text material. For example, when the target text material of the text to be generated includes sentences A and B, all sentences included in the target text material are directly input into the trained text reconstruction model. By supplementing the target text material with text content through the text reconstruction model, a matching sentence C can be added to the target text material to generate text that matches the target text material (which includes sentences A, B, and C).

[0088] Furthermore, as an optional implementation, the method may further include:

[0089] When it is determined that the trained text model does not converge, the trained text model is updated to the text model to be trained, and the model parameters of the updated text model to be trained are changed to trigger the execution of step 102, which involves performing model training operations on the text model to be trained based on all target training samples to obtain the trained text model and determining whether the trained text model has converged.

[0090] As can be seen, implementing the embodiments of the present invention can train a text reconstruction model with the assistance of video feature vectors. This not only helps users quickly create video text using the text reconstruction model, eliminating the need for manual editing and repeated correction of video text using fixed video text production templates, thus greatly shortening the video text production cycle, but also helps improve the matching degree between the generated video text and the video, thereby meeting users' needs for video text production.

[0091] Example 2

[0092] Please see Figure 2 , Figure 2 This is a flowchart illustrating a video-assisted text reconstruction model training method disclosed in an embodiment of the present invention. Wherein, Figure 2 The text reconstruction model trained by the video-assisted text reconstruction model training method described herein can supplement the target text material with additional text content to generate text that matches the target text material. It can also utilize the output model results to perform subsequent classification work on the text to be classified. This embodiment of the invention is not limited in its scope. Optionally, this method can be implemented by a text reconstruction model training system, which can be integrated into a text reconstruction model training device, or it can be a local server or cloud server used to process the text reconstruction model training process, etc. This embodiment of the invention is not limited in its scope. Figure 2As shown, the training method for this video-assisted text reconstruction model may include the following operations:

[0093] 201. Determine the target training sample set.

[0094] 202. Input all target training samples into the text model to be trained, and perform text vector transformation operation on the sample text included in each target training sample through the embedding layer of the text model to be trained, so as to obtain the text feature vector corresponding to each target training sample.

[0095] In this embodiment of the invention, the text vector conversion operation specifically includes word splitting, word vector conversion, vector concatenation, and masking of the sample text. That is, each target training sample contains a corresponding masked vector in its text feature vector. The word splitting operation can be implemented using a wordpiece encoding algorithm.

[0096] 203. For each input target training sample, the text feature vector corresponding to the target training sample is fused with the pre-generated video feature vector through the feature fusion layer of the text model to be trained, so as to obtain the fused feature vector corresponding to the target training sample.

[0097] In this embodiment of the invention, the fusion operation specifically includes vector concatenation, a first-dimensional transformation, and vector averaging. The first-dimensional transformation can be implemented using the vector dimension transformation functions of the transformation layer, fully connected layer, and activation layer in the feature fusion layer. Optionally, the order and number of layers in the feature fusion layer can be arbitrary and are not limited in this embodiment.

[0098] 204. For each input target training sample, the prediction and reconstruction layer of the text model to be trained is used to predict and reconstruct the masked vector content in the fused feature vector corresponding to the target training sample, so as to obtain the prediction and reconstruction vector content corresponding to the target training sample.

[0099] In this embodiment of the invention, specifically, the text reconstruction layer of the text model to be trained may include several encoders and several decoders. The prediction reconstruction operation can be understood as follows: for each input target training sample, the semantic feature information of the target training sample is extracted by using all encoders in the prediction reconstruction layer and the masked vector content in the fused feature vector corresponding to the target training sample. Then, the masked vector content in the fused feature vector corresponding to the target training sample is predicted and reconstructed by using all decoders in the prediction reconstruction layer and the extracted semantic feature information of the target training sample, so as to obtain the prediction reconstruction vector content corresponding to the target training sample.

[0100] 205. Determine whether the text model has converged after training.

[0101] 206. When the judgment result is yes, the trained text model is determined as the text reconstruction model.

[0102] In this embodiment of the invention, for other descriptions of steps 201, 205 and 206, please refer to the detailed description of steps 101-103 in Embodiment 1. This embodiment of the invention will not repeat them.

[0103] As can be seen, implementing the embodiments of the present invention can perform vector transformation, vector fusion, and vector reconstruction prediction on each target training sample through the embedding layer, feature fusion layer, and prediction reconstruction layer of the text model to be trained, so that the text model to be trained can construct the relationship between the video's screen change information and the word order information of the visible text. This is beneficial to improving the reliability, accuracy, and effectiveness of the trained text reconstruction model, and thus facilitates the subsequent accurate generation of text that matches the relevant text materials and the text that matches the corresponding video, thereby meeting the user's needs for video text production.

[0104] In an optional embodiment, step 202 above, which involves performing a text vector transformation operation on the sample text included in each target training sample through the embedding layer of the text model to be trained, to obtain the text feature vector corresponding to each target training sample, includes:

[0105] For each input target training sample, the embedding layer of the text model to be trained performs a character splitting operation on the sample text in the target training sample to obtain all target characters of the sample text, and performs a character vector transformation operation on all target characters of the sample text to obtain all character feature vectors corresponding to the target training sample.

[0106] For each input target training sample, perform a concatenation operation on all character feature vectors corresponding to the target training sample to obtain all sentence feature vectors corresponding to the target training sample, and determine the undetermined text feature vector corresponding to the target training sample based on all sentence feature vectors corresponding to the target training sample.

[0107] For each input target training sample, according to the preset mask parameters, the vector content in the undetermined text feature vector corresponding to the target training sample that matches the mask parameters is masked to obtain the text feature vector corresponding to the target training sample.

[0108] In this optional embodiment, specifically, the word splitting operation can be understood as: splitting "adidas shoes hot sale" into ["a", "d", "i", "d", "a", "s", "shoes", "child", "hot", "sell"]. Optionally, the preset mask parameters include the text position of the desired mask and / or the text proportion of the desired mask. Further, the mask parameters can be determined by: obtaining the target parameters corresponding to the mask parameters, and determining the mask parameters based on the target parameters; wherein, the target parameters include the proportion of missing video feature vectors corresponding to the sample text in the target training sample and / or the clarity of the video frame set included in the target training sample. For example, if the proportion of missing video feature vectors corresponding to the sample text in the target training sample is high, a smaller text proportion of the desired mask can be determined; while if the clarity of the video frame set included in the target training sample is high, a larger text proportion of the desired mask can be determined. Optionally, the text feature vector corresponding to the target training sample may include at least one parameter among the total number of sentences, sentence length, and feature vector dimension.

[0109] As can be seen, this optional embodiment can perform text vector transformation operations on the sample text in each target training sample in a definite and targeted manner, which is beneficial to improving the reliability and accuracy of the text vector transformation operation in the embedding layer, and thus improving the reliability and accuracy of the obtained text feature vector, thereby facilitating the accurate performance of subsequent text and video feature vector fusion operations.

[0110] In another optional embodiment, in step 203 above, for each input target training sample, the text feature vector corresponding to the target training sample is fused with the pre-generated video feature vector through the feature fusion layer of the text model to be trained, to obtain the fused feature vector corresponding to the target training sample, including:

[0111] For each input target training sample, the text feature vector corresponding to the target training sample is concatenated with the pre-generated video feature vector through the feature fusion layer of the text model to be trained, resulting in the concatenated feature vector corresponding to the target training sample. Then, the first dimension transformation operation is performed on the concatenated feature vector corresponding to the target training sample to obtain the transformed feature vector corresponding to the target training sample. Based on the pre-determined video feature parameters, the vector averaging operation is performed on the transformed feature vector corresponding to the target training sample to obtain the averaged feature vector corresponding to the target training sample, which is used as the fused feature vector corresponding to the target training sample.

[0112] In this optional embodiment, the video feature parameters may optionally include at least one of the following: the number of frames in the video frame set included in the target training sample, the degree of video frame change, and the video duration. For example, the vector averaging operation can be understood as follows: if the video frame set has 'a' frames, and the transformed feature vector corresponding to the target training sample is (a, b, c), then after performing vector averaging, the averaged feature vector corresponding to the target training sample can be obtained as (1, b, c), where the above letters can all represent any natural number.

[0113] As can be seen, this optional embodiment can fuse the text feature vector and the corresponding video feature vector for each target training sample in a definite and targeted manner. In this way, with the assistance of the video feature vector, the learning effect of the text model to be trained can be improved, so that the target text generated by it can be matched with the corresponding video, reducing the occurrence of errors in the production of target text.

[0114] In another optional embodiment, before performing a fusion operation between the text feature vector corresponding to the target training sample and the pre-generated video feature vector through the feature fusion layer of the text model to be trained for each input target training sample in step 203 above, to obtain the fused feature vector corresponding to the target training sample, the method may further include:

[0115] Obtain the video feature vector corresponding to the sample text in each target training sample pre-generated by the embedding layer;

[0116] Based on the first dimension feature information of the text feature vector corresponding to the sample text in all target training samples and the second dimension feature information of the corresponding video feature vector, determine whether the text feature vector corresponding to all target training samples matches the corresponding video feature vector.

[0117] When the judgment result is negative, determine all video feature vectors that do not match the corresponding text feature vectors from the video feature vectors corresponding to the sample text in all target training samples. Then, based on the first dimension feature information of the text feature vectors corresponding to all video feature vectors to be processed, perform the second dimension transformation operation on all video feature vectors to be processed to obtain all transformed video feature vectors to be processed.

[0118] Based on the transformed video feature vectors of all the target training samples, update the video feature vectors corresponding to the sample text in all the target training samples, and trigger the execution of the fusion operation of the text feature vector corresponding to the target training sample and the pre-generated video feature vector through the feature fusion layer of the text model to be trained for each input target training sample, so as to obtain the fused feature vector corresponding to the target training sample.

[0119] In this optional embodiment, all transformed video feature vectors to be processed are matched with their corresponding text feature vectors. Optionally, the first dimension feature information of the text feature vector corresponding to the sample text in each target training sample and the second dimension feature information of the corresponding video feature vector both include at least one of the following: dimension type, number of dimensions, and dimension order. That is, it is necessary to ensure that the text feature vector corresponding to the target training sample input to the feature fusion layer matches the dimension type, number of dimensions, and dimension order of the pre-generated video feature vector.

[0120] Furthermore, in this optional embodiment, the method may further include: when it is determined that the text feature vectors corresponding to all target training samples do not match the corresponding video feature vectors, determining all text feature vectors to be processed that do not match the corresponding video feature vectors from the text feature vectors corresponding to all target training samples, and performing a third-dimensional transformation operation on all text feature vectors to be processed based on the second-dimensional feature information of the video feature vectors corresponding to all text feature vectors to be processed, to obtain all transformed text feature vectors to be processed; updating the text feature vectors corresponding to all target training samples based on the transformed text feature vectors to be processed, and triggering the execution of the operation of performing a fusion operation on the text feature vector corresponding to the target training sample and the pre-generated video feature vector through the feature fusion layer of the text model to be trained for each input target training sample, to obtain the fused feature vector corresponding to the target training sample.

[0121] As can be seen, this optional embodiment can ensure that the text feature vector of the sample text input to the feature fusion layer matches the corresponding video feature vector by performing dimensional transformation on the mismatched related feature vectors. This is beneficial to improving the reliability, accuracy and effectiveness of the feature fusion layer fusion operation, thereby improving the reliability, accuracy and effectiveness of the fused feature vector corresponding to each target training sample.

[0122] In another optional embodiment, in step 204 above, for each input target training sample, the prediction and reconstruction layer of the text model to be trained predicts and reconstructs the masked vector content in the fused feature vector corresponding to the target training sample to obtain the prediction and reconstruction vector content corresponding to the target training sample, including:

[0123] For each target training sample, perform a vector order transformation operation on the masked vector content in the fused feature vector to update the masked vector content in the fused feature vector.

[0124] The masked vector content in the fused feature vector corresponding to all updated target training samples is input into the prediction reconstruction layer of the text model to be trained. The prediction reconstruction layer of the text model to be trained performs the following operations: For each target training sample, based on the masked vector content in the fused feature vector corresponding to the target training sample, the semantic feature information of the target training sample is extracted. Based on the semantic feature information of the target training sample, a vector order restoration operation is performed on the masked vector content in the fused feature vector corresponding to the target training sample to update the masked vector content in the fused feature vector corresponding to the target training sample again. Based on the semantic feature information of the target training sample, prediction reconstruction is performed on the masked vector content in the fused feature vector corresponding to the target training sample after the second update to obtain the prediction reconstruction vector content corresponding to the target training sample.

[0125] In this optional embodiment, the vector order transformation operation and the vector order restoration operation can be understood as performing text order shuffling and text order restoration operations on the multiple texts to be processed that match the masking parameters obtained in the above steps, respectively. Specifically, the prediction reconstruction layer may include several encoders and several decoders. All encoders extract semantic feature information from the target training samples and perform vector order restoration operations on the masked vector content in the fused feature vector corresponding to the target training samples. All decoders predict and reconstruct the masked vector content in the fused feature vector corresponding to the updated target training samples. It should be noted that through this prediction reconstruction operation, the text model to be trained can learn to supplement existing text information with text content based on existing text information, whether or not a corresponding video is available, to generate a more complete text that matches the existing text information.

[0126] As can be seen, this optional embodiment can specifically reconstruct and predict the content of the masked vector in the target training sample through the prediction and reconstruction layer of the text model to be trained, which is beneficial to improving the training reliability and accuracy of the text model to be trained, thereby improving the reliability and accuracy of the trained text reconstruction model, so as to ensure that the text reconstruction model can obtain video text that matches the corresponding video.

[0127] In another optional embodiment, step 205 above, determining whether the trained text model has converged, includes:

[0128] Obtain the distance regression loss parameter between the predicted reconstruction vector content of each target training sample calculated by the prediction reconstruction layer and the corresponding undetermined text feature vector, and determine the target reconstruction loss value corresponding to the target training sample set based on the distance regression loss parameter corresponding to all target training samples.

[0129] Determine whether the target reconstruction loss value is less than or equal to a preset reconstruction loss threshold;

[0130] When the judgment result is yes, it is determined that the trained text model has converged.

[0131] If the judgment result is negative, it is determined that the trained text model has not converged.

[0132] In this optional embodiment, the target reconstruction loss value can optionally be: Loss = smooth_L1_1 + ... + smooth_L1_n, where smooth_L1_n is the distance regression loss parameter corresponding to the target training sample. Specifically, the calculation of the distance regression loss parameter can be understood as comparing the predicted reconstruction vector content predicted by the prediction reconstruction layer with the vector content before masking (i.e., the true value) to observe the learning effect of the text model after training. If the target reconstruction loss value determined by the distance regression loss parameter corresponding to all target training samples is less than or equal to the preset reconstruction loss threshold, it indicates that the learning effect of the text model after training has reached the preset state; otherwise, a new round of model training is required.

[0133] As can be seen, this optional embodiment can intelligently repeat the training of the post-trained text model until the post-trained text model converges. This helps to improve the training reliability, training accuracy and training effectiveness of the entire model training process, thereby helping to train a reliable text reconstruction model so that the text generated by the text reconstruction model can match the corresponding video and meet the user's video production needs.

[0134] Example 3

[0135] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a video-assisted text reconstruction model training device disclosed in an embodiment of the present invention. Figure 3 As shown, the training device for the video-assisted text reconstruction model may include:

[0136] Module 301 is used to determine the target training sample set;

[0137] Training module 302 is used to perform model training operations on the text model to be trained based on all target training samples, and obtain the trained text model;

[0138] Module 303 is used to determine whether the trained text model has converged.

[0139] The determination module 301 is also used to determine the trained text model as the text reconstruction model when the judgment result of the judgment module 303 is yes.

[0140] In this embodiment of the invention, the target training sample set includes several target training samples, and each target training sample includes at least sample text; wherein, when performing model training operations on the text model to be trained, for any target training sample, the input content of the feature fusion layer of the text model to be trained includes the text feature vector corresponding to the sample text in the target training sample and the pre-generated video feature vector corresponding to the sample text; the text reconstruction model is used to supplement the text content of the target text material to be generated, so as to generate text that matches the target text material.

[0141] It is evident that implementation Figure 3 The described video-assisted text reconstruction model training device can train a text reconstruction model with the assistance of video feature vectors. This not only helps users quickly create video text using the text reconstruction model, eliminating the need for manual editing and repeated correction of video text using fixed video text production templates, thus greatly shortening the video text production cycle, but also helps improve the matching degree between the generated video text and the video, thereby meeting users' needs for video text production.

[0142] In an optional embodiment, the training module 302 includes:

[0143] The input submodule 3021 is used to input all target training samples into the text model to be trained;

[0144] The vector transformation submodule 3022 is used to perform text vector transformation operations on the sample text included in each target training sample through the embedding layer of the text model to be trained, so as to obtain the text feature vector corresponding to each target training sample.

[0145] The vector fusion submodule 3023 is used to perform a fusion operation between the text feature vector corresponding to the target training sample and the pre-generated video feature vector through the feature fusion layer of the text model to be trained for each input target training sample, so as to obtain the fused feature vector corresponding to the target training sample.

[0146] The prediction and reconstruction submodule 3024 is used to predict and reconstruct the masked vector content in the fused feature vector corresponding to each target training sample by passing the prediction and reconstruction layer of the text model to be trained, so as to obtain the prediction and reconstruction vector content corresponding to the target training sample.

[0147] In this optional embodiment, the text feature vector corresponding to each target training sample contains the corresponding masked vector content.

[0148] It is evident that implementation Figure 4 The described video-assisted text reconstruction model training device can perform vector transformation, vector fusion, and vector reconstruction prediction on each target training sample through the embedding layer, feature fusion layer, and prediction reconstruction layer of the text model to be trained. This enables the text model to construct the relationship between the video's visual changes and the word order information of the visible text. This improves the reliability, accuracy, and effectiveness of the trained text reconstruction model, and further facilitates the subsequent accurate generation of text that matches the relevant text materials and the corresponding video, thereby meeting the user's needs for video text production.

[0149] In another optional embodiment, the vector transformation submodule 3022 performs text vector transformation operations on the sample text included in each target training sample through the embedding layer of the text model to be trained, and obtains the text feature vector corresponding to each target training sample in the following specific way:

[0150] For each input target training sample, the embedding layer of the text model to be trained performs a character splitting operation on the sample text in the target training sample to obtain all target characters of the sample text, and performs a character vector transformation operation on all target characters of the sample text to obtain all character feature vectors corresponding to the target training sample.

[0151] For each input target training sample, perform a concatenation operation on all character feature vectors corresponding to the target training sample to obtain all sentence feature vectors corresponding to the target training sample, and determine the undetermined text feature vector corresponding to the target training sample based on all sentence feature vectors corresponding to the target training sample.

[0152] For each input target training sample, according to the preset mask parameters, the vector content in the undetermined text feature vector corresponding to the target training sample that matches the mask parameters is masked to obtain the text feature vector corresponding to the target training sample.

[0153] It is evident that implementation Figure 4 The described video-assisted text reconstruction model training device can perform text vector transformation operations on the sample text in each target training sample in a definite and targeted manner, which is conducive to improving the reliability and accuracy of the text vector transformation operation in the embedding layer, and thus improving the reliability and accuracy of the obtained text feature vectors, thereby facilitating the accurate fusion of subsequent text and video feature vectors.

[0154] In another optional embodiment, for each input target training sample, the vector fusion submodule 3023 performs a fusion operation between the text feature vector corresponding to the target training sample and the pre-generated video feature vector through the feature fusion layer of the text model to be trained, specifically to obtain the fused feature vector corresponding to the target training sample:

[0155] For each input target training sample, the text feature vector corresponding to the target training sample is concatenated with the pre-generated video feature vector through the feature fusion layer of the text model to be trained, resulting in the concatenated feature vector corresponding to the target training sample. Then, the first dimension transformation operation is performed on the concatenated feature vector corresponding to the target training sample to obtain the transformed feature vector corresponding to the target training sample. Based on the pre-determined video feature parameters, the vector averaging operation is performed on the transformed feature vector corresponding to the target training sample to obtain the averaged feature vector corresponding to the target training sample, which is used as the fused feature vector corresponding to the target training sample.

[0156] It is evident that implementation Figure 4 The described video-assisted text reconstruction model training device can fuse the text feature vector and the corresponding video feature vector for each target training sample in a definite and targeted manner. In this way, with the assistance of the video feature vector, the learning effect of the text model to be trained can be improved, so that the target text generated by it can match the corresponding video and reduce the occurrence of errors in the production of target text.

[0157] In yet another optional embodiment, the training module 302 further includes:

[0158] The acquisition submodule 3025 is used to acquire the video feature vector corresponding to the sample text in each target training sample pre-generated by the embedding layer before the vector fusion submodule 3023 performs a fusion operation on the text feature vector corresponding to the target training sample and the pre-generated video feature vector through the feature fusion layer of the text model to be trained for each input target training sample.

[0159] The judgment submodule 3026 is used to determine whether the text feature vectors corresponding to all target training samples match the corresponding video feature vectors based on the first dimension feature information of the text feature vectors corresponding to the sample texts in all target training samples and the second dimension feature information of the corresponding video feature vectors.

[0160] The determination submodule 3027 is used to determine all video feature vectors that do not match the corresponding text feature vectors from the video feature vectors corresponding to the sample text in all target training samples when the judgment result of the judgment submodule 3026 is negative.

[0161] The transformation submodule 3028 is used to perform a second-dimensional transformation operation on all video feature vectors to be processed based on the first-dimensional feature information of the text feature vectors corresponding to all video feature vectors to be processed, so as to obtain all transformed video feature vectors to be processed.

[0162] The update submodule 3029 is used to update the video feature vectors corresponding to the sample text in all target training samples according to the transformed video feature vectors to be processed, and to trigger the vector fusion submodule 3023 to perform the operation of fusing the text feature vector corresponding to the target training sample with the pre-generated video feature vector through the feature fusion layer of the text model to be trained for each input target training sample, so as to obtain the fused feature vector corresponding to the target training sample.

[0163] In this optional embodiment, the first dimension feature information of the text feature vector corresponding to the sample text in each target training sample and the second dimension feature information of the corresponding video feature vector both include at least one of the corresponding dimension type, dimension number, and dimension order; all transformed video feature vectors to be processed are matched with the corresponding text feature vectors.

[0164] It is evident that implementation Figure 4 The described video-assisted text reconstruction model training device can ensure that the text feature vectors of the sample text input to the feature fusion layer match the corresponding video feature vectors by performing dimensional transformation on the mismatched related feature vectors. This helps to improve the reliability, accuracy and effectiveness of the feature fusion layer fusion operation, thereby improving the reliability, accuracy and effectiveness of the fused feature vectors corresponding to each target training sample.

[0165] In another optional embodiment, for each input target training sample, the prediction reconstruction submodule 3024 predicts and reconstructs the masked vector content in the fused feature vector corresponding to the target training sample through the prediction reconstruction layer of the text model to be trained, and obtains the prediction reconstruction vector content corresponding to the target training sample in the following specific way:

[0166] For each target training sample, perform a vector order transformation operation on the masked vector content in the fused feature vector to update the masked vector content in the fused feature vector.

[0167] The masked vector content in the fused feature vector corresponding to all updated target training samples is input into the prediction reconstruction layer of the text model to be trained. The prediction reconstruction layer of the text model to be trained performs the following operations: For each target training sample, based on the masked vector content in the fused feature vector corresponding to the target training sample, the semantic feature information of the target training sample is extracted. Based on the semantic feature information of the target training sample, a vector order restoration operation is performed on the masked vector content in the fused feature vector corresponding to the target training sample to update the masked vector content in the fused feature vector corresponding to the target training sample again. Based on the semantic feature information of the target training sample, prediction reconstruction is performed on the masked vector content in the fused feature vector corresponding to the target training sample after the second update to obtain the prediction reconstruction vector content corresponding to the target training sample.

[0168] It is evident that implementation Figure 4 The described video-assisted text reconstruction model training device can selectively reconstruct and predict the content of the masked vector in the target training sample through the prediction reconstruction layer of the text model to be trained. This is beneficial to improving the training reliability and accuracy of the text model to be trained, thereby improving the reliability and accuracy of the trained text reconstruction model, so as to ensure that the text reconstruction model can obtain video text that matches the corresponding video.

[0169] In another optional embodiment, the determination module 303 determines whether the trained text model has converged in the following specific way:

[0170] Obtain the distance regression loss parameter between the predicted reconstruction vector content of each target training sample calculated by the prediction reconstruction layer and the corresponding undetermined text feature vector, and determine the target reconstruction loss value corresponding to the target training sample set based on the distance regression loss parameter corresponding to all target training samples.

[0171] Determine whether the target reconstruction loss value is less than or equal to a preset reconstruction loss threshold;

[0172] When the judgment result is yes, it is determined that the trained text model has converged.

[0173] If the judgment result is negative, it is determined that the trained text model has not converged.

[0174] It is evident that implementation Figure 4The described video-assisted text reconstruction model training device can intelligently repeat the training of the text model until it converges. This improves the reliability, accuracy, and effectiveness of the entire model training process, thereby enabling the training of a reliable text reconstruction model that can match the text generated by the model with the corresponding video, thus meeting the user's video production needs.

[0175] Example 4

[0176] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of another video-assisted text reconstruction model training device disclosed in an embodiment of the present invention. For example... Figure 5 As shown, the training device for the video-assisted text reconstruction model may include:

[0177] Memory 401 storing executable program code;

[0178] Processor 402 coupled to memory 401;

[0179] The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the video-assisted text reconstruction model training method described in Embodiment 1 or Embodiment 2 of the present invention.

[0180] Example 5

[0181] This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute the steps in the video-assisted text reconstruction model training method described in Embodiment 1 or Embodiment 2 of this invention.

[0182] Example 6

[0183] This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the video-assisted text reconstruction model training method described in Embodiment 1 or Embodiment 2.

[0184] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0185] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0186] Finally, it should be noted that the video-assisted text reconstruction model training method and apparatus disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for training a video-assisted text reconstruction model, characterized in that, The method includes: Determine a target training sample set; the target training sample set includes a plurality of target training samples, and each target training sample includes at least sample text; Based on all the target training samples, a model training operation is performed on the text model to be trained to obtain a trained text model, and it is determined whether the trained text model has converged; wherein, when performing the model training operation on the text model to be trained, for any target training sample, the input content of the feature fusion layer of the text model to be trained includes the text feature vector corresponding to the sample text in the target training sample and the pre-generated video feature vector corresponding to the sample text. When the judgment result is yes, the trained text model is determined as the text reconstruction model; the text reconstruction model is used to supplement the text content of the target text material to be generated, so as to generate text that matches the target text material; The step of performing model training operations on the text model to be trained based on all the target training samples to obtain the trained text model includes: All the target training samples are input into the text model to be trained, and a text vector transformation operation is performed on the sample text included in each target training sample through the embedding layer of the text model to be trained to obtain the text feature vector corresponding to each target training sample; each text feature vector corresponding to each target training sample contains the corresponding masked vector content; For each of the input target training samples, the feature fusion layer of the text model to be trained performs a fusion operation on the text feature vector corresponding to the target training sample and the pre-generated video feature vector to obtain the fused feature vector corresponding to the target training sample. For each of the input target training samples, the prediction and reconstruction layer of the text model to be trained is used to predict and reconstruct the masked vector content in the fused feature vector corresponding to the target training sample, so as to obtain the prediction and reconstruction vector content corresponding to the target training sample.

2. The method for training a video-assisted text reconstruction model according to claim 1, characterized in that, The step of performing a text vector transformation operation on the sample text included in each target training sample through the embedding layer of the text model to be trained, to obtain the text feature vector corresponding to each target training sample, includes: For each of the input target training samples, the embedding layer of the text model to be trained performs a character splitting operation on the sample text in the target training sample to obtain all the target characters of the sample text, and performs a character vector transformation operation on all the target characters of the sample text to obtain all the character feature vectors corresponding to the target training sample. For each of the input target training samples, a concatenation operation is performed on all the character feature vectors corresponding to the target training sample to obtain all the sentence feature vectors corresponding to the target training sample, and the undetermined text feature vector corresponding to the target training sample is determined based on all the sentence feature vectors corresponding to the target training sample. For each input target training sample, according to the preset mask parameters, the vector content in the undetermined text feature vector corresponding to the target training sample that matches the mask parameters is masked to obtain the text feature vector corresponding to the target training sample.

3. The method for training a video-assisted text reconstruction model according to claim 2, characterized in that, For each of the input target training samples, the feature fusion layer of the text model to be trained performs a fusion operation between the text feature vector corresponding to the target training sample and the pre-generated video feature vector to obtain the fused feature vector corresponding to the target training sample, including: For each input target training sample, the text feature vector corresponding to the target training sample is concatenated with the pre-generated video feature vector through the feature fusion layer of the text model to be trained, to obtain the concatenated feature vector corresponding to the target training sample. Then, a first-dimensional transformation operation is performed on the concatenated feature vector corresponding to the target training sample to obtain the transformed feature vector corresponding to the target training sample. Based on the pre-determined video feature parameters, a vector averaging operation is performed on the transformed feature vector corresponding to the target training sample to obtain the averaged feature vector corresponding to the target training sample, which is used as the fused feature vector corresponding to the target training sample.

4. The method for training a video-assisted text reconstruction model according to claim 2 or 3, characterized in that, Before performing a fusion operation between the text feature vector corresponding to the target training sample and the pre-generated video feature vector through the feature fusion layer of the text model to be trained for each input target training sample, to obtain the fused feature vector corresponding to the target training sample, the method further includes: Obtain the video feature vector corresponding to the sample text in each of the target training samples pre-generated by the embedding layer; Based on the first dimension feature information of the text feature vector corresponding to the sample text in all the target training samples and the second dimension feature information of the corresponding video feature vector, it is determined whether the text feature vector corresponding to all the target training samples matches the corresponding video feature vector; the first dimension feature information of the text feature vector corresponding to the sample text in each target training sample and the second dimension feature information of the corresponding video feature vector each include at least one of the following: dimension type, number of dimensions, and dimension order. When the judgment result is negative, all video feature vectors to be processed that do not match the corresponding text feature vectors are determined from the video feature vectors corresponding to the sample text in all the target training samples. Then, based on the first dimension feature information of the text feature vectors corresponding to all the video feature vectors to be processed, a second dimension transformation operation is performed on all the video feature vectors to be processed to obtain all the transformed video feature vectors to be processed. All the transformed video feature vectors to be processed match the corresponding text feature vectors. Based on all the transformed video feature vectors to be processed, update the video feature vectors corresponding to the sample text in all the target training samples, and trigger the execution of the operation that, for each of the input target training samples, through the feature fusion layer of the text model to be trained, performs a fusion operation between the text feature vector corresponding to the target training sample and the pre-generated video feature vector to obtain the fused feature vector corresponding to the target training sample.

5. The method for training a video-assisted text reconstruction model according to claim 4, characterized in that, For each of the input target training samples, the prediction and reconstruction layer of the text model to be trained predicts and reconstructs the masked vector content in the fused feature vector corresponding to the target training sample, to obtain the prediction and reconstruction vector content corresponding to the target training sample, including: For each target training sample, a vector order transformation operation is performed on the masked vector content in the fused feature vector to update the masked vector content in the fused feature vector. The masked vector content in the fused feature vector corresponding to all the updated target training samples is input into the prediction reconstruction layer of the text model to be trained, so that the prediction reconstruction layer of the text model to be trained performs the following operations: For each target training sample, based on the masked vector content in the fused feature vector corresponding to the target training sample, the semantic feature information of the target training sample is extracted, and based on the semantic feature information of the target training sample, a vector order restoration operation is performed on the masked vector content in the fused feature vector corresponding to the target training sample to update the masked vector content in the fused feature vector corresponding to the target training sample again; based on the semantic feature information of the target training sample, prediction reconstruction is performed on the masked vector content in the fused feature vector corresponding to the target training sample after the second update to obtain the prediction reconstruction vector content corresponding to the target training sample.

6. The method for training a video-assisted text reconstruction model according to claim 5, characterized in that, The determination of whether the trained text model has converged includes: Obtain the distance regression loss parameter between the predicted reconstruction vector content of each target training sample calculated by the prediction reconstruction layer and the corresponding undetermined text feature vector, and determine the target reconstruction loss value corresponding to the target training sample set based on the distance regression loss parameter corresponding to all target training samples; Determine whether the target reconstruction loss value is less than or equal to a preset reconstruction loss threshold; When the judgment result is yes, it is determined that the trained text model has converged; If the judgment result is negative, it is determined that the trained text model has not converged.

7. A training device for a video-assisted text reconstruction model, characterized in that, The apparatus is used to perform the video-assisted text reconstruction model training method as described in any one of claims 1-6, and the apparatus comprises: A determination module is used to determine a target training sample set; the target training sample set includes a plurality of target training samples, and each target training sample includes at least sample text; The training module is used to perform model training operations on the text model to be trained based on all the target training samples to obtain the trained text model; wherein, when performing the model training operation on the text model to be trained, for any target training sample, the input content of the feature fusion layer of the text model to be trained includes the text feature vector corresponding to the sample text in the target training sample and the pre-generated video feature vector corresponding to the sample text. The judgment module is used to determine whether the trained text model has converged. The determining module is further configured to determine the trained text model as a text reconstruction model when the judgment result of the judgment module is yes; the text reconstruction model is used to supplement the text content of the target text material to be generated, so as to generate text that matches the target text material.

8. A training device for a video-assisted text reconstruction model, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the video-assisted text reconstruction model training method as described in any one of claims 1-6.

9. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute the video-assisted text reconstruction model training method as described in any one of claims 1-6.

Images