Image and text-based model training methods and apparatus

Abstract

This invention discloses a model training method and apparatus based on images and text. The method includes: performing frame extraction on a training video to obtain multiple training video frames; determining a target representation frame image from the multiple training video frames; copying the target representation frame image to obtain multiple copied representation frame images; inputting the multiple copied representation frame images and descriptive text into a video reconstruction prediction model for training; calculating the video loss function value and text loss function value output by the video reconstruction prediction model during training; and optimizing the model parameters of the video reconstruction prediction model based on the video loss function value and text loss function value until convergence, thereby obtaining a trained video reconstruction prediction model. It is evident that this invention enables the model to learn the image relationship between specific video frames and the entire video, as well as the text relationship with the descriptive text, during training, thereby improving the video reconstruction capability of the finally trained model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of algorithm model training, in particular to a model training method and device based on images and texts. BACKGROUND

[0002] With the development of algorithm technology, more and more enterprises begin to use algorithm models to perform video-related data prediction tasks, such as video reconstruction, which requires algorithm models to fully extract the features of videos and learn. However, the existing technology does not consider that video training data can be obtained by copying based on an image of a frame, and does not consider calculating the difference loss between the description text and the video frame in the training, so the model obtained by training cannot achieve the effect of video reconstruction based on an image of a frame. It can be seen that the existing technology has defects and needs to be solved. SUMMARY

[0003] The technical problem to be solved by the present application is to provide a model training determination method and device based on images and texts, which can enable the model to learn the image relationship between a specific video frame and the entire video and the text relationship with the description text in the training, thereby improving the video reconstruction capability of the model obtained by the final training.

[0004] To solve the above technical problem, the present application discloses a model training method based on images and texts, which comprises:

[0005] determining a training video and a corresponding description text for training a model;

[0006] frame extraction is performed on the training video to obtain a plurality of training video frames corresponding to the training video;

[0007] determining a target feature frame image from the plurality of training video frames, and copying the target feature frame image to obtain a plurality of copied feature frame images;

[0008] inputting the plurality of copied feature frame images and the description text into a video reconstruction prediction model for training;

[0009] calculating a video loss function value between a plurality of predicted video frames output by the video reconstruction prediction model and the plurality of training video frames input in the training, and a text loss function value between the plurality of predicted video frames and the input description text;

[0010] optimizing the model parameters of the video reconstruction prediction model according to the video loss function value and the text loss function value until convergence, to obtain a trained video reconstruction prediction model.

[0011] As an optional implementation, in the first aspect of the present application, the frame extraction operation on the training video is performed to obtain a plurality of training video frames corresponding to the training video, comprising:

[0012] According to the video parameters of the training video, the frame extraction interval corresponding to the training video is determined;

[0013] According to the frame extraction interval, the frame extraction operation is performed on the training video to obtain a plurality of training video frames corresponding to the training video.

[0014] As an optional implementation, in the first aspect of the present application, according to the video parameters of the training video, the frame extraction interval corresponding to the training video is determined, comprising:

[0015] The picture change parameter of the training video is determined;

[0016] According to the picture change parameter and the preset parameter threshold condition, the frame extraction interval corresponding to the training video is determined;

[0017] And / or,

[0018] The video scene parameter of the training video is determined;

[0019] According to the video scene parameter and the preset scene frame extraction corresponding relationship, the frame extraction interval corresponding to the training video is determined.

[0020] As an optional implementation, in the first aspect of the present application, the frame extraction interval comprises a plurality of different frame extraction intervals; according to the frame extraction interval, the frame extraction operation is performed on the training video to obtain a plurality of training video frames corresponding to the training video, comprising:

[0021] According to the plurality of different frame extraction intervals, the frame extraction operation is performed on the training video to obtain a plurality of first training video frame groups corresponding to the training video respectively; each first training video frame group is used as single input training data when training the video reconstruction prediction model.

[0022] As an optional implementation, in the first aspect of the present application, the video reconstruction prediction model comprises a video reconstruction network and a text reconstruction network; the video reconstruction network is used to receive the plurality of copied representation frame images and the description text and reconstruct the plurality of predicted video frames; the text reconstruction network is used to extract the predicted text features corresponding to the plurality of predicted video frames; the predicted text features are used to compare with the description text to calculate the text loss function value; the text reconstruction network is trained and converged by the training data set comprising a plurality of training images and corresponding training description texts;

[0023] and the model parameters of the video reconstruction prediction model are optimized according to the video loss function value and the text loss function value until convergence, to obtain the trained video reconstruction prediction model, including:

[0024] The parameters of the text reconstruction network are kept unchanged in the training, and the network parameters of the video reconstruction network are optimized according to the video loss function value and the text loss function value until convergence, to obtain the trained video reconstruction network.

[0025] As an optional implementation, in the first aspect of the present application, the text reconstruction network comprises an embedding module, a Transformer module and a fully connected layer module.

[0026] As an optional implementation, in the first aspect of the present application, the optimization of the model parameters of the video reconstruction prediction model according to the video loss function value and the text loss function value until convergence comprises:

[0027] The weighted sum value of the video loss function value and the text loss function value is calculated;

[0028] According to the weighted sum value, the model parameters of the video reconstruction prediction model are optimized until convergence.

[0029] As an optional implementation, in the first aspect of the present application, the video loss function value is calculated as follows:

[0030] For any predicted video frame, the frame loss function value between the predicted video frame and the corresponding training video frame is calculated;

[0031] The average value of the frame loss function values of all the predicted video frames is calculated to obtain the loss function value between the plurality of predicted video frames and the plurality of input training video frames.

[0032] The second aspect of the present application discloses a model training device based on images and texts, the device comprising:

[0033] A data determination module is configured to determine training videos and corresponding description texts for training a model.

[0034] A frame extraction operation module is configured to perform frame extraction operation on the training videos to obtain a plurality of training video frames corresponding to the training videos.

[0035] An image copying module is configured to determine target feature frame images from the plurality of training video frames, and copy the target feature frame images to obtain a plurality of copied feature frame images.

[0036] a model training module configured to train the video reconstruction prediction model by inputting the plurality of copied representation frames and the description text into the video reconstruction prediction model;

[0037] a loss calculation module configured to calculate, in the training, a video loss function value between a plurality of predicted video frames output by the video reconstruction prediction model and the plurality of training video frames input, and a text loss function value between the plurality of predicted video frames and the description text input;

[0038] a model optimization module configured to optimize model parameters of the video reconstruction prediction model according to the video loss function value and the text loss function value until convergence, to obtain the trained video reconstruction prediction model.

[0039] As an optional implementation, in the second aspect of the present application, the frame extraction operation module comprises:

[0040] an interval determination unit configured to determine a frame extraction interval corresponding to the training video according to a video parameter of the training video;

[0041] a frame extraction operation unit configured to perform frame extraction operation on the training video according to the frame extraction interval, to obtain a plurality of training video frames corresponding to the training video.

[0042] As an optional implementation, in the second aspect of the present application, the interval determination unit determines the frame extraction interval corresponding to the training video according to a video parameter of the training video in the following specific manner:

[0043] determining a picture change parameter of the training video;

[0044] determining the frame extraction interval corresponding to the training video according to the picture change parameter and a preset parameter threshold condition;

[0045] and / or,

[0046] determining a video scene parameter of the training video;

[0047] determining the frame extraction interval corresponding to the training video according to the video scene parameter and a preset scene-frame extraction corresponding relationship.

[0048] As an optional implementation, in the second aspect of the present application, the frame extraction interval comprises a plurality of different frame extraction intervals; and the frame extraction operation unit performs frame extraction operation on the training video according to the frame extraction interval in the following specific manner to obtain a plurality of training video frames corresponding to the training video:

[0049] The training video is subjected to frame extraction operations according to the multiple different frame extraction intervals to obtain multiple first training video frame groups corresponding to the training video; each first training video frame group is used as training data for a single input when training the video reconstruction prediction model.

[0050] As an optional implementation, in a second aspect of the present invention, the video reconstruction prediction model includes a video reconstruction network and a text reconstruction network; the video reconstruction network is used to receive the plurality of replicated representation frame images and the descriptive text and reconstruct the plurality of predicted video frames; the text reconstruction network is used to extract predicted text features corresponding to the plurality of predicted video frames; the predicted text features are used to compare with the descriptive text to calculate the text loss function value; the text reconstruction network is trained and converged using a training dataset including a plurality of training images and corresponding training descriptive texts;

[0051] Furthermore, the specific method by which the model optimization module optimizes the model parameters of the video reconstruction prediction model based on the video loss function value and the text loss function value until convergence, to obtain the trained video reconstruction prediction model, includes:

[0052] During training, the parameters of the text reconstruction network are kept constant. The network parameters of the video reconstruction network are optimized based on the video loss function value and the text loss function value until convergence, thus obtaining the trained video reconstruction network.

[0053] As an optional implementation, in a second aspect of the invention, the text reconstruction network includes an embedding module, a Transformer module, and a fully connected layer module.

[0054] As an optional implementation, in a second aspect of the present invention, the specific method by which the model optimization module optimizes the model parameters of the video reconstruction prediction model until convergence based on the video loss function value and the text loss function value includes:

[0055] Calculate the weighted sum of the video loss function value and the text loss function value;

[0056] Based on the weighted sum, the model parameters of the video reconstruction prediction model are optimized until convergence.

[0057] As an optional implementation, in the second aspect of the present invention, the video loss function value is calculated as follows:

[0058] For any of the predicted video frames, calculate the frame loss function value between the predicted video frame and the corresponding training video frame;

[0059] Calculate the average of the frame loss function values ​​for all the predicted video frames to obtain the loss function values ​​between the multiple predicted video frames and the multiple input training video frames.

[0060] A third aspect of the present invention discloses another image and text-based model training apparatus, the apparatus comprising:

[0061] Memory containing executable program code;

[0062] A processor coupled to the memory;

[0063] The processor calls the executable program code stored in the memory to execute some or all of the steps in the image and text-based model training method disclosed in the first aspect of the present invention.

[0064] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute some or all of the steps in the image and text-based model training method disclosed in the first aspect of the present invention.

[0065] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0066] In this embodiment of the invention, a training video and corresponding descriptive text for training the model are determined; a frame extraction operation is performed on the training video to obtain multiple training video frames corresponding to the training video; a target representation frame image is determined from the multiple training video frames, and the target representation frame image is copied to obtain multiple copied representation frame images; the multiple copied representation frame images and the descriptive text are input into the video reconstruction prediction model for training; during the training, the video loss function value between the multiple predicted video frames output by the video reconstruction prediction model and the multiple training video frames input, and the text loss function value between the multiple predicted video frames and the input descriptive text are calculated; the model parameters of the video reconstruction prediction model are optimized according to the video loss function value and the text loss function value until convergence, to obtain the trained video reconstruction prediction model. As can be seen, the present invention can extract multiple video frame images from the training video, and copy the image of a certain frame to obtain video training data. At the same time, the difference loss calculation between the descriptive text and the video frame is considered in the training, so that the model can learn the image relationship between a specific video frame and the whole video, as well as the text relationship with the descriptive text, thereby improving the video reconstruction ability of the finally trained model. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a schematic flowchart of a model training method based on images and text disclosed in an embodiment of the present invention;

[0069] Figure 2 This is a schematic flowchart of another image and text-based model training method disclosed in an embodiment of the present invention;

[0070] Figure 3 This is a schematic diagram of the structure of a model training device based on images and text disclosed in an embodiment of the present invention;

[0071] Figure 4 This is a schematic diagram of another image and text-based model training device disclosed in an embodiment of the present invention;

[0072] Figure 5 This is a schematic diagram of another image and text-based model training device disclosed in an embodiment of the present invention. Detailed Implementation

[0073] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0075] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0076] This invention discloses a model training method and apparatus based on images and text. It can extract multiple video frame images from a training video, and reproduce video training data based on a specific frame image. Simultaneously, it considers the difference loss calculation between the descriptive text and the video frames during training, thereby enabling the model to learn the image relationship between a specific video frame and the entire video, as well as the text relationship with the descriptive text, thus improving the video reconstruction capability of the final trained model. These will be described in detail below.

[0077] Example 1

[0078] Please see Figure 1 , Figure 1 This is a flowchart illustrating a model training method based on images and text disclosed in an embodiment of the present invention. Wherein, Figure 1 The described method is applied in a video data processing device, which can be a corresponding processing terminal, processing equipment, or processing server. The server can be a local server or a cloud server; this embodiment of the invention does not impose any limitations. Figure 1 As shown, this image and text-based model training method can include the following operations:

[0079] 101. Determine the training videos and corresponding descriptive text to be used to train the model.

[0080] Optionally, the descriptive text is related to the content of the training video. It can be written by the operator based on the content of the training video, or it can be automatically predicted and generated by other video content algorithms. This invention does not limit this.

[0081] Optionally, the video content of the training video can be determined based on the target processing video content of the model to be trained. It can also include multiple types of video content to improve the adaptive characteristics of the trained model. Optionally, the video content of the training video can be a continuous action or a continuous scene. Since the solution of this invention incorporates descriptive text, the camera angles or scenes in the training video content can switch.

[0082] 102. Perform frame extraction on the training video to obtain multiple training video frames corresponding to the training video.

[0083] Optionally, the frame extraction operation can be performed manually or automatically by an algorithm according to a predetermined frame extraction interval; this invention does not impose any limitation on this.

[0084] 103. Determine the target representation frame image from multiple training video frames, and copy the target representation frame image to obtain multiple copied representation frame images.

[0085] Optionally, the target representation frame image can be the first training video frame among multiple training video frames, i.e., the first frame image. This setting allows the model to learn the image relationship between the first frame image and the entire video. Alternatively, the target representation frame image can also be a video frame among multiple training video frames that plays a key representational role, such as a frame showing a landmark or representing a turning point in the plot. This can also be used to train the model to learn the image relationship between keyframe images and the entire video. Optionally, the number of duplicate representation frame images input into the model for each training iteration can be the same as the number of training video frames to facilitate subsequent loss calculation.

[0086] 104. Train the video reconstruction prediction model by inputting multiple replicated representation frame images and descriptive text.

[0087] 105. During training, calculate the video loss function values ​​between multiple predicted video frames output by the video reconstruction prediction model and multiple training video frames input, as well as the text loss function values ​​between multiple predicted video frames and the input descriptive text.

[0088] Optionally, the loss function value can be an L1 loss function value, an L2 loss function value, or other loss function values ​​suitable for calculating the similarity between images or texts; this invention does not limit the specific loss function value.

[0089] 106. Optimize the model parameters of the video reconstruction prediction model based on the video loss function value and the text loss function value until convergence, and obtain the trained video reconstruction prediction model.

[0090] Optionally, the gradient descent method can be used to continuously optimize the model parameters until the loss function value reaches its minimum, thereby enabling the model to converge and obtaining a trained video reconstruction prediction model.

[0091] As can be seen, the method described in the embodiments of the present invention can extract multiple video frame images from the training video, and copy the image of a certain frame to obtain video training data. At the same time, the difference loss calculation between the descriptive text and the video frame is considered in the training, so that the model can learn the image relationship between a specific video frame and the whole video, as well as the text relationship with the descriptive text, thereby improving the video reconstruction ability of the finally trained model.

[0092] As an optional implementation, the video reconstruction prediction model includes a video reconstruction network and a text reconstruction network. The video reconstruction network is used to receive multiple replicated representation frame images and descriptive text and reconstruct multiple predicted video frames. The text reconstruction network is used to extract the predicted text features corresponding to the multiple predicted video frames.

[0093] Specifically, the predicted text features are compared with the descriptive text to calculate the text loss function value. The text reconstruction network is trained and converged through a training dataset that includes multiple training images and corresponding training descriptive texts. In other words, it is trained before the video reconstruction prediction model is trained and can be directly used to extract text features.

[0094] Optionally, the text reconstruction network described above can be trained in advance using the original video frames and corresponding text information. That is, the video frames are input to the text reconstruction module, the SmoothL1 loss between the output feature vector and the text feature vector is calculated, and the text reconstruction module is trained by optimizing this loss value until convergence.

[0095] Accordingly, the steps described above, including optimizing the model parameters of the video reconstruction prediction model based on the video loss function values ​​and the text loss function values ​​until convergence, to obtain the trained video reconstruction prediction model, include:

[0096] During training, the parameters of the text reconstruction network are kept constant. The network parameters of the video reconstruction network are optimized based on the video loss function value and the text loss function value until convergence, resulting in a well-trained video reconstruction network.

[0097] Specifically, during training, the parameters of the text reconstruction network are frozen, and only the network parameters of the video reconstruction network are optimized. Finally, the trained video reconstruction network can be used for video reconstruction.

[0098] As an optional implementation, the text reconstruction network includes an embedding module, a Transformer module, and a fully connected layer module.

[0099] Specifically, a text reconstruction network can be constructed to maintain the semantic association between video frames and text. It consists of an embedding layer, two Transformer modules and a fully connected layer. It receives the video frames reconstructed by the decoder of the video reconstruction network, reconstructs the text feature vectors from the video frames through the text reconstruction network, and optimizes the distance between the text feature vectors and the initial encoded feature vectors corresponding to the text through the text loss function.

[0100] As an optional implementation, the video reconstruction network includes an encoder and a decoder, wherein the encoder is used to extract features, and the decoder is used to reconstruct the features. Optionally, the encoder includes a video embedding layer, a text embedding layer, a feature fusion layer, and a first Transformer layer. The video embedding layer receives training video frames and processes them to obtain video features; the text embedding layer receives descriptive text and processes it to obtain text features; and the feature fusion layer fuses the video features and text features to obtain training features, which are then input into the first Transformer layer.

[0101] The video embedding layer includes two-dimensional convolutional layers and / or three-dimensional convolutional layers.

[0102] Optionally, a two-dimensional convolutional patch embedding layer can be constructed as a video embedding layer, which is used to replace the high-dimensional original image features with a low-dimensional vector. Optionally, the patch embedding layer consists of a single convolutional layer with equal kernel size and stride, and the number of output channels can be selected as 768 (other options are also possible).

[0103] Optionally, a 3D convolutional video patch embedding layer can be constructed as the video embedding layer, i.e., a 3D PatchEmbedding layer. This 3D convolutional layer performs convolutions simultaneously in both spatial and temporal dimensions, thereby extracting additional correlation features between consecutive frames. Specifically, to ensure that each frame possesses correlation features, the stride of the convolution is not necessarily equal to the depth of the convolution. That is, the stride can be equal to 1. For example, if the convolution depth is 2 and the stride is 1, the first and second frames will obtain a feature vector through 3D convolution, and subsequently, the second and third frames will also obtain a feature vector, and so on. If the convolution depth is 2 and the stride is also 2, the first and second frames will convolve to obtain a feature vector, and then the convolutional features between the third and fourth frames will be directly calculated, without calculating the features between the second and third frames.

[0104] Specifically, this video embedding layer converts the input video frames into corresponding feature vectors. Taking a specific implementation of a two-dimensional convolutional image patch embedding layer as an example, consider 30 video frames, each with an image size of 224*224, resulting in dimensions (30, 3, 224, 224), where 3 represents the RGB three channels. The convolution kernel size is (16, 16), and the stride is 16. After conversion, it becomes (30, 768, 14, 14), which is then transformed into a feature vector (30, 768, 196). Subsequently, the order of the dimensions is reversed, and a class label is added at the initial position of dimension 196 for downstream tasks such as classification. The final feature vector has dimensions (30, 197, 768).

[0105] Optionally, the text embedding layer may include a Tokenizer module and an Embedding module, wherein the Tokenizer module is used to convert the text characters of the input descriptive text into token indices, and the Embedding module encodes the token indices into feature vectors.

[0106] Optionally, the feature fusion layer may include a first fully connected layer, a second fully connected layer, a feature fusion module, and a GELU activation layer. The input feature dimensions of the first and second fully connected layers must be equal to facilitate feature fusion. The first fully connected layer receives the output features from the video embedding layer and performs feature space transformation. The second fully connected layer receives the output features from the text embedding layer and performs feature space transformation. The feature fusion module can use feature addition and averaging, concatenation, or other fusion methods to fuse the features of the first and second fully connected layers, and then transforms them through the GELU activation layer to obtain the final fused features.

[0107] In a specific scheme, to better fuse video frame features and text features, two fully connected layers are constructed for feature space transformation. The feature vector dimension of the video frame is (30, 197, 768). Since no position markers are added, the feature vector dimension of the text features is (1, 196, 768) when using the maximum text length of 196. The two types of features are concatenated on the first dimension (starting from 0), and then the dimensional order of the concatenated features is changed, that is: (30, 197+196, 768) -> (30, 768, 197+196). Then, it is input into a fully connected layer, and the output dimension is (30, 768, 197). Finally, a GELU activation layer is used for non-linear transformation to obtain the fused features, and the dimension is transformed again to (30, 197, 768).

[0108] Optionally, the first Transformer layer includes multiple stacked Transformer modules, the structure of which can be referenced from the encoder structure of the VIT (Vision Transformer) network.

[0109] As can be seen, by implementing this optional implementation method, an encoder structure that can fully extract and fuse features from video frames and descriptive text can be constructed, resulting in better prediction performance of the model.

[0110] As an optional implementation, the decoder includes a first fully connected layer, a second Transformer layer, and a second fully connected layer.

[0111] Specifically, the decoder first employs a fully connected layer to fuse and transform the encoder's features. This is followed by multiple Transformer modules, and finally, another fully connected layer is used to generate pixel-level video frames.

[0112] As can be seen, by implementing this optional implementation method, a decoder structure that can fully reconstruct the features of video frames can be constructed, thereby improving the prediction performance of the model.

[0113] As an optional implementation, the step described above, optimizing the model parameters of the video reconstruction prediction model based on the video loss function value and the text loss function value until convergence, includes:

[0114] Calculate the weighted sum of the video loss function values ​​and the text loss function values;

[0115] The model parameters of the video reconstruction prediction model are optimized based on the weighted summation until convergence.

[0116] Optionally, the weights corresponding to the video loss function value and the text loss function value can be adjusted by technicians based on experimental or empirical values ​​to achieve the best representation effect.

[0117] Optionally, the video loss function value can be calculated as follows:

[0118] For any predicted video frame, calculate the frame loss function value between the predicted video frame and the corresponding training video frame;

[0119] Calculate the average of the frame loss function values ​​for all predicted video frames to obtain the loss function values ​​between multiple predicted video frames and multiple input training video frames.

[0120] In a specific implementation, the final video frame reconstruction process involves two parts in its training and optimization loss. The first part is the image reconstruction loss between the decoded video frame and the original video frame, which can be achieved using SmoothL1 loss. The reconstruction loss between each decoded video frame and its corresponding ground truth frame is calculated, and the average loss for each frame is taken as the final reconstruction loss. The second part is the semantic association loss between the video and the text, i.e., the text loss function. This is achieved by using the trained text reconstruction module to re-represent the text's feature vector from the reconstructed video frame and calculating the distance between it and the original text's feature vector. This part of the loss is also optimized using SmoothL1 loss. During this training process, the parameters of the text reconstruction module are frozen and do not participate in the parameter update process.

[0121] Specifically, since the video frames input to the encoder are all identical (composed of stacked specific frame images), the decoder's task is actually to perform local image corrections based on the input text description within these existing video frames. If only the video frame reconstruction loss is used, during training, some frames may reconstruct well while others reconstruct poorly. Furthermore, the semantic information expressed by the reconstructed video frame sequence is chaotic because it only considers the similarity between the reconstructed and original frames, neglecting the semantic continuity between the reconstructed frames as a whole. Therefore, a text loss function is needed to re-represent the text information from the reconstructed video frames. By narrowing the distance between the feature vectors of the reconstructed and original texts, the reconstructed video frame sequence can also express complete text information, thus maintaining the semantic continuity of the reconstructed video frames.

[0122] Specifically, the two losses mentioned above are balanced by setting different weighting factors. The video loss function considers the similarity between each pair of video frames from the perspective of each frame, while the text loss function considers semantic relevance as a whole. Since the parameters of the text reconstruction network are fixed, the value of the text loss function directly depends on the overall similarity between the reconstructed video frame sequence and the original video frame sequence. The closer the reconstructed video frame sequence is to the original video frame sequence, the closer the reconstructed text features are to the original text features, ultimately leading to a model with better video reconstruction performance.

[0123] Example 2

[0124] Please see Figure 2 , Figure 2 This is a flowchart illustrating another image and text-based model training method disclosed in an embodiment of the present invention. Wherein, Figure 2 The described method is applied in a video data processing device, which can be a corresponding processing terminal, processing equipment, or processing server. The server can be a local server or a cloud server; this embodiment of the invention does not impose any limitations. Figure 2 As shown, this image and text-based model training method can include the following operations:

[0125] 201. Determine the training videos and corresponding descriptive text to be used to train the model.

[0126] 202. Determine the frame extraction interval corresponding to the training video based on the video parameters of the training video.

[0127] Optionally, video parameters may include image change parameters and / or video scene parameters, which can be used to characterize the complexity of the video content. When the complexity is high, the frame extraction interval should be shortened to obtain more video frames so that the video frames can fully reflect the video content. When the complexity is low, the frame extraction interval can be appropriately increased to obtain fewer video frames.

[0128] 203. Perform frame extraction on the training video according to the frame extraction interval to obtain multiple training video frames corresponding to the training video.

[0129] Optionally, the frame extraction interval described in this invention can be a time interval or a frame number interval.

[0130] Optionally, multiple frame extraction points can be determined based on chronological or frame-by-frame order, with the extraction interval as the interval. Then, the video frames corresponding to the extraction points in the training video are obtained to acquire multiple training video frames. It should be noted that the time interval or frame-by-frame interval between adjacent video frames in the multiple training video frames is not necessarily strictly the same as the aforementioned frame extraction interval. This is because when extracting frames according to chronological or frame-by-frame intervals, if the interval at the end is insufficient, the last frame may be directly determined as a training video frame.

[0131] Optionally, after obtaining multiple training video frames corresponding to the training video, the multiple training video frames can be bound to and saved with the identifier of the training video (such as storage path, video ID, etc.) for subsequent training.

[0132] 204. Determine the target representation frame image from multiple training video frames, and copy the target representation frame image to obtain multiple copied representation frame images.

[0133] 205. Train the video reconstruction prediction model by inputting multiple replicated representation frame images and descriptive text.

[0134] 206. During training, calculate the video loss function values ​​between multiple predicted video frames output by the video reconstruction prediction model and multiple training video frames input, as well as the text loss function values ​​between multiple predicted video frames and the input descriptive text.

[0135] 207. Optimize the model parameters of the video reconstruction prediction model based on the video loss function value and the text loss function value until convergence, and obtain the trained video reconstruction prediction model.

[0136] The specific technical details and explanations of technical terms for steps 201 and 204-207 above can be found in the descriptions of steps 101 and 103-106 in Implementation 1, and will not be repeated here.

[0137] As can be seen, implementing the method described in the embodiments of the present invention can determine a reasonable frame extraction interval, thereby extracting appropriate video frames that can represent the video content, resulting in better prediction performance of the trained model.

[0138] As an optional implementation, the frame extraction interval includes multiple different frame extraction intervals. Accordingly, the step described above, performing frame extraction on the training video according to the frame extraction interval to obtain multiple training video frames corresponding to the training video, may include:

[0139] The training video is subjected to frame extraction operations at multiple different frame extraction intervals to obtain multiple training video frame groups corresponding to the training video.

[0140] Optionally, each group of training video frames is used as training data as a single input when training the video reconstruction prediction model.

[0141] Optionally, each group of training video frames may include multiple training video frames.

[0142] Specifically, to augment the dataset, the same training video may be sampled at various intervals, such as 5 frames per second and 2 frames per second, resulting in a difference in the speed of the video frame sequence. Optionally, frame sampling of the training video can be performed before the start of the entire training process. Early frame sampling can significantly reduce training time. If dynamic frame sampling intervals are used to sample short videos during the training phase, although this can greatly enrich the dataset and achieve data augmentation, the frame sampling speed is often slow, requiring re-sampling for each training iteration, which undoubtedly severely hinders the entire training process.

[0143] As an optional implementation, the step described above, which involves performing frame extraction on the training video according to the frame extraction interval to obtain multiple training video frames corresponding to the training video, includes:

[0144] The training video is subjected to frame extraction operation according to the frame extraction interval to obtain multiple candidate video frames corresponding to the training video.

[0145] For any two adjacent candidate video frames, calculate the image similarity between the two candidate video frames;

[0146] Determine whether the image similarity meets the preset similarity threshold condition;

[0147] If the judgment result is yes, then the two candidate video frames are determined as key video frames;

[0148] Based on all the key video frames in the multiple candidate video frames, determine the multiple training video frames corresponding to the training video.

[0149] Optionally, the image similarity can be a structural similarity parameter or a cosine similarity parameter. Optionally, the similarity threshold condition can be that the image similarity is greater than a certain similarity threshold. In this case, the two candidate video frames are not similar, and it can be considered that an image switch occurred between these two frames. These two frames are then retained as critical keyframes.

[0150] As an optional implementation, the step described above, determining the multiple training video frames corresponding to the training video based on all key video frames among the multiple candidate video frames, includes:

[0151] For the candidate video frames other than the key video frame among multiple candidate video frames, a frame extraction operation is performed according to the second frame extraction interval to obtain multiple extracted video frames; wherein, the second frame extraction interval is greater than the frame extraction interval.

[0152] All key video frames and extracted video frames are identified as multiple training video frames corresponding to the training video.

[0153] Optionally, the second frame extraction interval can be determined as follows:

[0154] Determine the total number of frames for all other candidate video frames;

[0155] The second frame skipping interval is determined based on the total number of frames and the preset frame-interval correspondence.

[0156] The second frame extraction interval is directly proportional to the total number of frames. That is, the more total frames there are, the more video content remains. Therefore, the larger the second frame extraction interval, the fewer frames are extracted. This is because for videos with a lot of content but few scene changes, it is not necessary to extract too many frames, and vice versa.

[0157] Optionally, determining multiple training video frames corresponding to the training video based on all key video frames among multiple candidate video frames may also include:

[0158] For multiple candidate video frames other than the key video frame, calculate the image similarity between at least two other candidate video frames.

[0159] Then, all candidate video frames whose image similarity meets the similarity threshold condition are retained to determine multiple training video frames corresponding to the training video.

[0160] Specifically, the video can first be frame-sampling at a relatively small interval. For example, if the video has a frame rate of 30, a 10-frame-per-second extraction interval would yield 50 frames for a 5-second video. Then, the similarity between any two consecutive frames within these 50 frames is calculated. When the similarity exceeds a set maximum threshold m, a scene transition is considered to have occurred between these two frames, and these two frames are retained as critical keyframes. After retaining all keyframes where scene transitions have occurred using this method, consecutive frames other than the keyframes are then subjected to equally spaced frame extraction. The second extraction interval can be determined based on the total length of the final sequence, or it can be adaptively selected by calculating the similarity between consecutive frames. For example, if the similarity between the first and second frames is less than a set minimum threshold n, indicating a small difference and high redundancy, only the first frame is retained, and the second frame is discarded. The third frame is then compared with the first frame; if the similarity is less than the set threshold n, the subsequent frame is discarded. This process continues until the subsequent frame exceeds the minimum threshold n or becomes a critical keyframe, at which point it is retained.

[0161] The keyframe selection method for scene transitions proposed above is intended to enable the model to learn the characteristic changes of scene transitions, so that the final reconstructed video can also exhibit certain scene transition effects or scene continuity.

[0162] As an optional implementation, the step of determining the frame extraction interval corresponding to the training video based on the video parameters of the training video includes:

[0163] Determine the parameters of the image changes in the training video;

[0164] The frame extraction interval corresponding to the training video is determined based on the image change parameters and preset parameter threshold conditions.

[0165] Optionally, the image change parameter can be an optical flow value parameter between different frames of the training video, such as the average, maximum, or weighted average of the optical flow motion between all adjacent frames. Correspondingly, the parameter threshold condition can be an optical flow value threshold condition.

[0166] Preferably, the optical flow value between each frame can be calculated, and the frame skipping interval can be determined by limiting the amount of optical flow movement between each frame. For example, a threshold for the amount of optical flow change can be set. If, after statistical analysis, it is found that the amount of optical flow change every K frames just exceeds this threshold, then the frame skipping interval can be set to K frames. If the amount of optical flow movement between each frame is large, a smaller frame skipping interval can be selected.

[0167] As can be seen, by implementing this optional implementation method, a more reasonable frame extraction interval can be determined based on the image change parameters, thereby enabling the video content of video frames to be determined reasonably and efficiently, thus improving the training efficiency of the model.

[0168] As an optional implementation, the step of determining the frame extraction interval corresponding to the training video based on the video parameters of the training video includes:

[0169] Determine the video scene parameters of the training video;

[0170] The frame extraction interval for the training video is determined based on the video scene parameters and the preset scene frame extraction correspondence.

[0171] Optionally, the video scene parameter can be the scene type of the training video. Optionally, the scene frame extraction correspondence is used to indicate the frame extraction interval corresponding to different scene types. For example, if a short video mainly shows changes in human movement, such as a layup, the movement is relatively fast, and the changes between each frame are relatively obvious; in this case, a smaller frame extraction interval can be used. If the content of the short video changes slowly or follows a certain pattern, such as a car slowly driving on a mountain road, a dashcam recording the road conditions ahead, and trees on both sides moving rhythmically towards the camera; this kind of regular or slow change can use a larger frame extraction interval. Specifically, the frame extraction interval varies depending on the different task scenarios.

[0172] Optionally, the number of different types of scenes appearing in the training video can be used to indicate the degree of scene change or the complexity of the video content. Optionally, the scene frame extraction correspondence is used to indicate the frame extraction interval corresponding to the number of different types of scenes. Generally speaking, the frame extraction interval is inversely proportional to the number of scenes. That is, the more scenes there are, the more changes there are in the video scenes, and the more complex the content is. In this case, the smaller the frame extraction interval, the more video frames are obtained, and vice versa.

[0173] As can be seen, by implementing this optional implementation method, a more reasonable frame extraction interval can be determined based on the video scene parameters, thereby enabling the video content of the video frame to be determined reasonably and efficiently, thus improving the training efficiency of the model.

[0174] As an optional implementation, in the above steps, after performing frame extraction on the training video according to the frame extraction interval to obtain multiple training video frames corresponding to the training video, the method further includes:

[0175] Determine whether the number of multiple training video frames exceeds a preset first frame number threshold;

[0176] If so, divide multiple training video frames into at least two training video frame groups with a number of video frames less than or equal to the first frame number threshold.

[0177] Each group of partitioned training video frames is used as training data as a single input when training the video reconstruction prediction model.

[0178] Specifically, after extracting frames from the video, the final frame count is calculated. To avoid excessive memory usage during training, a first frame count threshold, such as 30 frames, is set. If the result of extracting frames from a video is 54, the first 30 frames should be extracted as a sequence, and the last 24 frames should be extracted as a second sequence to segment the training data of a single input.

[0179] As can be seen, by implementing this optional implementation method, multiple training video frames can be divided into at least two training video frame groups with a number of video frames less than or equal to a first frame number threshold. This allows for the reasonable and efficient determination of training data for a single input, thereby reducing training costs and improving the training efficiency of the model.

[0180] As an optional implementation, in the above steps, after performing frame extraction on the training video according to the frame extraction interval to obtain multiple training video frames corresponding to the training video, the method further includes:

[0181] Determine whether the number of multiple training video frames is less than a preset second frame number threshold;

[0182] If so, extract video frames from the training video and fill them into multiple training video frames until the number of multiple training video frames equals the second frame number threshold.

[0183] Specifically, after extracting frames from the video, the final frame count is calculated. To avoid some videos having too few frames and failing to achieve the expected training effect, a second frame count threshold, such as 30 frames, is set. If the total number of frames in the video is less than 30, such as only 24 frames, 6 frames are randomly selected and copied and inserted in the original time order to complete the 30 frames.

[0184] As can be seen, by implementing this optional implementation method, video frames can be extracted from the training video and filled into multiple training video frames until the number of multiple training video frames is equal to the second frame number threshold. This can reasonably and efficiently complete the amount of training data input in a single instance, thereby improving the training efficiency and effectiveness of the model.

[0185] Example 3

[0186] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a model training device based on images and text disclosed in an embodiment of the present invention. Wherein, Figure 3 The described apparatus can be applied to corresponding video data processing devices, which can be corresponding processing terminals, processing equipment, or processing servers. The server can be a local server or a cloud server; this embodiment of the invention does not impose limitations. Figure 3 As shown, the device may include:

[0187] The data determination module 301 is used to determine the training video and corresponding descriptive text used to train the model;

[0188] The frame extraction module 302 is used to perform frame extraction on the training video to obtain multiple training video frames corresponding to the training video.

[0189] The image copying module 303 is used to determine the target representation frame image from multiple training video frames and copy the target representation frame image to obtain multiple copied representation frame images.

[0190] The model training module 304 is used to train the video reconstruction prediction model by inputting multiple replicated representation frame images and descriptive text.

[0191] The loss calculation module 305 is used to calculate the video loss function value between multiple predicted video frames output by the video reconstruction prediction model and multiple training video frames input during training, as well as the text loss function value between multiple predicted video frames and input descriptive text.

[0192] The model optimization module 306 is used to optimize the model parameters of the video reconstruction prediction model based on the video loss function value and the text loss function value until convergence, so as to obtain the trained video reconstruction prediction model.

[0193] As an optional implementation method, such as Figure 4 As shown, the frame extraction module 302 includes:

[0194] The interval determination unit 3021 is used to determine the frame extraction interval corresponding to the training video based on the video parameters of the training video.

[0195] The frame extraction unit 3022 is used to perform frame extraction operations on the training video according to the frame extraction interval to obtain multiple training video frames corresponding to the training video.

[0196] As an optional implementation, the interval determination unit 3021 determines the specific method of the frame extraction interval corresponding to the training video based on the video parameters of the training video, including:

[0197] Determine the parameters of the image changes in the training video;

[0198] The frame extraction interval corresponding to the training video is determined based on the image change parameters and the preset parameter threshold conditions.

[0199] And / or,

[0200] Determine the video scene parameters of the training video;

[0201] The frame extraction interval for the training video is determined based on the video scene parameters and the preset scene frame extraction correspondence.

[0202] As an optional implementation, the frame extraction interval includes multiple different frame extraction intervals; the specific method by which the frame extraction operation unit 3022 performs frame extraction operation on the training video according to the frame extraction interval to obtain multiple training video frames corresponding to the training video includes:

[0203] The training video is subjected to frame extraction operations according to multiple different frame extraction intervals to obtain multiple first training video frame groups corresponding to the training video; each first training video frame group is used as training data as a single input when training the video reconstruction prediction model.

[0204] As an optional implementation, the video reconstruction prediction model includes a video reconstruction network and a text reconstruction network; the video reconstruction network is used to receive multiple duplicate representation frame images and descriptive text and reconstruct multiple predicted video frames; the text reconstruction network is used to extract predicted text features corresponding to multiple predicted video frames; the predicted text features are compared with the descriptive text to calculate the text loss function value; the text reconstruction network is trained and converged through a training dataset including multiple training images and corresponding training descriptive text.

[0205] Furthermore, the model optimization module 306 optimizes the model parameters of the video reconstruction prediction model based on the video loss function value and the text loss function value until convergence, obtaining the specific method of the trained video reconstruction prediction model, including:

[0206] During training, the parameters of the text reconstruction network are kept constant. The network parameters of the video reconstruction network are optimized based on the video loss function value and the text loss function value until convergence, resulting in a well-trained video reconstruction network.

[0207] As an optional implementation, the text reconstruction network includes an embedding module, a Transformer module, and a fully connected layer module.

[0208] As an optional implementation, the specific method by which the model optimization module 306 optimizes the model parameters of the video reconstruction prediction model until convergence based on the video loss function value and the text loss function value includes:

[0209] Calculate the weighted sum of the video loss function values ​​and the text loss function values;

[0210] The model parameters of the video reconstruction prediction model are optimized based on the weighted summation until convergence.

[0211] As an optional implementation method, the video loss function value is calculated as follows:

[0212] For any predicted video frame, calculate the frame loss function value between the predicted video frame and the corresponding training video frame;

[0213] Calculate the average of the frame loss function values ​​for all predicted video frames to obtain the loss function values ​​between multiple predicted video frames and multiple input training video frames.

[0214] Example 4

[0215] Please see Figure 5 , Figure 5 This is a schematic diagram of another image and text-based model training device disclosed in an embodiment of the present invention. Figure 5 As shown, the device may include:

[0216] Memory 401 storing executable program code;

[0217] Processor 402 coupled to memory 401;

[0218] The processor 402 calls the executable program code stored in the memory 401 to execute some or all of the steps in the image and text-based model training method disclosed in Embodiment 1 or Embodiment 2 of the present invention.

[0219] Example 5

[0220] This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute some or all of the steps in the image and text-based model training method disclosed in Embodiment 1 or Embodiment 2 of this invention.

[0221] The foregoing has described specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than those shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily have to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0222] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, devices, and non-volatile computer-readable storage media are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0223] The apparatus, device, non-volatile computer-readable storage medium and method provided in the embodiments of this specification are corresponding. Therefore, the apparatus, device and non-volatile computer storage medium also have similar beneficial technical effects as the corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the corresponding apparatus, device and non-volatile computer storage medium will not be repeated here.

[0224] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0225] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0226] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0227] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0228] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0229] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0230] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0231] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0232] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0233] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0234] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0235] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0236] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0237] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0238] Finally, it should be noted that the image and text-based model training method and apparatus disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model training method based on images and text, characterized in that, The method comprises: determining a training video and corresponding description text for training a model; frame extraction is performed on the training video to obtain a plurality of training video frames corresponding to the training video; a target feature frame image is determined from the plurality of training video frames, and the target feature frame image is copied to obtain a plurality of copied feature frame images; the plurality of copied feature frame images and the description text are input into a video reconstruction prediction model for training; in the training, a video loss function value between a plurality of predicted video frames output by the video reconstruction prediction model and the plurality of training video frames input, and a text loss function value between the plurality of predicted video frames and the input description text are calculated, the text loss function value is calculated by comparing a predicted text feature and an original text feature of the description text, and the predicted text feature is extracted based on the video reconstruction prediction model and the corresponding plurality of predicted video frames; model parameters of the video reconstruction prediction model are optimized according to the video loss function value and the text loss function value until convergence, so as to obtain a trained video reconstruction prediction model. 2.The image and text based model training method of claim 1, wherein, The frame extraction is performed on the training video to obtain a plurality of training video frames corresponding to the training video, comprising: determining a frame extraction interval corresponding to the training video according to a video parameter of the training video; frame extraction is performed on the training video according to the frame extraction interval to obtain a plurality of training video frames corresponding to the training video. 3.The image and text based model training method of claim 2, wherein, The frame extraction interval corresponding to the training video is determined according to the video parameter of the training video, comprising: determining a picture change parameter of the training video; determining the frame extraction interval corresponding to the training video according to the picture change parameter and a preset parameter threshold condition; and / or, determining a video scene parameter of the training video; determining the frame extraction interval corresponding to the training video according to the video scene parameter and a preset scene-frame extraction corresponding relationship. 4.The image and text based model training method of claim 2, wherein, The frame extraction interval comprises a plurality of different frame extraction intervals; The frame extraction is performed on the training video to obtain a plurality of training video frames corresponding to the training video, comprising: respectively according to the plurality of different frame extraction intervals, frame extraction is performed on the training video to obtain a plurality of first training video frame groups corresponding to the training video; each first training video frame group is used as single-time input training data when training a video reconstruction prediction model. 5.The image and text based model training method of claim 1, wherein, The video reconstruction prediction model comprises a video reconstruction network and a text reconstruction network; the video reconstruction network is used to receive the plurality of copied feature frame images and the description text and reconstruct the plurality of predicted video frames; the text reconstruction network is used to extract a predicted text feature corresponding to the plurality of predicted video frames; the text reconstruction network is trained and converged by a training data set comprising a plurality of training images and corresponding training description texts; and the text loss function value, and the model parameters of the video reconstruction prediction model are optimized until convergence according to the video loss function value and the text loss function value, to obtain the trained video reconstruction prediction model. In the training, the parameters of the text reconstruction network are kept unchanged, and the network parameters of the video reconstruction network are optimized until convergence according to the video loss function value and the text loss function value, to obtain the trained video reconstruction network. 6.The image and text based model training method of claim 5, wherein, The text reconstruction network comprises an embedding module, a Transformer module and a fully connected layer module. 7.The image and text based model training method of claim 1, wherein, The optimization of the model parameters of the video reconstruction prediction model until convergence comprises: The weighted sum value of the video loss function value and the text loss function value is calculated. The model parameters of the video reconstruction prediction model are optimized until convergence according to the weighted sum value. 8.The image and text based model training method of claim 7, wherein, The video loss function value is calculated in the following manner: For any predicted video frame, the frame loss function value between the predicted video frame and the corresponding training video frame is calculated. The average value of the frame loss function values of all the predicted video frames is calculated to obtain the loss function value between the plurality of predicted video frames and the plurality of input training video frames.

9. An image and text based model training apparatus, characterized by comprising: The device comprises: A data determination module for determining training videos and corresponding description texts for training a model; An extracting operation module for extracting frames of the training videos to obtain a plurality of training video frames corresponding to the training videos; An image copying module for determining a target feature frame image from the plurality of training video frames, and copying the target feature frame image to obtain a plurality of copied feature frame images; A model training module for inputting the plurality of copied feature frame images and the description texts into a video reconstruction prediction model for training; A loss calculation module for calculating, in the training, a video loss function value between a plurality of predicted video frames output by the video reconstruction prediction model and a plurality of input training video frames, and a text loss function value between the plurality of predicted video frames and input description texts, the text loss function value being calculated by comparing predicted text features with original text features of the description texts, the predicted text features being extracted based on the video reconstruction prediction model and the corresponding plurality of predicted video frames; A model optimization module for optimizing the model parameters of the video reconstruction prediction model until convergence according to the video loss function value and the text loss function value, to obtain the trained video reconstruction prediction model.

10. An image and text based model training apparatus, characterized by, The device comprises: A memory storing executable program codes; A processor coupled with the memory; The processor invokes the executable program codes stored in the memory to execute the image and text based model training method according to any one of claims 1-8.

Images