Video text generation method and device, equipment and storage medium

Abstract

The invention discloses a video text generation method and device, equipment and a storage medium, and belongs to the technical field of text generation, and the method comprises the steps: generating a text prediction representation according to a visual embedding vector of each video clip in video data; constructing a similarity matrix according to the visual embedding vector and the cosine similarity represented by text prediction; obtaining an optimal transmission matrix according to the similarity matrix and a preset target function, constructing a contrast learning loss according to the optimal transmission matrix, and constructing a target loss function according to the contrast learning loss and the traditional text loss; and training the network model according to the target loss function, and predicting video text content according to the trained model. According to the method, flexible matching between the video clip and the text prediction representation is established by using the optimal transmission algorithm, the transmission distance of the optimal transmission is used as a weight to construct the loss function, and the semantic relationship between the video clip and the text prediction representation is better captured, so that the accuracy and continuity of text generation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video text generation, and in particular to a video text generation method, device, equipment and storage medium. Background Art

[0002] With the proliferation of social platforms and digital devices, a vast amount of video has flooded into our daily lives. Multimodal learning from these videos and the corresponding text has become a key focus in natural language processing (NLP) research. Generating relevant text from videos has emerged as a key task for scene understanding and practical applications. Traditional video captioning methods have a long history of learning from short video clips. However, due to inherent long-term temporal dependencies, they often produce unsatisfactory results when applied to long instructional videos. Inspired by the superior performance of RNNs in learning long-range relations through attention mechanisms, Transformer-based models have emerged in this field. These models employ various strategies rooted in the Transformer architecture to address specific challenges. These methods can be categorized into three categories: 1) The first category is Visual Large Language Models (VLMs), a decoder-only architecture that integrates a visual encoder for text generation. By leveraging the advanced text generation capabilities of large language models, these methods have achieved impressive performance in generation tasks and strong results in zero-shot tasks. 2) The second category focuses on encoder-decoder frameworks, leveraging large video corpora to train video language base models for downstream AI tasks. These models employ different optimization objectives or incorporate different types of video information. 3) The third approach focuses on establishing dependencies between the current and previous segments of the instructional video. Currently, process text generation for long instructional videos is still underdeveloped. Optimizing such methods will help keep pace with current trends in information technology and promote the application of related technologies.

[0003] Among existing methods, traditional video description models (such as those based on RNN / CNN architectures) can handle short video clips, but struggle to model the global temporal relationships of long videos, resulting in redundant and logically disjointed generated text. Transformer-based generative models capture long-range dependencies through self-attention mechanisms, but still rely on coarse-grained video-text holistic matching and lack explicit modeling of fine-grained semantic alignment at the step level. Contrastive learning methods (such as the InfoNCE loss) indiscriminately penalize all negative samples, ignoring the soft positive correlations between potentially semantically related segments (such as the same entity manipulation in different steps), inhibiting the model's learning of fine-grained associations. There is a lack of exploration of step-level dynamic soft alignment between video clips and generated text, particularly lacking optimization for many-to-many allocation. In other words, existing methods fail to effectively balance the tension between temporal modeling of long videos, fine-grained cross-modal alignment, and computational efficiency, resulting in insufficient accuracy in generated text. Errors are particularly prone to occur in critical areas such as cross-step entity recurrence and state transitions (such as the food processing process).

[0004] Therefore, there is an urgent need for a video text generation method that can solve the technical problem that the existing technology cannot accurately generate video text. Summary of the Invention

[0005] In view of this, it is necessary to provide a video text generation method, device, equipment and storage medium that can solve the technical problem of accurately generating video text.

[0006] In order to solve the above technical problems, the present invention provides a method for generating video text, comprising: generating a text prediction representation based on a visual embedding vector of each video clip in the video data, wherein the video data includes entity labels, and the visual embedding vector includes a video vector and an entity vector of the entity label; Constructing a similarity matrix based on the cosine similarity between the visual embedding vector and the text prediction representation; Obtaining an optimal transfer matrix based on the similarity matrix and a preset objective function, constructing a contrastive learning loss based on the optimal transfer matrix, and constructing an objective loss function based on the contrastive learning loss and the loss between the true text and the text prediction representation; The network model is trained according to the target loss function, and the target video is input into the trained model to generate video text content.

[0007] In one possible implementation, generating a text prediction representation based on a visual embedding vector of each video clip in the video data includes: Extracting video vectors and entity vectors for each video clip in the video data; Fusing the video vector and the entity vector to obtain a visual embedding vector; Decoding the visual embedding vector to obtain text prediction information; Average pooling is performed on the text prediction information to obtain a text prediction representation.

[0008] In one possible implementation, determining a video vector and an entity vector for each video segment in the video data includes: Add entity tags to video data based on video content; Divide the video data with entity tags into multiple video segments according to the video content; Encoding each frame of the video clip to obtain a frame video vector, and splicing the frame video vectors to obtain a video vector; Each entity label corresponding to the video clip is encoded to obtain a single entity vector, and the entity vectors are concatenated to obtain an entity vector.

[0009] In one possible implementation, decoding the visual embedding vector to obtain text prediction information includes: Decoding the visual embedding vector to obtain multiple candidate words; Calculating the probability distribution of the words to be selected; Text prediction information is determined based on the probability representation.

[0010] In one possible implementation, the transfer matrix is ​​calculated as:

[0011] The preset objective function is: , in, is the transmission matrix, and The scaling vectors are and The diagonal matrix of S is the similarity matrix, δ is the hyperparameter that controls the smoothness of the transfer distribution, is the entropy term, is the transmission matrix and similarity matrix S The inner product of .

[0012] In one possible implementation, a contrastive learning loss is constructed based on the optimal transfer matrix, and a target loss function is constructed based on the contrastive learning loss and the loss between the true text and the text prediction representation, including: Performing alignment transformation on the optimal transmission matrix to obtain a soft alignment matrix; A contrastive learning loss function is obtained based on the contrastive loss between the visual embedding vector and the text prediction representation in the soft alignment matrix; Constructing a text loss function based on the text prediction information and the real text; The text loss function and the contrastive learning loss function are weighted to obtain the target loss function.

[0013] In one possible implementation, the formula of the contrastive learning loss function is:

[0014] in, To compare the learning loss function, is the soft alignment matrix i Rank k The value of the column, and for matrix, is the similarity matrix i Rank j The value of the column, is the similarity matrix i Rank k The value of the column, is the similarity matrix k Rank j The value of the column, and for matrix, τ is the temperature parameter that controls the softness of the softmax function.

[0015] In a second aspect, the present invention further provides a video text generation device, comprising: a predictive text module for generating a text prediction representation based on a visual embedding vector of each video segment in the video data, wherein the video data includes entity labels, and the visual embedding vector includes a video vector and an entity vector of the entity label; A similarity calculation module, configured to construct a similarity matrix based on the cosine similarity between the visual embedding vector and the text prediction representation; A loss function construction module is used to obtain an optimal transfer matrix based on the similarity matrix and a preset objective function, construct a contrastive learning loss based on the optimal transfer matrix, and construct a target loss function based on the contrastive learning loss and the loss between the true text and the text prediction representation; The text generation module is used to train the network model according to the target loss function, input the target video into the trained model, and generate video text content.

[0016] In a third aspect, a video text generation device includes a memory and a processor, wherein: The memory is used to store programs; The processor is coupled to the memory and is used to execute the program stored in the memory to implement the steps in the above-mentioned video text generation method.

[0017] In a fourth aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned method for generating video text.

[0018] The beneficial effects of the present invention are as follows: first, a long video is segmented according to the video content and divided into video clips of multiple different steps, wherein the video data includes entity labels, the video vector and the entity vector of the entity label of each video clip are extracted, and the video vector and the corresponding entity vector are fused to obtain a visual embedding vector, and a text prediction representation is generated according to the visual embedding vector; then, the cosine similarity between the visual embedding vectors and the text prediction representations of the video clips of different steps is calculated to construct a similarity matrix, and this similarity matrix can reflect the correlation between the video clips; then, the optimal transmission matrix is ​​determined according to the similarity matrix, and the optimal transmission is a method of minimizing the transmission cost according to the difference between two probability distributions to establish an effective correspondence between different distributions, thereby achieving more accurate distribution matching, and realizing a multi-to-many dynamic allocation of video clips and text prediction representations through optimal transmission. At the same time, the transmission distance calculated by the optimal transmission is used as a weight to construct a loss function to alleviate the transition penalty problem of traditional contrastive learning for related negative ideas; finally, the network model is iteratively trained according to the target loss function, and the target video is input into the trained model to obtain the video text content. The present invention generates text prediction representations through video vectors and entity vectors, calculates the similarity matrix between the visual embedding vectors of video vectors and entity vectors and the text prediction representation, uses the optimal transmission algorithm to establish a flexible match between video clips and text prediction representations, uses the transmission distance calculated by the optimal transmission as a weight, constructs a loss function, alleviates the transition penalty problem of traditional contrastive learning for correlated negative samples, and can better capture the semantic relationship between video clips and text prediction representations, thereby improving the accuracy and coherence of text generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0020] Figure 1 A schematic flow chart of an embodiment of a method for generating video text provided by the present invention; Figure 2For the present invention Figure 1 A flow chart of an embodiment of step S101; Figure 3 For the present invention Figure 2 A flow chart of an embodiment of step S201; Figure 4 For the present invention Figure 2 A flow chart of an embodiment of step S203; Figure 5 For the present invention Figure 1 A flow chart of an embodiment of step S103; Figure 6 A schematic structural diagram of an embodiment of a video text generation device provided by the present invention; Figure 7 A schematic structural diagram of an embodiment of the video text generation device provided by the present invention. DETAILED DESCRIPTION

[0021] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0022] In the description of the embodiments of the present invention, unless otherwise specified, “a plurality of” means two or more.

[0023] The terms "first," "second," and so on, used in the embodiments of the present invention are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature designated as "first" or "second" may explicitly or implicitly include at least one such feature.

[0024] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0025] The present invention provides a video text generation method, device, equipment and storage medium, which are described below respectively.

[0026] Figure 1 A flow chart of an embodiment of a method for generating video text provided by the present invention is shown as follows: Figure 1 As shown, the video text generation method includes: S101, generating a text prediction representation based on a visual embedding vector of each video clip in the video data, where the video data includes an entity label, and the visual embedding vector includes a video vector and an entity vector of the entity label; It should be noted that, in this embodiment, the video is processed by video processing software on a mobile terminal, wherein the mobile terminal is an electronic terminal with a video processing function, such as a PC, a mobile terminal, a portable terminal, etc.

[0027] Specifically, entity tags are added to the video data according to the video content. The entity tag is the physical content in a certain video content. For example, in a video tutorial of scrambled eggs with tomatoes, when tomatoes appear in the video, the physical tag added to the video at that moment is tomato. When tomatoes and eggs appear at the same time, two physical tags, tomato and egg, are added to the video respectively. Similarly, according to the operation steps in the video content, the video is divided into multiple video segments. For example, a video tutorial of scrambled eggs with tomatoes can be divided into segments such as accurate ingredients, scrambled eggs, scrambled tomatoes, mixed scrambled, and plating. Each video segment can contain multiple entity tags, or only one or none.

[0028] Furthermore, the video vector and entity vector corresponding to the entity label of each video clip are extracted. The extraction method is not specifically limited. In this embodiment, an encoder is preferably used to encode the video and entity label to obtain the video vector and entity vector. The video vector and the corresponding entity vector of a video clip are fused to obtain a fused embedding representation, namely a visual embedding vector, that can reflect the content of the corresponding step of the video clip. This visual embedding vector not only reflects the content of the video text information in the process of generating the text prediction representation, but also effectively utilizes the entity information, making the generated text information more accurate. Then, based on this visual embedding vector, decoding is performed to generate text information word by word, and the text information is converted into a text prediction representation for the next step.

[0029] S102, constructing a similarity matrix based on the cosine similarity of the visual embedding vector and the text prediction representation; Specifically, for the visual embedding vectors of multiple segments and the corresponding generated text prediction representations, the cosine similarity between them is calculated. In addition, unlike the traditional calculation of the similarity between the visual embedding vector of a segment and its corresponding text prediction representation, the similarity between the visual embedding vectors of all segments and all text prediction representations is calculated. For example, if a video data is divided into four video segments, four visual embedding vectors and four generated text prediction representations will be obtained, and the cosine similarity between them is calculated respectively to form a This similarity matrix can fully explain the association between the generated text prediction representation and the context of multiple video clips.

[0030] Furthermore, the similarity matrix is ​​expressed as: , in, is the cosine similarity between the visual embedding vector of the first video and the textual prediction representation of the first video, is the cosine similarity between the visual embedding vector of the first video and the textual prediction representation of the second video, and so on. is the visual embedding vector of the first video and the n The cosine similarity of the video text prediction representation, For the n The visual embedding vector of the video is n The cosine similarity of the video text prediction representation is constructed. The similarity matrix of .

[0031] S103, obtaining an optimal transfer matrix based on the similarity matrix and a preset objective function, constructing a contrastive learning loss based on the optimal transfer matrix, and constructing an objective loss function based on the contrastive learning loss and the loss between the true text and the text prediction representation; Specifically, this embodiment uses an optimal transmission algorithm to maximize global similarity to achieve flexible and optimal allocation between video clips and physical object labels. The calculation formula of the transmission matrix is: , in, is the transmission matrix, and The scaling vectors are and The diagonal matrix of , S is the similarity matrix, δ is the hyperparameter that controls the smoothness of the transfer distribution, The preset objective function is: , Among them, the transmission matrix The constraints are: and , in, The entropy term is used to promote uniform distribution. Adding the entropy term can make the elements of the transmission matrix more evenly distributed, avoid over-centralization, and thus increase the smoothness and robustness of the distribution. δ is a hyperparameter that controls the smoothness of the transmission distribution. It is used to control the influence of the entropy term. The larger δ is, the greater the influence of the entropy term and the more uniform the distribution. The smaller δ is, the smaller the influence of the entropy term and the more centralized the distribution is. The transmission matrixQ for matrix ， The constraint is that the sum of the probabilities of each segment’s visual embedding vector being assigned to all text prediction representations, and the sum of the probabilities of each text prediction representation being assigned to all segments’ visual embedding vectors, are equal, which reflects the fairness and globality of the distribution. Find an optimal transfer matrix to maximize global similarity, achieve many-to-many dynamic allocation, and allow a video clip to be associated with text prediction representations of multiple steps, breaking through the limitations of traditional hard alignment.

[0032] Furthermore, by converting the optimal transfer matrix into soft alignment probability and calculating the similarity between the visual embedding vector and the text prediction representation to construct the contrastive learning loss, it is possible to more flexibly adjust the attraction and repulsion between the visual embedding vector and the text prediction representation, thereby improving the effect of contrastive learning.

[0033] Furthermore, in order to measure the difference between the predicted text and the real text, the traditional loss function is used to calculate the loss between the predicted text and the real text. At the same time, the contrastive learning loss function is introduced on this basis, and a total target loss function is obtained through weighted fusion. The calculation formula of the total target loss function is: , in, is the target loss function value, is the loss function between the predicted text and the real text, is the contrastive learning loss function, and λ is a hyperparameter.

[0034] S104: Train the network model according to the target loss function, input the target video into the trained model, and generate video text content.

[0035] It should be noted that the network model is a model for generating video text, and there is no restriction on the type of model. In this embodiment, the network model is a video text generation model constructed based on an encoder and a decoder.

[0036] This embodiment segments a long video into multiple video clips at different steps based on video content. The video data includes entity labels. The video vector and entity vector of the entity label are extracted from each video clip, and the video vector and the corresponding entity vector are fused to obtain a visual embedding vector. A text prediction representation is generated based on the visual embedding vector. The cosine similarity between the visual embedding vectors and text prediction representations of the video clips at different steps is calculated to construct a similarity matrix. This similarity matrix can reflect the correlation between the video clips. Based on a preset objective function and constraints, an optimal transmission matrix is ​​determined from the similarity matrix. Optimal transmission is a method that minimizes transmission costs based on the difference between two probability distributions to establish an effective correspondence between different distributions, thereby achieving more accurate distribution matching. Optimal transmission achieves a dynamic, many-to-many allocation of video clips and text prediction representations. Furthermore, the transmission distance calculated by the optimal transmission is used as a weight to construct a loss function to alleviate the problem of over-penalization of correlated negative samples in traditional contrastive learning. The network model is iteratively trained based on the target loss function, and the target video is input into the trained model to obtain the video text content. This embodiment generates text prediction representations through video vectors and entity vectors, calculates the similarity matrix between the visual embedding vectors of the video vectors and entity vectors and the text prediction representation, uses the optimal transmission algorithm to establish a flexible match between the video clips and the text prediction representation, uses the transmission distance calculated by the optimal transmission as a weight, constructs a loss function, alleviates the transition penalty problem of traditional contrastive learning for related negative samples, and can better capture the semantic relationship between the video clips and the text prediction representation, thereby improving the accuracy and coherence of text generation.

[0037] In some embodiments of the present invention, Figure 2 As shown, Figure 2 The present invention provides Figure 1 A flow chart of an embodiment of step S101 in FIG. 1 includes: S201, extracting video vectors and entity vectors of each video clip in the video data; S202, fusing the video vector and the entity vector to obtain a visual embedding vector; S203, decoding the visual embedding vector to obtain text prediction information; S204: Perform average pooling on the text prediction information to obtain a text prediction representation.

[0038] It should be noted that this embodiment implements sequence conversion through an encoder-decoder structure, where the encoder encodes the input sequence element by element into a fixed-length context vector containing key semantic information, and the decoder uses this vector to generate elements in the output sequence one by one, captures timing information through a recurrent neural network, achieves accurate sequence-to-sequence conversion, effectively processes the deep semantics of sequence data and its conversion requirements, and realizes feature extraction and text generation of data video through a visual encoder and a text decoder.

[0039] Specifically, video data containing video, text, and object labels is divided into multiple video segments based on step content. A visual encoder is used to encode the video and object labels into video vectors and object vectors. The resulting visual embedding vector is used to represent the video features containing entity information within the video segment. The text decoder decodes the visual embedding vector to obtain text prediction information. To unify the text prediction information with the visual embedding vector for subsequent similarity calculation, the text prediction information is converted into a text vector through average pooling, i.e., the text prediction representation.

[0040] In some embodiments of the present invention, Figure 3 As shown, Figure 3 The present invention provides Figure 2 The flowchart of an embodiment of step S201 in FIG. 1 includes: S301, adding entity tags to video data according to video content; Specifically, the entity tag is the physical content in a certain video content. For example, in a video tutorial of scrambled eggs with tomatoes, when tomatoes appear in the video, the physical tag added to the video at that moment is tomato; when eggs appear, the physical tag added to the video is egg. S302, dividing the video data with entity tags into multiple video segments according to video content; Specifically, the video is divided into multiple video segments according to the operation steps in the video content. For example, a video tutorial of scrambled eggs with tomatoes can be divided into segments such as accurate ingredients, scrambled eggs, scrambled tomatoes, mixed stir-fry, and plating.

[0041] S303, encoding each frame of the video clip to obtain a frame video vector, and splicing the frame video vectors to obtain a video vector; Specifically, this embodiment encodes each video clip frame by frame through a visual encoder, considers its spatial and temporal information while encoding each frame of video, combines optical flow features and appearance features, encodes the video more comprehensively, and then splices the encoded vectors to form a video vector.

[0042] S304: Encode each entity tag corresponding to the video clip to obtain a single entity vector, and concatenate multiple single entity vectors to obtain an entity vector.

[0043] Specifically, this embodiment uses pre-trained GloVe embedding to encode entity labels to obtain entity vectors of single entities. Similarly, the single entity vectors are concatenated to obtain entity vectors.

[0044] In some embodiments of the present invention, Figure 4 As shown, Figure 4 The present invention provides Figure 2 A flow chart of an embodiment of step S203 in FIG. 1 includes: S401, decoding the visual embedding vector to obtain a word to be selected; S402, determining the hidden state of the words to be selected, and calculating the probability distribution according to the hidden state; S403: Determine text prediction information according to the probability representation.

[0045] Specifically, the visual embedding vector is decoded to generate the word corresponding to the visual embedding vector. This process is achieved by the following formula: , in, Indicates the k The hidden state of a word is the basis for generating the next word. is the video vector in the visual embedding vector, is the object vector in the visual embedding vector, Generate candidate words for the previous one; Through the softmax function, the hidden state is converted into a probability distribution, which represents the probability of generating each possible time. The calculation formula of the probability distribution is: , in, is the probability distribution, represents the hidden state of the k-th word, and It is a learnable parameter matrix that is optimized through training to make the generated text more accurate and smooth.

[0046] In some embodiments of the present invention, Figure 5 As shown, Figure 5 The present invention provides Figure 1 A flow chart of an embodiment of step S103 in FIG. 1 includes: S501, performing alignment transformation on the optimal transmission matrix to obtain a soft alignment matrix; Specifically, the conversion formula of the soft alignment matrix is: , in, is the soft alignment matrix, is the optimal transmission matrix, is the identity matrix, The weighting parameter is used to balance the weights of the identity matrix and the optimal transfer matrix. The traditional identity matrix is ​​replaced with the transformed soft alignment probability matrix, and the target is gradually realigned using the Sinkhorn algorithm. This soft alignment allows the model to more flexibly adjust the attraction and repulsion between the clip and the title, improving robustness to noise.

[0047] S502, obtaining a contrastive learning loss function based on the contrastive loss between the visual embedding vector and the text prediction representation in the soft alignment matrix; Specifically, the formula of the contrastive learning loss function is: , in, To compare the learning loss function, is the soft alignment matrix i Rank k The value of the column, and for matrix, is the similarity matrix i Rank j The value of the column, is the similarity matrix i Rank k The value of the column, is the similarity matrix k Rank j The value of the column, and for matrix, τ is the temperature parameter that controls the softness of the softmax function.

[0048] S503, constructing a text loss function based on the text prediction information and the real text; Specifically, the text decoder predicts the text information of the video based on the visual embedding vector to obtain the predicted text, calculates the loss function between the predicted text and the real text, that is, the text data of the video, to obtain the text loss function. There is no restriction on the text loss function here. In this embodiment, the cross entropy loss function is used.

[0049] S504: Weight the text loss function and the contrastive learning loss function to obtain a target loss function.

[0050] In order to better implement the video text generation method in the embodiment of the present invention, based on the video text generation method, correspondingly, Figure 6As shown, the embodiment of the present invention further provides a video text generation device 600, comprising: A predictive text module 601 is configured to generate a text prediction representation based on a visual embedding vector of each video segment in the video data, wherein the video data includes entity labels, and the visual embedding vector includes a video vector and an entity vector of the entity label; A similarity calculation module 602 is used to construct a similarity matrix based on the cosine similarity of the visual embedding vector and the text prediction representation; A loss function construction module 603 is configured to obtain an optimal transfer matrix based on the similarity matrix and a preset objective function, construct a contrastive learning loss based on the optimal transfer matrix, and construct a target loss function based on the contrastive learning loss and the loss between the true text and the text prediction representation; The text generation module 604 is used to train the network model according to the target loss function, input the target video into the trained model, and generate video text content.

[0051] like Figure 7 As shown, the present invention also provides a video text generation device 700. The video text generation device 700 includes a processor 701, a memory 702 and a display 703. Figure 7 Only some components of the video text generating apparatus 700 are shown, but it should be understood that it is not required to implement all of the shown components, and more or fewer components may be implemented instead.

[0052] In some embodiments, the processor 701 may be a central processing unit (CPU), a microprocessor, or other data processing chip, configured to execute program codes or process data stored in the memory 702 , such as the video text generation method of the present invention.

[0053] In some embodiments of the present invention, processor 701 may be a single server or a server group. The server group may be centralized or distributed. In some embodiments, processor 701 may be local or remote. In some embodiments, processor 701 may be implemented on a cloud platform. In one embodiment, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, multiple clouds, or any combination thereof.

[0054] In some embodiments, the memory 702 may be an internal storage unit of the video text generating device 700, such as a hard disk or memory of the video text generating device 700. In other embodiments, the memory 702 may also be an external storage device of the video text generating device 700, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the video text generating device 700.

[0055] Furthermore, the memory 702 may include both an internal storage unit of the video text generation device 700 and an external storage device. The memory 702 is used to store application software installed in the video text generation device 700 and various data.

[0056] In some embodiments, the display 703 can be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 703 is used to display information about the video text generation device 700 and to display a visual user interface. Components 701-703 of the video text generation device 700 communicate with each other via a system bus.

[0057] In some embodiments of the present invention, when the processor 701 executes the video text generation program in the memory 702, the following steps may be implemented: generating a text prediction representation based on a visual embedding vector of each video clip in the video data, where the video data includes entity labels and the visual embedding vector includes a video vector and an entity vector of the entity label; Construct a similarity matrix based on the cosine similarity between the visual embedding vector and the text prediction representation; The optimal transfer matrix is ​​obtained based on the similarity matrix and the preset objective function, and the contrastive learning loss is constructed based on the optimal transfer matrix. The objective loss function is constructed based on the contrastive learning loss and the loss between the true text and the text prediction representation; The network model is trained according to the target loss function, and the target video is input into the trained model to generate video text content.

[0058] Furthermore, the embodiment of the present invention does not specifically limit the type of the video text generation device 700 mentioned. The video text generation device 700 may be a portable video text generation device such as a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or the like. Exemplary embodiments of portable video text generation devices include, but are not limited to, portable video text generation devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable video text generation device may also be other portable video text generation devices. It should also be understood that in other embodiments of the present invention, the video text generation device 700 may not be a portable video text generation device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0059] Accordingly, an embodiment of the present invention also provides a computer-readable storage medium, which is used to store computer-readable programs or instructions. When the program or instructions are executed by a processor, the steps or functions in the video text generation method provided in the above-mentioned method embodiments can be implemented.

[0060] Those skilled in the art will appreciate that all or part of the process flow of the above-described method embodiment can be implemented by instructing related hardware (such as a processor, controller, etc.) through a computer program, and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a magnetic disk, an optical disk, a read-only memory, or a random access memory.

[0061] The above is a detailed introduction to the video text generation method, device, equipment and storage medium provided by the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A video text generation method, characterized in that: include: generating a text prediction representation based on a visual embedding vector of each video clip in the video data, wherein the video data includes entity labels, and the visual embedding vector includes a video vector and an entity vector of the entity label; Constructing a similarity matrix based on the cosine similarity between the visual embedding vector and the text prediction representation; Obtaining an optimal transfer matrix based on the similarity matrix and a preset objective function, constructing a contrastive learning loss based on the optimal transfer matrix, and constructing an objective loss function based on the contrastive learning loss and the loss between the true text and the text prediction representation; The network model is trained according to the target loss function, and the target video is input into the trained model to generate video text content.

2. A video text generation method according to claim 1, characterized in that: Generate text prediction representation based on the visual embedding vector of each video clip in the video data, including: Extracting video vectors and entity vectors for each video clip in the video data; Fusing the video vector and the entity vector to obtain a visual embedding vector; Decoding the visual embedding vector to obtain text prediction information; Average pooling is performed on the text prediction information to obtain a text prediction representation.

3. A video text generation method according to claim 2, characterized in that: Determine the video vector and entity vector of each video clip in the video data, including: Add entity tags to video data based on video content; Divide the video data with entity tags into multiple video segments according to the video content; Encoding each frame of the video clip to obtain a frame video vector, and splicing the frame video vectors to obtain a video vector; Each entity label corresponding to the video clip is encoded to obtain a single entity vector, and the single entity vectors are concatenated to obtain an entity vector.

4. A video text generation method according to claim 3, characterized in that: Decoding the visual embedding vector to obtain text prediction information includes: Decoding the visual embedding vector to obtain multiple candidate words; Calculating the probability distribution of the words to be selected; Text prediction information is determined based on the probability representation.

5. The video text generation method according to claim 1, characterized in that: The calculation formula of the transmission matrix is: The preset objective function is: ， in, is the transmission matrix, and The scaling vectors are and The diagonal matrix of S is the similarity matrix, δ is the hyperparameter that controls the smoothness of the transfer distribution, is the entropy term, is the transmission matrix and similarity matrix S The inner product of .

6. A video text generation method according to claim 4, characterized in that: Constructing a contrastive learning loss according to the optimal transfer matrix, and constructing a target loss function according to the contrastive learning loss and the loss between the true text and the text prediction representation, including: Performing alignment transformation on the optimal transmission matrix to obtain a soft alignment matrix; A contrastive learning loss function is obtained based on the contrastive loss between the visual embedding vector and the text prediction representation in the soft alignment matrix; Constructing a text loss function based on the text prediction information and the real text; The text loss function and the contrastive learning loss function are weighted to obtain the target loss function.

7. A video text generation method according to claim 6, characterized in that: The formula of the contrastive learning loss function is: in, To compare the learning loss function, is the soft alignment matrix i Rank k The value of the column, and for matrix, is the similarity matrix i Rank j The value of the column, is the similarity matrix i Rank k The value of the column, is the similarity matrix k Rank j The value of the column, and for matrix, τ is the temperature parameter that controls the softness of the softmax function.

8. A video text generation device, characterized in that: include: a predictive text module for generating a text prediction representation based on a visual embedding vector of each video segment in the video data, wherein the video data includes entity labels, and the visual embedding vector includes a video vector and an entity vector of the entity label; A similarity calculation module, configured to construct a similarity matrix based on the cosine similarity between the visual embedding vector and the text prediction representation; A loss function construction module is used to obtain an optimal transfer matrix based on the similarity matrix and a preset objective function, construct a contrastive learning loss based on the optimal transfer matrix, and construct a target loss function based on the contrastive learning loss and the loss between the true text and the text prediction representation; The text generation module is used to train the network model according to the target loss function, input the target video into the trained model, and generate video text content.

9. A video text generation device, characterized in that: comprising a memory and a processor, wherein, The memory is used to store programs; The processor is coupled to the memory and is configured to execute the program stored in the memory to implement the steps in the video text generation method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores computer program instructions, which, when executed by a computer, enable the computer to execute the video text generation method according to any one of claims 1 to 7.

Images