Intelligent conversion method and device based on video and text, electronic equipment and medium

Abstract

The application relates to the field of artificial intelligence, and discloses an intelligent conversion method based on videos and texts, which comprises the following steps: acquiring a training video and corresponding video text of the training video, and extracting training pictures in the training video; using an encoder in a pre-constructed text-video conversion model to perform feature vector coding, vector masking and vector splicing on the training pictures and the video text to obtain picture-text splicing vectors; using a semantic analysis network in the pre-constructed text-video conversion model to identify predicted pictures and predicted texts of the picture-text splicing vectors, and then decoding to obtain predicted videos and predicted video texts; calculating model loss of the pre-constructed text-video conversion model according to the predicted videos and the predicted video texts, and the training videos and the video texts, so as to generate a trained text-video conversion model, realize scene conversion on to-be-converted scene data, and obtain a scene conversion result. The application can improve the scene conversion efficiency between videos and texts.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and in particular to a method, apparatus, electronic device, and medium for intelligent conversion between video and text. Background Technology

[0002] With the development of information technology, there are more and more ways to record information, such as video recording and text recording. In actual business scenarios, it is often necessary to directly convert information recorded in different ways to facilitate users' direct reading. At present, the scene conversion between video and text is often one-way, that is, it is usually based on video to convert to text or based on text to video. Therefore, there is an urgent need for a solution to realize the mutual conversion between video and text and ensure the efficiency of scene conversion between video and text. Summary of the Invention

[0003] This invention provides a method, device, electronic device, and medium for intelligent conversion between video and text, the main purpose of which is to realize the mutual conversion between video and text and improve the scene conversion efficiency between video and text.

[0004] To achieve the above objectives, the present invention provides an intelligent conversion method based on video and text, comprising:

[0005] Obtain the training video and its corresponding video text, and extract the training images from the training video;

[0006] The training images are encoded using the image encoder in the pre-built text-to-video conversion model to obtain image feature vectors, and the video text is encoded using the text encoder in the pre-built text-to-video conversion model to obtain text feature vectors.

[0007] The image feature vector and the text feature vector are vector-masked using the masking layer in the pre-built text-video conversion model, and the image feature vector and the text feature vector after vector masking are concatenated to obtain the image-text concatenated vector.

[0008] The semantic analysis network in the pre-built text-video conversion model is used to identify the predicted image and predicted text of the image-text splicing vector;

[0009] The predicted image is decoded using the image decoder in the pre-built text-to-video conversion model to obtain the predicted video, and the predicted text is decoded using the text decoder in the pre-built text-to-video conversion model to obtain the predicted video text.

[0010] Calculate the model loss in the pre-built text-to-video conversion model based on the predicted video and the predicted video text, and the training video and the video text;

[0011] When the model loss is greater than the preset loss, the model parameters of the pre-built text-to-video conversion model are updated, and the step of using the image encoder in the pre-built text-to-video conversion model to encode the feature vector of the training image is returned.

[0012] When the model loss is not greater than the preset loss, a trained text-to-video conversion model is obtained, and the trained text-to-video conversion model is used to perform scene conversion on the scene data to be converted, thereby obtaining the scene conversion result.

[0013] Optionally, the step of using the image encoder in the pre-built text-to-video conversion model to encode the training images into feature vectors to obtain image feature vectors includes:

[0014] The training image is encoded using a positional encoding layer in the image encoder to obtain a positional encoding vector.

[0015] The weight values ​​between vectors in the position encoding vector are calculated using the self-attention module in the image encoder. Based on the weight values, the weight information of the position encoding vector is updated to obtain the weight encoding vector.

[0016] The image feature vector is obtained by extracting the feature vector from the weighted encoding vector using the feedforward neural network in the image encoder.

[0017] Optionally, the step of concatenating the image feature vector and the text feature vector after vector masking to obtain the image-text concatenated vector includes:

[0018] Obtain the sequence correspondence between the image feature vector and the text feature vector after obtaining the vector mask;

[0019] Based on the sequence correspondence, the image feature vector and the text feature vector after the vector mask are concatenated to obtain the image-text concatenated vector.

[0020] Optionally, the step of using the semantic analysis network in the pre-built text-video conversion model to identify the predicted image and predicted text of the image-text concatenation vector includes:

[0021] The classification layer in the semantic analysis network is used to predict the category of the image-text concatenation vector to obtain the predicted category.

[0022] Based on the predicted category, the image-text concatenation vector is embedded using the embedding layer in the semantic analysis network to obtain the vector dimension matrix of the image-text concatenation vector;

[0023] Based on the vector dimension matrix, the predicted image and predicted text of the image-text concatenation vector are detected using the fully connected layer in the semantic analysis network.

[0024] Optionally, the step of decoding the predicted image using the image decoder in the pre-built text-to-video conversion model to obtain the predicted video includes:

[0025] The information sequence of the predicted image is calculated using the attention module in the slice decoder to obtain the image information sequence;

[0026] The image information sequence is output using the fully connected neural network in the decoder, and then the images are merged to obtain the predicted video.

[0027] Optionally, calculating the model loss in the pre-built text-to-video conversion model based on the predicted video and the predicted video text, and the training video and the video text, includes:

[0028] Calculate the video loss in the pre-built text-video conversion model based on the predicted video and the training video;

[0029] Calculate the text loss in the pre-built text-to-video conversion model based on the predicted video text and the video text;

[0030] Calculate the model loss in the pre-built text-video conversion model based on the video loss and the text loss.

[0031] Optionally, calculating the video loss in the pre-built text-to-video conversion model based on the predicted video and the training video includes:

[0032] The video loss in the pre-built text-to-video conversion model is calculated using the following formula:

[0033] L1 = m g logm p +(1-m g log(1-m) p )

[0034] Where L1 represents the video loss, m g Indicates the predicted video, m p This refers to training videos.

[0035] To address the aforementioned problems, the present invention also provides a video-to-text intelligent conversion device, the device comprising:

[0036] The training data acquisition module is used to acquire training videos and their corresponding video text, and extract training images from the training videos;

[0037] The feature vector encoding module is used to encode the training images using the image encoder in the pre-built text-to-video conversion model to obtain image feature vectors, and to encode the video text using the text encoder in the pre-built text-to-video conversion model to obtain text feature vectors.

[0038] The feature vector concatenation module is used to perform vector masking on the image feature vector and the text feature vector respectively using the masking layer in the pre-built text-video conversion model, and then concatenate the vector-masked image feature vector and the text feature vector to obtain the image-text concatenated vector.

[0039] The vector semantic analysis module is used to identify the predicted image and predicted text of the image-text splicing vector using the semantic analysis network in the pre-built text-video conversion model;

[0040] The data decoding module is used to decode the predicted image using the image decoder in the pre-built text-video conversion model to obtain the predicted video, and to decode the predicted text using the text decoder in the pre-built text-video conversion model to obtain the predicted video text.

[0041] The model loss calculation module is used to calculate the model loss in the pre-built text-to-video conversion model based on the predicted video and the predicted video text, as well as the training video and the video text.

[0042] The model parameter adjustment module is used to update the model parameters of the pre-built text-to-video conversion model when the model loss is greater than the preset loss, and return the step of using the image encoder in the pre-built text-to-video conversion model to encode the feature vector of the training image;

[0043] The data scene conversion module is used to obtain a trained text-video conversion model when the model loss is not greater than the preset loss, and to use the trained text-video conversion model to perform scene conversion on the scene data to be converted, so as to obtain the scene conversion result.

[0044] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0045] At least one processor; and,

[0046] A memory communicatively connected to the at least one processor; wherein,

[0047] The memory stores a computer program that can be executed by the at least one processor to implement the above-described intelligent conversion method based on video and text.

[0048] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the aforementioned intelligent video-to-text conversion method.

[0049] As can be seen, this embodiment of the invention, by acquiring training videos and their corresponding video text, extracting training images from the training videos, can split the training videos into images of multiple video frames, ensuring the training prerequisites for subsequent models. Using the encoder in the pre-built text-video conversion model, feature vector encoding, vector masking, and vector concatenation are performed on the training images and video text to obtain image-text concatenated vectors. Feature information of the training images and video text can be extracted separately, used to compress the training images and video text into latent space representations. Furthermore, by masking some information of the image and text feature vectors, a mapping relationship between image and text feature vectors can be constructed, ensuring the prerequisite for corresponding relationship recognition during subsequent video-to-text conversion. Secondly, this embodiment of the invention utilizes pre-built text-video... The semantic analysis network in the conversion model identifies and decodes the predicted image and text from the image-text concatenated vectors to obtain the predicted video and predicted video text. It can detect the mask information corresponding to the image-text concatenated vectors, ensuring the prerequisite for the subsequent generation of predicted video and predicted video text. It can also verify the model capability of the pre-built text-video conversion model, ensuring the conversion capability during subsequent text-to-video conversion using the pre-built text-video conversion model. Furthermore, this embodiment of the invention calculates the model loss of the pre-built text-video conversion model based on the predicted video and predicted video text, as well as the training video and video text, to generate a trained text-video conversion model, ensuring the model's conversion capability and enabling scene interchange of the data to be converted, thus improving the scene conversion efficiency of the data to be converted. Therefore, the intelligent conversion method, device, electronic device, and medium based on video and text proposed in this embodiment of the invention can realize the mutual conversion between video and text, improving the scene conversion efficiency between video and text. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating an embodiment of the intelligent video-to-text conversion method provided by the present invention.

[0051] Figure 2 A schematic diagram of a video-to-text intelligent conversion device provided in an embodiment of the present invention;

[0052] Figure 3 A schematic diagram of the internal structure of an electronic device that implements a video-to-text intelligent conversion method according to an embodiment of the present invention;

[0053] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0054] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0055] This invention provides a video-to-text intelligent conversion method. The executing entity of this video-to-text intelligent conversion method includes, but is not limited to, at least one of the following: a server, a terminal, or other electronic devices configured to execute the method provided in this invention. In other words, the video-to-text intelligent conversion method can be executed by software or hardware installed on a terminal device or a server device, where the software may be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cluster of cloud servers. The server can be an independent server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0056] Reference Figure 1 The diagram shown is a flowchart illustrating an intelligent video-to-text conversion method according to an embodiment of the present invention. In this embodiment, the intelligent video-to-text conversion method includes:

[0057] S1. Obtain the training video and its corresponding video text, and extract the training images from the training video.

[0058] In this embodiment of the invention, the training video refers to the model training data used for subsequent text-to-video conversion, and the video text refers to the video text information corresponding to the training video, such as video subtitles. Optionally, the training video and the video text can be obtained from the Internet using web crawling technology, such as Node.js technology.

[0059] Furthermore, in this embodiment of the invention, training images are extracted from the training video to split the training video into images of multiple video frames, ensuring the prerequisite for subsequent model training. Optionally, the training images from the training video can be extracted in a frame-by-frame manner.

[0060] S2. Use the image encoder in the pre-built text-to-video conversion model to encode the feature vectors of the training images to obtain image feature vectors, and use the text encoder in the pre-built text-to-video conversion model to encode the feature vectors of the video text to obtain text feature vectors.

[0061] In this embodiment of the invention, the pre-built text-to-video conversion model includes an encoder, a masking layer, a semantic analysis network, and a decoder. The encoder includes an image encoder and a text encoder, used to extract feature information from the training images and training text. The masking layer is used to mask part of the feature vectors extracted by the encoder, ensuring subsequent semantic analysis. The semantic analysis network is used to detect the feature information masked by the masking layer. The decoder includes an image decoder and a text decoder, used to reconstruct the content of the feature information identified by the semantic analysis network, achieving information prediction of the training video and training text. Further, the image encoder and the image decoder can be constructed using a transform mechanism, and the text encoder, the text decoder, and the semantic analysis network can be constructed using a BERT network.

[0062] Furthermore, in this embodiment of the invention, feature vector encoding of the training images is performed using an image encoder in a pre-built text-to-video conversion model, and feature vector encoding of the video text is performed using a text encoder in the pre-built text-to-video conversion model, so as to extract feature information of the training images and the video text respectively, which is used to compress the training images and the video text into a latent space representation, ensuring the accuracy of subsequent data processing.

[0063] As an embodiment of the present invention, the step of using the image encoder in the pre-built text-video conversion model to encode the training image into a feature vector to obtain an image feature vector includes: using the position encoding layer in the image encoder to encode the training image into a position encoding vector to obtain a position encoding vector; using the self-attention module in the image encoder to calculate the weight values ​​between the vectors in the position encoding vector; updating the weight information of the position encoding vector according to the weight values ​​to obtain a weight encoding vector; and using the feedforward neural network in the image encoder to extract the feature vectors in the weight encoding vector to obtain an image feature vector.

[0064] The location encoding layer is used to segment the training image into multiple image blocks and determine the location sequence information of the training image to compensate for the lack of location information in the feature encoding process of subsequent training images. The self-attention module is used to identify the correlation between each location encoding vector and other vectors in the location encoding vector to update the weight information of the corresponding location encoding vector, so that the location encoding vector contains contextual feature information. The feedforward neural network is used to activate the location encoding vector to identify the feature vector in the location encoding vector.

[0065] Furthermore, in an optional embodiment of the present invention, the position vector encoding of the training image can be implemented by the encoding function of the position encoding layer, such as the PE function. The weight values ​​between vectors in the position encoding vector can be calculated by the dimension vector in the self-attention module. The dimension vector includes: Query, key, and value. The Query can be understood as the position encoding vector to which the weight value is to be calculated, the key can be understood as the position encoding vector, and the value can be understood as the context vector of the position encoding vector. The feature vector in the weight encoding vector can be detected by the activation function in the feedforward neural network, such as the softmax function.

[0066] It should be noted that, in this embodiment of the invention, the feature vector encoding of the video text is based on the same principle as the feature vector encoding of the training images, and will not be elaborated further here.

[0067] S3. Using the masking layer in the pre-built text-video conversion model, the image feature vector and the text feature vector are respectively vector masked, and the image feature vector and the text feature vector after vector masking are vector concatenated to obtain the image-text concatenated vector.

[0068] This invention utilizes a masking layer in the pre-built text-to-video conversion model to perform vector masking on the image feature vector and the text feature vector, respectively, to conceal some information in the image feature vector and the text feature vector, thus ensuring the premise for subsequent information prediction. Optionally, the vector masking of the image feature vector and the text feature vector is implemented in the form of random masking, that is, randomly selecting the vector position sequence in the image feature vector and the text feature vector for masking.

[0069] Furthermore, in this embodiment of the invention, the image feature vector and the text feature vector after vector masking are concatenated to construct a mapping relationship between the image feature vector and the text feature vector, ensuring the recognition of the correspondence relationship when performing subsequent video-to-text conversion.

[0070] As an embodiment of the present invention, the step of concatenating the image feature vector and the text feature vector after vector masking to obtain the image-text concatenated vector includes: obtaining the sequence correspondence between the image feature vector and the text feature vector after vector masking, and concatenating the image feature vector and the text feature vector after vector masking according to the sequence correspondence to obtain the image-text concatenated vector.

[0071] The sequence correspondence refers to the correspondence between the training image corresponding to the image feature vector and the text information corresponding to the text feature vector, which can be obtained by querying the image-text mapping table of the image feature vector and the text feature vector.

[0072] S4. Use the semantic analysis network in the pre-built text-video conversion model to identify the predicted image and predicted text of the image-text splicing vector.

[0073] This invention utilizes the semantic analysis network in the pre-built text-video conversion model to identify the predicted image and predicted text of the image-text concatenation vector, thereby detecting the mask information corresponding to the image-text concatenation vector and ensuring the prerequisite for the subsequent generation of predicted video and predicted video text.

[0074] As an embodiment of the present invention, the step of using the semantic analysis network in the pre-built text-video conversion model to identify the predicted image and predicted text of the image-text splicing vector includes: using the classification layer in the semantic analysis network to classify the image-text splicing vector into a predicted category, obtaining a predicted category; based on the predicted category, using the embedding layer in the semantic analysis network to embed a dimension matrix into the image-text splicing vector, obtaining a vector dimension matrix of the image-text splicing vector; and based on the vector dimension matrix, using the fully connected layer in the semantic analysis network to detect the predicted image and predicted text of the image-text splicing vector.

[0075] The classification layer is used to determine the data category to be predicted by the image-text concatenation vector, which includes text category and image category. The embedding layer is used to convert the image-text concatenation vector into a vocabulary dimension, which is a prerequisite for subsequent prediction data detection. The predicted image and predicted text of the image-text concatenation vector are detected by the activation function in the fully connected layer.

[0076] S5. The predicted image is decoded using the image decoder in the pre-built text-to-video conversion model to obtain the predicted video, and the predicted text is decoded using the text decoder in the pre-built text-to-video conversion model to obtain the predicted video text.

[0077] This invention utilizes the image decoder in the pre-built text-to-video conversion model to decode and merge the predicted images, and utilizes the text decoder in the pre-built text-to-video conversion model to decode the predicted text, in order to obtain the prediction results corresponding to the training video and the video text. This is to verify the model capability of the pre-built text-to-video conversion model and ensure the conversion capability in subsequent text-to-video conversion processes using the pre-built text-to-video conversion model.

[0078] As an embodiment of the present invention, the step of decoding the predicted image using the image decoder in the pre-built text-video conversion model to obtain the predicted video includes: calculating the information sequence of the predicted image using the attention module in the image decoder to obtain the image information sequence; and merging the images after outputting the image information sequence using the fully connected neural network in the decoder to obtain the predicted video.

[0079] The information sequence is used to determine the length sequence and frame sequence of the predicted image, and the output of the image information sequence is implemented through the activation function of the fully connected neural network.

[0080] Furthermore, in this invention, the decoding principle of the predicted text is the same as that of the predicted image, and will not be elaborated further here.

[0081] S6. Calculate the model loss in the pre-built text-to-video conversion model based on the predicted video and the predicted video text, as well as the training video and the video text.

[0082] As an embodiment of the present invention, the step of calculating the model loss in the pre-built text-to-video conversion model based on the predicted video and the predicted video text, and the training video and the video text, includes: calculating the video loss in the pre-built text-to-video conversion model based on the predicted video and the training video, calculating the text loss in the pre-built text-to-video conversion model based on the predicted video text and the video text, and calculating the model loss in the pre-built text-to-video conversion model based on the video loss and the text loss.

[0083] Furthermore, in an optional embodiment of the present invention, the video loss in the pre-built text-to-video conversion model is calculated using the following formula:

[0084] L1 = m g logm p +(1-m g log(1-m) p )

[0085] Where L1 represents the video loss, m g Indicates the predicted video, m p This refers to training videos.

[0086] Furthermore, in an optional embodiment of the present invention, the text loss in the pre-built text-to-video conversion model is calculated using the following formula:

[0087]

[0088] Where L2 represents the text loss, k represents the number of predicted video texts, and y i Let y represent the i-th predicted video text. i ′ represents the i-th video text.

[0089] Furthermore, in an optional embodiment of the present invention, the model loss in the pre-built text-to-video conversion model is calculated using the following formula:

[0090] L = L1 + L2

[0091] Where L represents the model loss, L1 represents the video loss, and L2 represents the text loss.

[0092] S7. When the model loss is greater than the preset loss, update the model parameters of the pre-built text-to-video conversion model, and return to the step of using the image encoder in the pre-built text-to-video conversion model to encode the feature vector of the training image.

[0093] It should be understood that when the model loss is greater than the preset loss, it indicates that the conversion capability of the pre-built text-to-video conversion model is not good enough. Therefore, this embodiment of the invention updates the model parameters of the pre-built text-to-video conversion model and returns to the step of using the image encoder in the pre-built text-to-video conversion model to encode the feature vectors of the training images, in order to ensure the conversion capability of the pre-built text-to-video conversion model. Optionally, the preset loss can be set to 0.1, or it can be set according to the actual business scenario. The model parameters of the pre-built text-to-video conversion model can be updated using a gradient descent algorithm, such as stochastic gradient descent.

[0094] S8. When the model loss is not greater than the preset loss, a trained text-video conversion model is obtained, and the trained text-video conversion model is used to perform scene conversion on the scene data to be converted, and the scene conversion result is obtained.

[0095] It should be understood that when the model loss is not greater than the preset loss, it indicates that the pre-built text-to-video conversion model has good conversion capabilities. Therefore, in this embodiment of the invention, a trained text-to-video conversion model is directly generated to perform scene conversion on the scene data to be converted, thereby obtaining the scene conversion result. The scene data to be converted is generated based on different conversion scenarios, including video data and text data. For example, in a text-to-video scenario, the scene data to be converted is text data; in a video-to-text scenario, the scene data to be converted is video data.

[0096] As can be seen, this embodiment of the invention, by acquiring training videos and their corresponding video text, extracting training images from the training videos, can split the training videos into images of multiple video frames, ensuring the training prerequisites for subsequent models. Using the encoder in the pre-built text-video conversion model, feature vector encoding, vector masking, and vector concatenation are performed on the training images and video text to obtain image-text concatenated vectors. Feature information of the training images and video text can be extracted separately, used to compress the training images and video text into latent space representations. Furthermore, by masking some information of the image and text feature vectors, a mapping relationship between image and text feature vectors can be constructed, ensuring the prerequisite for corresponding relationship recognition during subsequent video-to-text conversion. Secondly, this embodiment of the invention utilizes pre-built text-video... The semantic analysis network in the conversion model identifies and decodes the predicted image and text from the image-text concatenated vectors to obtain the predicted video and predicted video text. It can detect the mask information corresponding to the image-text concatenated vectors, ensuring the prerequisite for the subsequent generation of predicted video and predicted video text. It can also verify the model capability of the pre-built text-video conversion model, ensuring the conversion capability during subsequent text-to-video conversion using the pre-built text-video conversion model. Furthermore, this embodiment of the invention calculates the model loss of the pre-built text-video conversion model based on the predicted video and predicted video text, as well as the training video and video text, to generate a trained text-video conversion model, ensuring the model's conversion capability and enabling scene interchange of the data to be converted, thus improving the scene conversion efficiency of the data to be converted. Therefore, the intelligent conversion method based on video and text proposed in this embodiment of the invention can realize the mutual conversion between video and text, improving the scene conversion efficiency between video and text.

[0097] like Figure 2 The diagram shown is a functional block diagram of the intelligent video-to-text conversion device of the present invention.

[0098] The intelligent video-to-text conversion device 100 described in this invention can be installed in an electronic device. Depending on the functions implemented, the intelligent video-to-text conversion device may include a training data acquisition module 101, a feature vector encoding module 102, a feature vector concatenation module 103, a vector semantic analysis module 104, a data decoding module 105, a model loss calculation module 106, a model parameter adjustment module 107, and a data scene conversion module 108. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, stored in the memory of the electronic device.

[0099] In this embodiment, the functions of each module / unit are as follows:

[0100] The training data acquisition module 101 is used to acquire training videos and their corresponding video text, and extract training images from the training videos.

[0101] The feature vector encoding module 102 is used to encode the training image using the image encoder in the pre-built text-video conversion model to obtain the image feature vector, and to encode the video text using the text encoder in the pre-built text-video conversion model to obtain the text feature vector.

[0102] The feature vector concatenation module 103 is used to perform vector masking on the image feature vector and the text feature vector respectively using the masking layer in the pre-built text-video conversion model, and then concatenate the image feature vector and the text feature vector after vector masking to obtain the image-text concatenation vector.

[0103] The vector semantic analysis module 104 is used to identify the predicted image and predicted text of the image-text splicing vector using the semantic analysis network in the pre-built text-video conversion model;

[0104] The data decoding module 105 is used to decode the predicted image using the image decoder in the pre-built text-video conversion model to obtain the predicted video, and to decode the predicted text using the text decoder in the pre-built text-video conversion model to obtain the predicted video text.

[0105] The model loss calculation module 106 is used to calculate the model loss in the pre-built text-to-video conversion model based on the predicted video and the predicted video text, and the training video and the video text.

[0106] The model parameter adjustment module 107 is used to update the model parameters of the pre-built text-to-video conversion model when the model loss is greater than the preset loss, and return to the step of using the image encoder in the pre-built text-to-video conversion model to encode the feature vector of the training image.

[0107] The data scene conversion module 108 is used to obtain a trained text-video conversion model when the model loss is not greater than the preset loss, and to use the trained text-video conversion model to perform scene conversion on the scene data to be converted, so as to obtain the scene conversion result.

[0108] In detail, the modules in the intelligent video-to-text conversion device 100 described in this embodiment of the invention employ the same methods as described above during use. Figure 1 The method uses the same technical means as the intelligent conversion method based on video and text described above, and can produce the same technical effect, so it will not be repeated here.

[0109] like Figure 3 The diagram shown is a structural schematic of an electronic device 1 that implements the intelligent conversion method based on video and text according to the present invention.

[0110] The electronic device 1 may include a processor 10, a memory 11, a communication bus 12, and a communication interface 13. It may also include a computer program stored in the memory 11 and capable of running on the processor 10, such as a video-to-text intelligent conversion program.

[0111] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device 1, connecting various components of the electronic device 1 via various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., executing intelligent conversion programs based on video and text) and calls data stored in the memory 11 to perform various functions and process data of the electronic device 1.

[0112] The memory 11 includes at least one type of readable storage medium, including flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive of the electronic device 1. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 1. Furthermore, the memory 11 can include both internal and external storage units of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as the code of a video-to-text intelligent conversion program, but also to temporarily store data that has been output or will be output.

[0113] The communication bus 12 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0114] The communication interface 13 is used for communication between the aforementioned electronic device 1 and other devices, including a network interface and an employee interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, Bluetooth interface, etc.), typically used to establish a communication connection between the electronic device 1 and other electronic devices 1. The employee interface may be a display, an input unit (such as a keyboard), or, optionally, a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen, etc. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual employee interface.

[0115] Figure 3 Only electronic device 1 with components is shown; those skilled in the art will understand that... Figure 3The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0116] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0117] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in terms of the scope of the patent invention.

[0118] The video-to-text intelligent conversion program stored in the memory 11 of the electronic device 1 is a combination of multiple computer programs. When run in the processor 10, it can achieve the following:

[0119] Obtain the training video and its corresponding video text, and extract the training images from the training video;

[0120] The training images are encoded using the image encoder in the pre-built text-to-video conversion model to obtain image feature vectors, and the video text is encoded using the text encoder in the pre-built text-to-video conversion model to obtain text feature vectors.

[0121] The image feature vector and the text feature vector are vector-masked using the masking layer in the pre-built text-video conversion model, and the image feature vector and the text feature vector after vector masking are concatenated to obtain the image-text concatenated vector.

[0122] The semantic analysis network in the pre-built text-video conversion model is used to identify the predicted image and predicted text of the image-text splicing vector;

[0123] The predicted image is decoded using the image decoder in the pre-built text-to-video conversion model to obtain the predicted video, and the predicted text is decoded using the text decoder in the pre-built text-to-video conversion model to obtain the predicted video text.

[0124] Calculate the model loss in the pre-built text-to-video conversion model based on the predicted video and the predicted video text, and the training video and the video text;

[0125] When the model loss is greater than the preset loss, the model parameters of the pre-built text-to-video conversion model are updated, and the step of using the image encoder in the pre-built text-to-video conversion model to encode the feature vector of the training image is returned.

[0126] When the model loss is not greater than the preset loss, a trained text-to-video conversion model is obtained, and the trained text-to-video conversion model is used to perform scene conversion on the scene data to be converted, thereby obtaining the scene conversion result.

[0127] Specifically, the processor 10's implementation method of the above-mentioned computer program can be found in [reference needed]. Figure 1 The descriptions of the relevant steps in the corresponding embodiments are not repeated here.

[0128] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0129] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device 1, can perform the following:

[0130] Obtain the training video and its corresponding video text, and extract the training images from the training video;

[0131] The training images are encoded using the image encoder in the pre-built text-to-video conversion model to obtain image feature vectors, and the video text is encoded using the text encoder in the pre-built text-to-video conversion model to obtain text feature vectors.

[0132] The image feature vector and the text feature vector are vector-masked using the masking layer in the pre-built text-video conversion model, and the image feature vector and the text feature vector after vector masking are concatenated to obtain the image-text concatenated vector.

[0133] The semantic analysis network in the pre-built text-video conversion model is used to identify the predicted image and predicted text of the image-text splicing vector;

[0134] The predicted image is decoded using the image decoder in the pre-built text-to-video conversion model to obtain the predicted video, and the predicted text is decoded using the text decoder in the pre-built text-to-video conversion model to obtain the predicted video text.

[0135] Calculate the model loss in the pre-built text-to-video conversion model based on the predicted video and the predicted video text, and the training video and the video text;

[0136] When the model loss is greater than the preset loss, the model parameters of the pre-built text-to-video conversion model are updated, and the step of using the image encoder in the pre-built text-to-video conversion model to encode the feature vector of the training image is returned.

[0137] When the model loss is not greater than the preset loss, a trained text-to-video conversion model is obtained, and the trained text-to-video conversion model is used to perform scene conversion on the scene data to be converted, thereby obtaining the scene conversion result.

[0138] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and other division methods may be used in actual implementation.

[0139] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0141] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0142] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0143] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0144] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for intelligent conversion based on video and text, characterized in that, The method comprises: acquiring a training video and its corresponding video text, and extracting a training picture in the training video; using a picture encoder in a pre-constructed text-video conversion model to encode a feature vector of the training picture to obtain a picture feature vector, and using a text encoder in the pre-constructed text-video conversion model to encode a feature vector of the video text to obtain a text feature vector; using a mask layer in the pre-constructed text-video conversion model to respectively perform vector masking on the picture feature vector and the text feature vector, and performing vector splicing on the vector-masked picture feature vector and the text feature vector to obtain a picture-text splicing vector; using a semantic analysis network in the pre-constructed text-video conversion model to identify a predicted picture and a predicted text of the picture-text splicing vector, comprising: using a classification layer in the semantic analysis network to perform predicted category classification on the picture-text splicing vector to obtain a predicted category of data corresponding to the picture-text splicing vector, including a text category and a picture category; according to the predicted category, using an embedding layer in the semantic analysis network to perform dimension matrix embedding on the picture-text splicing vector, converting the picture-text splicing vector to a word dimension to obtain a vector dimension matrix of the picture-text splicing vector; according to the vector dimension matrix, using a full connection layer in the semantic analysis network to detect the predicted picture and the predicted text of the picture-text splicing vector; using a picture decoder in the pre-constructed text-video conversion model to decode the predicted picture to obtain a predicted video, and using a text decoder in the pre-constructed text-video conversion model to decode the predicted text to obtain a predicted video text; according to the predicted video and the predicted video text, and the training video and the video text, calculating a model loss in the pre-constructed text-video conversion model; when the model loss is greater than a preset loss, updating model parameters of the pre-constructed text-video conversion model, and returning to the step of using the picture encoder in the pre-constructed text-video conversion model to encode a feature vector of the training picture; when the model loss is not greater than the preset loss, obtaining a trained text-video conversion model, and using the trained text-video conversion model to perform scene conversion on to-be-converted scene data to obtain a scene conversion result.

2. The intelligent conversion method based on video and text as claimed in claim 1, wherein, The step of using the picture encoder in the pre-constructed text-video conversion model to encode a feature vector of the training picture to obtain a picture feature vector comprises: using a position encoding layer in the picture encoder to encode a position vector of the training picture to obtain a position encoding vector; using a self-attention module in the picture encoder to calculate a weight value between vectors in the position encoding vector, updating weight information of the position encoding vector according to the weight value to obtain a weight encoding vector; using a feedforward neural network in the picture encoder to extract a feature vector in the weight encoding vector to obtain a picture feature vector.

3. The intelligent conversion method based on video and text as claimed in claim 1, wherein, The vector splicing of the vector-masked picture feature vector and the text feature vector comprises: Obtaining a sequence correspondence relationship of the vector-masked picture feature vector and the text feature vector; According to the sequence correspondence relationship, the vector-masked picture feature vector and the text feature vector are spliced to obtain the picture-text splicing vector.

4. The intelligent conversion method based on video and text as claimed in claim 1, wherein, The decoding of the predicted picture by using the picture decoder in the pre-constructed text-video conversion model comprises: Calculating an information sequence of the predicted picture by using an attention module in the slice decoder to obtain a picture information sequence; Performing picture merging on the picture information sequence output by the full-connection neural network in the decoder to obtain a predicted video.

5. The intelligent conversion method based on video and text according to any one of claims 1 to 4, characterized in that, The calculation of the model loss in the pre-constructed text-video conversion model according to the predicted video and the predicted video text, and the training video and the video text comprises: According to the predicted video and the training video, a video loss in the pre-constructed text-video conversion model is calculated; According to the predicted video text and the video text, a text loss in the pre-constructed text-video conversion model is calculated; According to the video loss and the text loss, a model loss in the pre-constructed text-video conversion model is calculated.

6. The intelligent conversion method based on video and text according to claim 5, wherein, The calculation of the video loss in the pre-constructed text-video conversion model according to the predicted video and the training video comprises: The video loss in the pre-constructed text-video conversion model is calculated by using the following formula: wherein L1represents a video loss, represents a predicted video, represents a training video.

7. A device for intelligent conversion based on video and text, characterized by, The device comprises: A training data acquisition module is configured to acquire a training video and a corresponding video text thereof, and extract a training picture in the training video; A feature vector encoding module is configured to encode a picture feature vector of the training picture by using a picture encoder in a pre-constructed text-video conversion model, and encode a text feature vector of the video text by using a text encoder in the pre-constructed text-video conversion model; A feature vector splicing module is configured to perform vector masking on the picture feature vector and the text feature vector by using a mask layer in the pre-constructed text-video conversion model, and perform vector splicing on the vector-masked picture feature vector and the text feature vector to obtain a picture-text splicing vector. The vector semantic analysis module is configured to identify the predicted picture and the predicted text of the picture-text splicing vector by using a semantic analysis network in the pre-constructed text-video conversion model, including: performing predicted category classification on the picture-text splicing vector by using a classification layer in the semantic analysis network to obtain a predicted category of data corresponding to the picture-text splicing vector, including a text category and a picture category; performing dimension matrix embedding on the picture-text splicing vector by using an embedding layer in the semantic analysis network according to the predicted category, to convert the picture-text splicing vector into a vector dimension to obtain a vector dimension matrix of the picture-text splicing vector; and detecting the predicted picture and the predicted text of the picture-text splicing vector by using a full connection layer in the semantic analysis network according to the vector dimension matrix; The data decoding module is configured to decode the predicted picture by using a picture decoder in the pre-constructed text-video conversion model to obtain a predicted video, and decode the predicted text by using a text decoder in the pre-constructed text-video conversion model to obtain a predicted video text; The model loss calculation module is configured to calculate a model loss in the pre-constructed text-video conversion model according to the predicted video and the predicted video text, and the training video and the video text; The model parameter adjustment module is configured to update model parameters of the pre-constructed text-video conversion model when the model loss is greater than a preset loss, and return to the step of performing feature vector encoding on the training picture by using a picture encoder in the pre-constructed text-video conversion model; The data scene conversion module is configured to obtain a trained text-video conversion model when the model loss is not greater than the preset loss, and perform scene conversion on to-be-converted scene data by using the trained text-video conversion model to obtain a scene conversion result.

8. An electronic device, comprising: The electronic device includes: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the intelligent conversion method based on video and text according to any one of claims 1 to 6.

9. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the intelligent conversion method based on video and text according to any one of claims 1 to 6.

Images