Video description generation method and device, electronic equipment and storage medium

Abstract

The invention provides a video description generation method and device, electronic equipment and a storage medium, and belongs to the technical field of financial science and technology. The method comprises the following steps: encoding sample video image content of a sample video into sample image features based on an image modal encoder, and encoding sample video audio content of the sample video into sample audio features based on an audio modal encoder, encoding the sample video subtitle content of the sample video into sample subtitle features based on a text modal encoder; calculating feature similar data based on the sample image features, the sample audio features and the sample subtitle features; and jointly training an image modal encoder, an audio modal encoder and a text modal encoder based on the feature similar data to obtain a video description generation model, and inputting the video image, the video subtitle content and the video audio content of the target video into the video description generation model to generate a video description text. The video description generation accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, is applicable to the field of financial technology, and in particular to a video description generation method and device, an electronic device, and a storage medium. Background Art

[0002] In the field of video content processing, especially in the insurance industry, when business personnel obtain the video description copy corresponding to the insurance video, they often need to watch all the video content (background introduction, business introduction, product introduction, etc.) from beginning to end, and generate the corresponding insurance copy (for example, promotional marketing or product copy, etc.) by manually summarizing the video content. This method often has the problems of low efficiency in generating insurance description copy and difficulty in accurately reflecting the main content of the insurance video.

[0003] In this regard, the relevant technology considers the use of image description generation technology, which generates video frame images by extracting frames from the insurance video, and then uses an image encoder to perform semantic extraction on the video frame images, thereby generating a video description text corresponding to the insurance video. Although this method improves the generation efficiency of insurance description text to a certain extent, the generated insurance description text is still inaccurate because the image encoder cannot well understand the semantic feature information of the video frame image. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a video description generation method and device, an electronic device and a storage medium, aiming to improve the generation accuracy of the video description.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for generating a video description, the method comprising:

[0006] Get sample video;

[0007] Extracting sample video image content, sample video audio content, and sample video subtitle content of the sample video from the sample video;

[0008] Based on a preset image modality encoder, feature encoding is performed on the sample video image content to obtain sample image features, based on a preset audio modality encoder, feature encoding is performed on the sample video audio content to obtain sample audio features, and based on a preset text modality encoder, feature encoding is performed on the sample video subtitle content to obtain sample subtitle features;

[0009] Calculating similarity based on the sample image features, the sample audio features, and the sample subtitle features to obtain feature similarity data;

[0010] Based on the feature similarity data, jointly training the image modality encoder, the audio modality encoder, and the text modality encoder to obtain a video description generation model;

[0011] The video image, video subtitle content and video audio content of the target video are input into the video description generation model to generate a video description, thereby obtaining a video description text of the target video.

[0012] In some embodiments, extracting sample video image content of the sample video in the sample video includes:

[0013] Extracting video frames from the sample video to obtain multiple video frame images;

[0014] Performing image clustering on the multiple video frame images to obtain video frame image classes and cluster center images of the video frame image classes, wherein there are at least two video frame image classes;

[0015] The cluster center image is flattened in time sequence to obtain the sample video image content.

[0016] In some embodiments, clustering the multiple video frame images to obtain video frame image classes and cluster center images of the video frame image classes includes:

[0017] For each of the video frame images, obtaining a video frame image fingerprint feature of the video frame image;

[0018] Based on the video frame image fingerprint features, similarity calculation is performed on every two video frame images to obtain image fingerprint similarity;

[0019] Performing image clustering on the plurality of video frame images based on the image fingerprint similarity to obtain the video frame image class, so that the image fingerprint similarity of any two of the video frame images in the video frame image class is greater than a preset threshold;

[0020] For any video frame image in the video image class, determining the overall similarity of the video frame image based on the image fingerprint similarity between the video frame image and other video frame images in the video image class;

[0021] Based on the overall similarity, the cluster center image is determined among the multiple video frame images of the video image class.

[0022] In some embodiments, the performing similarity calculation based on the sample image feature, the sample audio feature, and the sample subtitle feature to obtain feature similarity data includes:

[0023] Performing cross-similarity calculation based on the sample image feature and the sample audio feature to obtain first similarity data;

[0024] Performing cross-similarity calculation based on the sample image features and the sample subtitle features to obtain second similarity data;

[0025] Performing cross-similarity calculation based on the sample audio feature and the sample subtitle feature to obtain third similarity data;

[0026] The first similarity data, the second similarity data and the third similarity data are integrated to obtain the feature similarity data.

[0027] In some embodiments, the sample image features include global image features and local image features, and the sample audio features include global audio features and local audio features;

[0028] The performing cross-similarity calculation based on the sample image feature and the sample audio feature to obtain first similarity data includes:

[0029] Calculating a first similarity between the global image feature and the global audio feature, calculating a second similarity between the global image feature and the local audio feature, calculating a third similarity between the local image feature and the global audio feature, and calculating a fourth similarity between the local image feature and the local audio feature;

[0030] Performing a comparison calculation based on the third similarity to obtain a first comparison similarity, and performing a comparison calculation based on the second similarity to obtain a second comparison similarity;

[0031] Performing a comparison calculation based on the fourth similarity to obtain a third comparison similarity;

[0032] The third similarity, the first comparison similarity, the second comparison similarity, and the third comparison similarity are averaged to obtain the first similarity data.

[0033] In some embodiments, the feature similarity data includes first similarity data, second similarity data, and third similarity data;

[0034] The step of jointly training the image modality encoder, the audio modality encoder, and the text modality encoder based on the feature similarity data to obtain a video description generation model includes:

[0035] Performing loss calculation based on the first similar data to obtain a first loss value, performing loss calculation based on the second similar data to obtain a second loss value, and performing loss calculation based on the third similar data to obtain a third loss value;

[0036] The image modality encoder is trained based on the first loss value and the third loss value, the audio modality encoder is trained based on the first loss value and the second loss value, and the text modality encoder is trained based on the second loss value and the third loss value to obtain the video description generation model.

[0037] In some embodiments, the video description generation model includes an image modality encoder, an audio modality encoder, a text modality encoder, and a text decoder;

[0038] The step of inputting the video image, video subtitle content, and video audio content of the target video into the video description generation model to generate a video description to obtain a video description text of the target video includes:

[0039] Based on the image modality encoder, feature encoding is performed on the video image to obtain video image features; based on the audio modality encoder, feature encoding is performed on the video audio content to obtain video audio features; based on the text modality encoder, feature encoding is performed on the video subtitle content to obtain video subtitle features;

[0040] Performing splicing processing on the video image feature, the video audio feature, and the video subtitle feature to obtain a video splicing feature;

[0041] The video splicing features are decoded based on the text decoder to obtain the video description text.

[0042] In some embodiments, extracting the sample video audio content of the sample video in the sample video includes:

[0043] Determining the video length of the sample video;

[0044] Based on the video length, tone features are extracted from the sample video to obtain the sample video audio content, wherein the audio length of the sample video audio content is equal to the video length.

[0045] In some embodiments, extracting sample video subtitle content of the sample video in the sample video includes:

[0046] Performing audio content recognition on the sample video to obtain a first video text;

[0047] Extracting subtitle features from the sample video to obtain a second video text;

[0048] The first video text and the second video text are integrated into the sample video subtitle content.

[0049] In some embodiments, the step of decoding the video splicing feature based on the text decoder to obtain the video description text includes:

[0050] Decoding the video splicing feature based on the text decoder to obtain a first description word;

[0051] Initializing the target vector to the word embedding feature corresponding to the first description word;

[0052] Decoding the target vector based on the text decoder to obtain a next description word;

[0053] The target vector is updated with the word embedding feature of the next description word, and the step of decoding the target vector based on the text decoder to obtain the next description word is returned until a stop symbol is generated, and the obtained multiple description words are spliced ​​into the video description text.

[0054] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a video description generating device, the device comprising:

[0055] An acquisition unit, used for acquiring a sample video;

[0056] An extraction unit, configured to extract, from the sample video, sample video image content, sample video audio content, and sample video subtitle content of the sample video;

[0057] an encoding unit, configured to perform feature encoding on the sample video image content based on a preset image mode encoder to obtain sample image features, perform feature encoding on the sample video audio content based on a preset audio mode encoder to obtain sample audio features, and perform feature encoding on the sample video subtitle content based on a preset text mode encoder to obtain sample subtitle features;

[0058] A calculation unit, configured to perform similarity calculation based on the sample image feature, the sample audio feature, and the sample subtitle feature to obtain feature similarity data;

[0059] A training unit, configured to jointly train the image modality encoder, the audio modality encoder, and the text modality encoder based on the feature similarity data to obtain a video description generation model;

[0060] The generating unit is used to input the video image, video subtitle content and video audio content of the target video into the video description generating model to generate the video description, so as to obtain the video description text of the target video.

[0061] To achieve the above-mentioned purpose, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the video description generation method described in the first aspect is implemented.

[0062] To achieve the above objectives, a fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the video description generation method described in the first aspect is implemented.

[0063] The video description generation method and device, electronic device and storage medium proposed in the present application obtain a sample video; in the sample video, extract the sample video image content, sample video audio content, and sample video subtitle content of the sample video, so as to realize the extraction of content information of multiple modes of the sample video, which is conducive to improving the comprehensiveness of feature extraction. Furthermore, based on the preset image modality encoder, the sample video image content is feature encoded to obtain sample image features, based on the preset audio modality encoder, the sample video audio content is feature encoded to obtain sample audio features, and based on the preset text modality encoder, the sample video subtitle content is feature encoded to obtain sample subtitle features, which can realize the unified encoding of content information of multiple modes of the sample video, and convert the feature content of the image modality, audio modality and text modality into the same feature dimension. Furthermore, similarity calculation is performed based on sample image features, sample audio features, and sample subtitle features to obtain feature similarity data, and based on the feature similarity data, the image modality encoder, audio modality encoder, and text modality encoder are jointly trained to obtain a video description generation model. This method realizes cross-similarity calculation of feature content of multiple modalities, and takes into account feature alignment of feature content of multiple modalities, so that the video description generation model can learn more about the correlation of feature information of different modalities during training, which is conducive to training a video description generation model with better model performance. Finally, the video image, video subtitle content, and video audio content of the target video are input into the video description generation model for video description generation to obtain the video description text of the target video, which can improve the accuracy and efficiency of the video description text of the generated video. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a flowchart of a method for generating a video description provided in an embodiment of the present application;

[0065] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.

[0066] Figure 3 yes Figure 1 Flow chart of step S102 in FIG.

[0067] Figure 4 yes Figure 1 Flow chart of step S104 in FIG.

[0068] Figure 5 yes Figure 4 Flow chart of step S401 in FIG.

[0069] Figure 6 yes Figure 1 Flow chart of step S105 in FIG.

[0070] Figure 7 yes Figure 1 Flow chart of step S106 in FIG.

[0071] Figure 8 is a structural schematic diagram of a video description generating device provided in an embodiment of the present application;

[0072] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0074] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0076] The video description generation method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the video description generation method in the embodiments of the present application is described.

[0077] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0078] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0079] The video description generation method provided in the embodiment of the present application relates to the fields of financial technology and artificial intelligence technology. The video description generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the video description generation method, etc., but is not limited to the above forms.

[0080] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0081] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the identity or characteristics of the object, such as object information, object behavior data, object historical data, and object location information, the permission or consent of the object will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the personal information of the object, the separate permission or separate consent of the object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the object, the necessary object-related data for the normal operation of the embodiment of the present application will be obtained.

[0082] Figure 1 is an optional flowchart of the video description generation method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.

[0083] Step S101, obtaining a sample video;

[0084] Step S102, extracting sample video image content, sample video audio content, and sample video subtitle content of the sample video from the sample video;

[0085] Step S103, feature encoding the sample video image content based on a preset image mode encoder to obtain sample image features, feature encoding the sample video audio content based on a preset audio mode encoder to obtain sample audio features, and feature encoding the sample video subtitle content based on a preset text mode encoder to obtain sample subtitle features;

[0086] Step S104, performing similarity calculation based on sample image features, sample audio features, and sample subtitle features to obtain feature similarity data;

[0087] Step S105, based on the feature similarity data, jointly training the image modality encoder, the audio modality encoder, and the text modality encoder to obtain a video description generation model;

[0088] Step S106, inputting the video image, video subtitle content and video audio content of the target video into a video description generation model to generate a video description, thereby obtaining a video description text of the target video.

[0089] Steps S101 to S106 shown in the embodiment of the present application, by acquiring a sample video; in the sample video, extracting the sample video image content, the sample video audio content, and the sample video subtitle content of the sample video, can achieve the extraction of content information of multiple modes of the sample video, which is conducive to improving the comprehensiveness of feature extraction. Further, based on the preset image modality encoder, feature encoding is performed on the sample video image content to obtain sample image features, based on the preset audio modality encoder, feature encoding is performed on the sample video audio content to obtain sample audio features, and based on the preset text modality encoder, feature encoding is performed on the sample video subtitle content to obtain sample subtitle features, which can achieve unified encoding of content information of multiple modes of the sample video, and convert the feature content of the image modality, audio modality, and text modality into the same feature dimension. Furthermore, similarity calculation is performed based on sample image features, sample audio features, and sample subtitle features to obtain feature similarity data, and based on the feature similarity data, the image modality encoder, audio modality encoder, and text modality encoder are jointly trained to obtain a video description generation model. This method realizes cross-similarity calculation of feature content of multiple modalities, and takes into account feature alignment of feature content of multiple modalities, so that the video description generation model can learn more about the correlation of feature information of different modalities during training, which is conducive to training a video description generation model with better model performance. Finally, the video image, video subtitle content, and video audio content of the target video are input into the video description generation model for video description generation to obtain the video description text of the target video, which can improve the accuracy and efficiency of generating video description text for dangerous videos.

[0090] In step S101 of some embodiments, a sample video is obtained.

[0091] Sample videos refer to insurance video data that include background introductions, business introductions, product introductions, insurance talk training, and other content in the insurance field.

[0092] In the specific implementation of this embodiment, when training the video description generation model, with authorization and permission, insurance-related video materials can be obtained from various insurance business platforms or public websites, and video materials containing background introductions, business introductions, product introductions, and various insurance speech popular science popularizations in the insurance field can be used as sample videos.

[0093] In step S102 of some embodiments, sample video image content, sample video audio content, and sample video subtitle content of the sample video are extracted from the sample video.

[0094] The sample video image content is used to indicate the image information contained in the sample video.

[0095] The sample video audio content is used to indicate the audio information contained in the sample video.

[0096] The sample video subtitle content is used to indicate the subtitle information contained in the sample video.

[0097] See also Figure 2 In some embodiments, in a sample video, a specific process of extracting sample video image content of the sample video may include but is not limited to steps S201 to S203:

[0098] Step S201, extracting video frames from a sample video to obtain a plurality of video frame images;

[0099] Step S202, performing image clustering on a plurality of video frame images to obtain video frame image classes and cluster center images of the video frame image classes;

[0100] Step S203, flattening the cluster center image in time sequence to obtain sample video image content.

[0101] Steps S201 to S203 are described in detail below.

[0102] In step S201 of some embodiments, the video frame image refers to a video image corresponding to each frame in the sample video.

[0103] When this embodiment is implemented, a video frame capture tool or video editing software may be used to capture video frames of the sample video, and the sample video may be divided into video images one by one in time sequence to obtain a plurality of video frame images.

[0104] In step S202 of some embodiments, the video frame image class refers to a large class consisting of multiple video frame images with relatively similar video content. There are at least two video frame image classes. The cluster center image refers to a video frame image in the video frame image class that has image features that are relatively similar to those of each video frame image.

[0105] In the specific implementation of this embodiment, step S202 specifically includes but is not limited to the following steps:

[0106] For each video frame image, obtaining a video frame image fingerprint feature of the video frame image;

[0107] Based on the video frame image fingerprint features, the similarity between every two video frame images is calculated to obtain the image fingerprint similarity;

[0108] Performing image clustering on multiple video frame images based on image fingerprint similarity to obtain a video frame image class, so that the image fingerprint similarity of any two video frame images in the video frame image class is greater than a preset threshold;

[0109] For any video frame image in the video image class, based on the image fingerprint similarity between the video frame image and other video frame images in the video image class, determine the overall similarity of the video frame image;

[0110] Based on the overall similarity, a cluster center image is determined among the multiple video frame images of the video image class.

[0111] The video frame image fingerprint feature refers to the feature information that can be used to uniquely identify or distinguish the video frame image. Image fingerprint similarity is used to indicate the similarity between the image features of two video frame images. The preset threshold is used to measure whether the two video frame images can be classified into the same category. The overall similarity is used to indicate the overall similarity between the image features of the video frame image and other video frame images of the same video frame image class.

[0112] Specifically, for each video frame image, first, feature extraction is performed on the video frame image to obtain image feature information of the video frame image, and the video frame image fingerprint feature of the video frame image is obtained. Then, for each video frame image, the video frame image fingerprint feature of the video frame image and the video frame image fingerprint features of other video frame images are respectively calculated by cosine similarity to obtain the image fingerprint similarity between every two video frame images. Further, according to the image fingerprint similarity of each video frame image with other video frame images, image clustering is performed on multiple video frame images, and multiple video frame images are divided into at least two video frame image classes, so that the image fingerprint similarity of any two video frame images in the same video frame image class is greater than a preset threshold, and the image fingerprint similarity of any two video frame images in different video frame image classes is less than or equal to the preset threshold. Then, for each video frame image in a video frame image class, the image fingerprint similarity of the video frame image and other video frame images in the video frame image class is summed to obtain the overall similarity of the video frame image. Furthermore, in the video frame image class, the video frame image with the largest overall similarity is determined as the cluster center image of the video image class.

[0113] In step S203 of some embodiments, first, the video frame time point of each cluster center image is obtained. Then, according to the video frame time point, the cluster center image is flattened in time sequence, and multiple cluster center images are spliced ​​in order from early to late in the video frame time point to obtain sample video image content.

[0114] The benefit of this embodiment is that when extracting sample video image content from a sample video, the sample video is first divided into multiple video frame images, and the video frame images are used as the basic unit. Image clustering is implemented according to the similarity of the video frame image fingerprint features of every two video frame images, and video frame images with relatively similar image fingerprint feature information are classified into the same major category. Since the cluster center images of each video frame image class can represent the key significant information of the storyline in the sample video, the cluster center images of each video frame image class are used as the basic images selected to construct the sample video image content, and the cluster center images are spliced ​​into the sample video image content. This method can capture fewer video frame images corresponding to significant scenes with significant changes to represent the image content of the entire sample video while retaining fewer static video frames and more dynamic scene frames, which can improve the accuracy and simplicity of the image content of the extracted sample video.

[0115] See also Figure 3 In some embodiments, the specific process of extracting the sample video audio content of the sample video from the sample video may include but is not limited to steps S301 to S302:

[0116] Step S301, determining the video length of the sample video;

[0117] Step S302: extracting the pitch features of the sample video based on the video length to obtain the audio content of the sample video.

[0118] Steps S301 to S302 are described in detail below.

[0119] In step S301 of some embodiments, the video length is used to indicate the total duration of the sample video.

[0120] When this embodiment is implemented, a timer or other time counting tools may be used to collect the overall video duration of the sample video to obtain the video length of the sample video.

[0121] In step S302 of some embodiments, the audio length of the sample video audio content is equal to the video length.

[0122] When this embodiment is implemented, the multimedia framework FFmpeg can be used to extract audio information from the sample video. Specifically, the multimedia framework is called, and when the instruction parameters are set, the designated input file is determined as the sample video FFmpeg, and the video stream of the sample video is not retained, but the audio stream of the sample video is copied, so as to obtain the preliminary video audio content, wherein the audio length of the preliminary video audio content is equal to the video length. Furthermore, the audio source separation tool Bytesep can also be used to decouple the tone data in the preliminary video audio content from the mixed audio containing various sounds. Among them, the result of the decoupled sound can be regarded as audio data that shares the content plot with the fragment video in the sample video without any language information, based on which, the result of the decoupled sound is used as the sample video audio content.

[0123] It should be noted that the audio of the sample video is separated into speech because the video subtitles already contain the information of the language sound, so the separated speech does not need speech information, and only the pitch intensity information is retained.

[0124] The advantage of this embodiment is that, taking into account the large overlap between the language information in the audio content and the subtitle information of the sample video, when extracting the sample audio content of the sample video, the multimedia framework is first used to obtain the audio stream of the sample video, and then the obtained audio stream is subjected to a speech separation operation, and the portion of the audio stream that does not contain language information but contains tone information is separated out as the sample video audio content. This method can improve the content accuracy and conciseness of the extracted sample video audio content.

[0125] In some embodiments, the specific process of extracting the sample video subtitle content of the sample video from the sample video may include but is not limited to the following steps:

[0126] Performing audio content recognition on the sample video to obtain a first video text;

[0127] Extracting subtitle features from the sample video to obtain a second video text;

[0128] The first video text and the second video text are integrated into sample video subtitle content.

[0129] The first video text is used to indicate the text content formed according to the audio information of the sample video, and the second video text is used to indicate the text content formed according to the subtitle information of the sample video.

[0130] In the specific implementation of this embodiment, first, when performing audio content recognition on the sample video, the automatic speech recognition technology (ASR) can be used to convert the speech content in the sample video into text to obtain the first video text. At the same time, when performing subtitle feature extraction on the sample video to obtain the second video text, the optical character recognition technology (OCR) is used to perform text recognition on each video frame image of the sample video to extract the text content in the sample video to obtain the second video text. Finally, according to the first video text and the second video text, the text content of the first video text and the text content of the second video text are integrated to obtain the subtitle content of the sample video.

[0131] It should be noted that if the text content in the sample video is dynamic or synchronized with the audio, it is easier to extract more accurate subtitle content of the sample video using automatic speech recognition technology (ASR); if the text content in the sample video is static or presented in the form of an image, it is easier to extract more accurate subtitle content of the sample video using optical character recognition technology (OCR).

[0132] Furthermore, when the sample video contains human voice or subtitle content, the above-mentioned automatic speech recognition technology (ASR) or optical character recognition technology (OCR) can be directly used. When there is no human voice and subtitle content in the sample video, it is necessary to use a method that combines command detection and long short-term memory deep convolutional network to generate sample video subtitle content of the sample video. Specifically, first, command features are extracted from the audio of the sample video using command detection technology. At the same time, the long short-term memory deep convolutional network is used to extract features of the sample video to capture the deep feature information of the sample video and obtain the video deep feature. Then, the command features and the video deep features are directly added or outer product operations are performed to achieve feature fusion and obtain video fusion features. Furthermore, the video fusion features are input into the text generation model to generate text information, and the text information is used as the sample video subtitle content, wherein the text generation model can be an LSTM model or a transformer model, etc.

[0133] The advantage of this embodiment is that when extracting the sample video subtitle content of the sample video, whether the sample video contains character voices and subtitle content is taken into consideration, and multiple methods for generating sample video subtitle content are provided, such as based on automatic speech recognition technology, based on optical character recognition technology, or based on the fusion of command detection and long short-term memory deep convolutional network. This can improve the universality of generating sample video subtitle content and meet the needs of extracting sample video subtitle content in various situations.

[0134] In step S103 of some embodiments, feature encoding is performed on the sample video image content based on a preset image modality encoder to obtain sample image features, feature encoding is performed on the sample video audio content based on a preset audio modality encoder to obtain sample audio features, and feature encoding is performed on the sample video subtitle content based on a preset text modality encoder to obtain sample subtitle features.

[0135] The image modality encoder refers to an encoder that converts image data into a feature representation that the model can understand. The sample image features are used to indicate the image feature information contained in each video frame image in the sample video image content. The audio modality encoder refers to an encoder that converts audio data into a feature representation that the model can understand. The sample audio features are used to indicate the audio feature information contained in each audio clip in the sample video audio content. The text modality encoder refers to an encoder that converts text data into a feature representation that the model can understand. The sample subtitle features are used to indicate the feature information reflected by the sample video subtitle content.

[0136] In the specific implementation of this embodiment, first, the image modality encoder is used to perform feature encoding on the sample video image content, and each video frame image in the sample video image content is mapped to a latent vector space with a fixed feature dimension, so as to obtain sample image features corresponding to the sample video image content. Next, the audio modality encoder is used to perform feature encoding on the sample video audio content, and each audio clip in the sample video audio content is mapped to a latent vector space with a fixed feature dimension, so as to obtain sample audio features corresponding to the sample video audio content. Finally, the text modality encoder is used to perform feature encoding on the sample video subtitle content, and the sample video subtitle content is mapped to a latent vector space with a fixed feature dimension, so as to obtain sample subtitle features corresponding to the sample video subtitle content.

[0137] It should be noted that the feature dimensions of the sample image features, sample audio features, and sample subtitle features are the same. Furthermore, the sample image features include the image features of each video frame image in the sample video image content (local image features ff), and the overall image features (global image features fv) obtained by averaging the image features of each video frame image. The sample audio features include the audio features of each audio clip in the sample video audio content (local audio features fas), and the overall audio features (global audio features fa) obtained by averaging the audio features of each audio clip. The sample subtitle features include the word features of each word in the sample video subtitle content (local subtitle features fw), and the overall subtitle features (global subtitle features fs) obtained by averaging the word features of each word.

[0138] In step S104 of some embodiments, similarity calculation is performed based on sample image features, sample audio features, and sample subtitle features to obtain feature similarity data.

[0139] See also Figure 4 In some embodiments, step S104 may include but is not limited to steps S401 to S404:

[0140] Step S401, performing cross-similarity calculation based on sample image features and sample audio features to obtain first similarity data;

[0141] Step S402, performing cross-similarity calculation based on sample image features and sample subtitle features to obtain second similarity data;

[0142] Step S403, performing cross-similarity calculation based on the sample audio features and the sample subtitle features to obtain a third similarity data

[0143] Step S404: Integrate the first similar data, the second similar data and the third similar data to obtain feature similar data.

[0144] Steps S401 to S404 are described in detail below.

[0145] In step S401 of some embodiments, the first similarity data is used to indicate the degree of similarity between the sample image features and the sample audio features.

[0146] To save space, the specific process of performing cross-similarity calculation based on sample image features and sample audio features in the embodiment of the present application will be described in detail below and will not be repeated here.

[0147] In step S402 of some embodiments, the second similarity data is used to indicate the degree of similarity between the sample image feature and the sample subtitle feature.

[0148] In the specific implementation of this embodiment, the specific process of step S402 is similar to the above step S401, and will not be described in detail to save space.

[0149] In step S403 of some embodiments, the third similarity data is used to indicate the degree of similarity between the sample audio feature and the sample subtitle feature.

[0150] In the specific implementation of this embodiment, the specific process of step S403 is similar to the above step S401, and will not be described in detail to save space.

[0151] In step S404 of some embodiments, the first similar data, the second similar data and the third similar data are integrated, and the first similar data, the second similar data and the third similar data are included in the same data set, and all the data in the data set are used as feature similar data.

[0152] The benefit of this embodiment is that it takes into account pairwise cross-similarity calculations of sample image features, sample subtitle features, and sample audio features of the sample video, and implements cross-feature alignment of feature information of the image modality, audio modality, and text modality of the sample video, which enables the model to better learn the correlation between different modalities during training, thereby improving the accuracy of video description texts generated by the trained video description generation model for various videos.

[0153] In the embodiment of the present application, the sample image features include global image features and local image features, and the sample audio features include global audio features and local audio features.

[0154] The global image feature is used to indicate the overall feature information of multiple video frame images in the sample video image content.

[0155] The local image features are used to indicate feature information of each video frame image in the sample video image content.

[0156] The global audio feature is used to indicate the overall feature information of multiple audio clips in the sample video audio content.

[0157] The local audio feature is used to indicate the feature information of a single audio segment in the sample video audio content.

[0158] See also Figure 5 In some embodiments, step S401 may include but is not limited to steps S501 to S504:

[0159] Step S501, calculating a first similarity between a global image feature and a global audio feature, calculating a second similarity between a global image feature and a local audio feature, calculating a third similarity between a local image feature and a global audio feature, and calculating a fourth similarity between a local image feature and a local audio feature;

[0160] Step S502, performing a comparison calculation based on the third similarity to obtain a first comparison similarity, and performing a comparison calculation based on the second similarity to obtain a second comparison similarity;

[0161] Step S503, performing a comparison calculation based on the fourth similarity to obtain a third comparison similarity;

[0162] Step S504: perform mean processing on the third similarity, the first comparison similarity, the second comparison similarity, and the third comparison similarity to obtain first similarity data.

[0163] Steps S501 to S504 are described in detail below.

[0164] In step S501 of some embodiments, the first similarity is used to indicate the overall similarity between the sample video image content and the sample video audio content of the sample video. The second similarity is used to indicate the similarity between the sample video image content and the multiple audio clips in the sample video audio content. The third similarity is used to indicate the similarity between the multiple video frame images in the sample video image content of the sample video and the sample video audio content. The fourth similarity is used to indicate the feature similarity between a single video frame image and a single audio clip.

[0165] In the specific implementation of this embodiment, the first similarity between the global image feature and the global audio feature is calculated using cosine similarity. At the same time, the second similarity between the global image feature and each local audio feature is calculated using cosine similarity. The third similarity between each local image feature and the global audio feature is calculated using cosine similarity. The fourth similarity between each local image feature and each local audio feature is calculated using cosine similarity.

[0166] It should be noted that when there are m cluster center images in the sample video image content of the sample video and n audio clips in the sample video audio content of the sample video, 1 first similarity, n second similarities, m third similarities and m×n fourth similarities will be obtained, where m and n are both positive integers.

[0167] In a specific example, the first similarity S a,v It can be expressed as: S a,v =(f a ) T (f v ), where f v is the global image feature, f a is the global audio feature, (f a ) T Refers to the transposed result of the global image features. The first similarity S a,v It is data in one-dimensional real number space.

[0168] In a specific example, the second similarity S as,v It can be expressed as: S as,v =f as f v , where f v is the global image feature, f asis the local audio feature corresponding to a single audio clip. The second similarity S formed by all audio clips as,v is a real matrix with n rows and one column.

[0169] In a specific example, the third similarity S a,f It can be expressed as: S a,f =(f a f f ) T , where f f is the local image feature corresponding to a single cluster center image, f a is the global audio feature, (f a f f ) T It refers to the transposed result of the product of the local image feature and the global audio feature. All cluster center images form the third similarity S a,f is a real matrix with one row and m columns.

[0170] In a specific example, the fourth similarity S as,f It can be expressed as: S as,f =(f f ) T (f as ), where f f is the local image feature corresponding to a single cluster center image, f as is the local audio feature corresponding to a single audio clip, (f f ) T Refers to the transposition result of the local image features of a single cluster center image. The fourth similarity S formed by all audio clips and all cluster center images as,f is a real matrix with n rows and m columns.

[0171] In step S502 of some embodiments, the first comparative similarity is used to indicate the overall similarity between the local image features of all cluster center images and the global audio features, and the second comparative similarity is used to indicate the overall similarity between the local audio features of all audio clips and the global image features.

[0172] It should be noted that the first comparison similarity and the second comparison similarity are both instance-level similarities. Instance-level similarity is generally used to evaluate the semantic relevance between two specific instances in training data.

[0173] In the specific implementation of this embodiment, when performing the comparison calculation based on the third similarity, for each cluster center image, the third similarity of the cluster center image is normalized to obtain the first comparison similarity of the cluster center image. Similarly, when performing the comparison calculation based on the second similarity, for each audio segment, the second similarity of the audio segment is normalized to obtain the second comparison similarity of the audio segment.

[0174] It should be noted that the above normalization process is mainly used to convert the original second similarity and the third similarity into a probability distribution form.

[0175] In a specific example, the first comparison similarity can be expressed as:

[0176]

[0177] Among them, S a,f is the third similarity, S′ a,f is the first comparison similarity, S a,f(1,i) is the element in the 1st row and the ith column of the third similarity (i.e., the similarity between the local image feature of the ith cluster center image and the global audio feature); exp(S a,f(1,i) ) means that the base is a natural constant, with S a,f(1,i) is the result of the index calculation; S a,f(1,j) Represents the element in the 1st row and the jth column in the third similarity (i.e., the similarity between the local image feature of the jth cluster center image and the global audio feature); is the sum of m numbers with natural constants as base and S a,f(1,j) The first comparison similarity is to compare the normalized probability with the original similarity matrix S a,f The multiplication is performed to achieve the probability weighting of each element in the third similarity. At this time, i and j are positive integers not greater than m.

[0178] The second comparison similarity can be expressed as:

[0179]

[0180] Among them, S as,v is the second similarity, S′ as,v is the second comparison similarity, S as,v(p,1) is the element in the pth row and the first column of the second similarity (i.e., the similarity between the local audio feature of the pth audio clip and the global image feature); exp(S as,v(p,1) ) means that the base is a natural constant, with S as,v(p,1) is the result of the index calculation; S as,v(q,1)represents the element in the qth row and the first column in the second similarity (i.e., the similarity between the local audio feature of the qth audio clip and the global image feature); is the sum of n numbers with natural constants as base and S ss,v(q,1) The second similarity comparison is to compare the normalized probability with the original similarity matrix S as,v The multiplication is performed to achieve the probability weighting of each element in the second similarity. At this time, p and q are positive integers not greater than n.

[0181] In step S503 of some embodiments, the third comparative similarity is used to indicate the comprehensive similarity of the feature information of the sample video at the audio level and the image level.

[0182] In the specific implementation of this embodiment, first, for a single audio clip, the fourth similarity between the audio clip and all cluster center images is normalized to obtain a first normalized similarity. Then, for a single cluster center image, the fourth similarity between the cluster center image and all audio clips is normalized to obtain a second normalized similarity. Further, a weighted calculation is performed based on the first normalized similarity and the second normalized similarity to obtain a third comparison similarity.

[0183] In a specific example, the first normalized similarity can be expressed as follows:

[0184]

[0185] Among them, S avd(1,i) Refers to the similarity value corresponding to the cluster center image in the 1st row and ith column in the first similarity matrix; S avd(1,j) Represents the similarity value corresponding to the cluster center image in the 1st row and the jth column in the first similarity matrix, where i and j are positive integers not greater than m.

[0186] Furthermore, the first similarity matrix is ​​expressed as follows:

[0187]

[0188] Among them, S as,f(p,*) represents the fourth similarity between the pth audio segment and all cluster center images; S as,f(q,*) represents the fourth similarity between the qth audio segment and all cluster center images; It represents the sum of the exponential results of the natural base exponential calculation using the fourth similarity between each of the n audio clips and all the cluster center images as the index. avd is a 1×m vector matrix, S avd S represents the normalized similarity between each audio clip and all cluster center images.as,f(p,) represents the fourth similarity between the p-th audio segment and each cluster center image. represents the normalization coefficient of the pth audio segment, represents the normalized similarity between the pth audio clip and each cluster center image, which is a 1×m vector matrix. Based on this, Indicates that for each cluster center image, the normalized similarities of all audio clips corresponding to the cluster center image are added to obtain a similarity value. The vector matrix composed of the similarity values ​​of the M cluster center images is the first similarity matrix S avd The first similarity matrix is ​​also a 1×m vector matrix, where p and q are positive integers not greater than n.

[0189] Similarly, the second normalized similarity can be expressed as follows:

[0190]

[0191] Among them, S vid(p,1) Refers to the similarity value corresponding to the audio clip in the pth row and the first column in the second similarity matrix; S vid(q,1) represents the similarity value corresponding to the audio segment in the qth row and the first column in the second similarity matrix, where p and q are positive integers not greater than n.

[0192] Furthermore, the second similarity matrix is ​​expressed as follows:

[0193]

[0194] Among them, S as,f(*,i) represents the fourth similarity between the i-th cluster center image and all audio clips; S as,f(*,j) represents the fourth similarity between the jth cluster center image and all audio clip images; It represents the sum of the exponential results of the natural base exponential calculation with the fourth similarity between each of the m cluster center images and all the audio clips as the index. vid is a 1×n vector matrix, S vid S represents the normalized similarity between each cluster center image and all audio clips. as , f( , i) represents the fourth similarity between the i-th cluster center image and each audio segment. represents the normalization coefficient of the i-th cluster center image, represents the normalized similarity between the i-th cluster center image and each audio clip, which is an n×1 vector matrix. Based on this, Indicates that for each audio clip, the normalized similarities of all cluster center images corresponding to the audio clip are added to obtain a similarity value. The vector matrix composed of the similarity values ​​of N audio clips is the second similarity matrix S vid The second similarity matrix is ​​also an n×1 vector matrix, where i and j are positive integers not greater than m.

[0195] In step S504 of some embodiments, the third similarity, the first comparison similarity, the second comparison similarity, and the third comparison similarity are averaged. The second similarity, the first comparison similarity, the second comparison similarity, and the third comparison similarity are first added together to obtain a total similarity; then, the total similarity is divided by four to obtain a similarity average, and this similarity average is used as the first similarity data of the sample video.

[0196] In a specific example, the first similarity data Sim of the sample video AV It can be expressed as follows:

[0197] Sim AV =(S a,f +S' a,f +S′ as,v +S′ as,f ) / 4;

[0198] Among them, S a,f is the second similarity, S' a,f is the first comparison similarity, S′ as,v is the second comparison similarity, S′ as,f is the third comparison similarity.

[0199] The advantage of this embodiment is that after calculating the similarity of the feature contents of the sample video in the image modality, audio modality and text modality respectively, the feature alignment of the feature contents of each modality is considered, and the idea of ​​cross-similarity calculation is used to align the feature contents of the audio modality and image modality. The similar feature alignment at the instance level, the similar feature alignment at the audio level and the similar feature alignment at the image level are combined, which can improve the accuracy and effectiveness of the feature alignment.

[0200] In step S105 of some embodiments, based on feature similarity data, the image modality encoder, the audio modality encoder, and the text modality encoder are jointly trained to obtain a video description generation model.

[0201] The video description generation model refers to an artificial intelligence model that can generate corresponding video description text based on the video. The video description generation model generally takes the video as input and the corresponding video description text as output.

[0202] In an embodiment of the present application, the video description generation model includes an image modality encoder, an audio modality encoder, a text modality encoder, and a text decoder.

[0203] The text decoder is used to map the feature vector generated by the video description generation model according to the input data from the latent vector space to the text data space, so as to output the text content corresponding to the input data.

[0204] In the embodiment of the present application, the text decoder may be an autoregressive model of a transformer structure.

[0205] See also Figure 6 In some embodiments, step S105 may include but is not limited to steps S601 to S602:

[0206] Step S601, performing loss calculation based on the first similar data to obtain a first loss value, performing loss calculation based on the second similar data to obtain a second loss value, and performing loss calculation based on the third similar data to obtain a third loss value;

[0207] Step S602: train an image modality encoder based on the first loss value and the third loss value, train an audio modality encoder based on the first loss value and the second loss value, and train a text modality encoder based on the second loss value and the third loss value to obtain a video description generation model.

[0208] Steps S601 to S602 are described in detail below.

[0209] In step S601 of some embodiments, the first loss value is used to indicate the difference between the similarity between the sample image feature and the sample audio feature and the training expectation. The second loss value is used to indicate the difference between the similarity between the sample image feature and the sample subtitle feature and the training expectation. The third loss value is used to indicate the difference between the similarity between the sample audio feature and the sample subtitle feature and the training expectation.

[0210] When this embodiment is specifically implemented, the idea of ​​the contrast loss function can be used in the embodiment of the present application to calculate the first loss function. Specifically, since there are generally multiple sample videos during training, the first similarity data between the sample image features and the sample audio features of the same sample video can be used as the positive sample value, and the first similarity data between the sample image features and the sample audio features of different sample videos can be used as the negative sample value, wherein, for a sample video, the positive sample value is generally one, and the negative sample value is generally multiple. Then, for a single sample video, the result with the natural parameter as the base and the positive sample value as the exponent is used as the numerator, and the sum of the results with the natural constant as the base and the negative sample value as the exponent is used as the denominator to obtain the first result. Further, take the negative logarithm of the first result to obtain the sub-loss value of a single sample video, and average the sub-loss values ​​of multiple sample videos to obtain the first loss value.

[0211] For example, when there are three sample videos, namely, sample video A, sample video B, and sample video C, for sample video A, the first similarity data between the sample image feature of sample video A and the sample audio feature is used as the positive sample value 1. The first similarity data between the sample audio feature of sample video A and the sample image feature of sample video B is used as the negative sample value 1, and the first similarity data between the sample audio feature of sample video A and the sample image feature of sample video C is used as the negative sample value 2. When calculating the sub-loss value of sample video A, the result with the natural parameter as the base and the positive sample value 1 as the exponent is used as the numerator, and the sum of the result with the natural constant as the base and the negative sample value 1 as the exponent and the result with the natural constant as the base and the negative sample value 2 as the exponent is used as the denominator to obtain the first result.

[0212] In step S602 of some embodiments, when training the image modality encoder based on the first loss value and the third loss value, the first loss value and the third loss value are first added to obtain a total loss value. Then, the total loss value is compared with a preset loss threshold. If the total loss value is not less than the preset loss threshold, the parameters of the image modality encoder are adjusted, and the process returns to the above steps S101-S105, and the image modality encoder is continuously iterated and trained until the total loss value is less than the preset loss threshold in a certain training round, the training is stopped, the parameters of the image modality encoder of the training round are used as the final model parameters, and the image modality encoder of the training round is used as the final image modality encoder for constructing the video description generation model.

[0213] Further, the specific process of training the audio modality encoder based on the first loss value and the second loss value is similar to the specific process of training the image modality encoder based on the first loss value and the third loss value. The specific process of training the text modality encoder based on the second loss value and the third loss value is similar to the specific process of training the image modality encoder based on the first loss value and the third loss value. To save space, it will not be repeated.

[0214] Furthermore, after the image modality encoder, audio modality encoder and text modality encoder are all trained, the image modality encoder, audio modality encoder and text modality encoder are connected in parallel, and the parallel network structure is connected in series with the text decoder to form a complete video description generation model.

[0215] The benefit of this embodiment is that it takes into account the joint training of the image modality encoder, audio modality encoder and text modality encoder based on feature similarity data, takes the similarity data of the feature content between different modalities of the sample video as the basis for calculating the contrast loss, and performs contrast loss calculation on the image modality encoder, audio modality encoder and text modality encoder respectively. It can maximize the similarity of the feature content of each modality of the same sample video and minimize the similarity of the feature content of each model of different sample videos, thereby improving the accuracy of the insurance description text generated by the trained video description generation model.

[0216] In step S106 of some embodiments, the video image, video subtitle content, and video audio content of the target video are input into a video description generation model to generate a video description, thereby obtaining a video description text of the target video.

[0217] The target video refers to the video for which a video description needs to be generated.

[0218] The video description text of the target video refers to text data output by the video description generation model based on the target video and capable of describing the main content of the target video.

[0219] See also Figure 7 In some embodiments, step S106 may include but is not limited to steps S701 to S703:

[0220] Step S701, feature encoding the video image based on the image modality encoder to obtain video image features; feature encoding the video audio content based on the audio modality encoder to obtain video audio features; feature encoding the video subtitle content based on the text modality encoder to obtain video subtitle features;

[0221] Step S702, performing splicing processing on the video image features, the video audio features, and the video subtitle features to obtain video splicing features;

[0222] Step S703: Decode the video splicing features based on the text decoder to obtain a video description text.

[0223] Steps S701 to S703 are described in detail below.

[0224] In step S701 of some embodiments, the specific implementation process of the embodiment is similar to the above step S103, which will not be described in detail to save space.

[0225] In step S702 of some embodiments, since the video image features, video audio features, and video subtitle features are feature vectors of the same feature dimension, based on this, the video image features, video audio features, and video subtitle features can be directly spliced ​​to form a longer feature vector, and the spliced ​​feature vector is used as a video splicing feature. The video splicing feature is used to comprehensively represent the feature information of the sample video in the image modality, text modality, and audio modality.

[0226] In step S703 of some embodiments, the specific implementation of this embodiment may include but is not limited to the following steps:

[0227] Decode the video splicing features based on the text decoder to obtain the first description word;

[0228] Initialize the target vector to the word embedding feature corresponding to the first description word;

[0229] Decode the target vector based on the text decoder to obtain the next description word;

[0230] The target vector is updated with the word embedding feature of the next description word, and the target vector is decoded based on the text decoder to obtain the step of the next description word until a stop symbol is generated, and the obtained multiple description words are concatenated into a video description text.

[0231] The first descriptor refers to the descriptor predicted by the text decoder based on the video splicing features. The word embedding feature is used to indicate the semantic feature information of the first descriptor. The next descriptor refers to the descriptor predicted by the text decoder based on the target vector. The stop symbol is used to define when the decoding process of the text decoder stops.

[0232] Specifically, first, when the video splicing feature is decoded by the text decoder, the video splicing feature is linearly projected to obtain the query feature, key feature and value feature corresponding to the video splicing feature, and attention calculation is performed based on the query feature, key feature and value feature to obtain the video attention calculation result corresponding to the video splicing feature, and the video attention calculation result is feedforward processed to obtain the video feedforward feature, and the video feedforward feature is mapped from the latent vector space to the data space to obtain the first description word. Further, the first description word is embedded to obtain the word embedding feature corresponding to the first description word, and the target vector is initialized to the word embedding feature corresponding to the first description word. Further, the target vector is decoded based on the text decoder to obtain the next description word, and this process is similar to the process of decoding the video splicing feature based on the text decoder to obtain the first description word. Next, the target vector is updated with the word embedding feature of the next description word, and the target vector is decoded based on the text decoder to obtain the next description word, until a stop symbol is decoded according to the word embedding feature of a description word, the decoding process is terminated, and multiple description words are concatenated in sequence according to the generation time to obtain the video description text.

[0233] For example, the text decoder generates the first word based on the video splicing feature at the first moment, and uses the vector of the first word as input to generate the second word at the second moment, until a stop symbol is generated. The words generated at all moments are then concatenated to form the video description text of the target video, which is used to represent the entire copy content of the target video.

[0234] The benefit of this embodiment is that the video description is generated by using the video image, video subtitle content and video audio content of the target video, and the feature information of the target video in the image mode, text mode and audio mode is combined, which can improve the comprehensiveness and diversity of the feature information of the video splicing feature. Furthermore, the video splicing feature is decoded in time sequence by using a text decoder based on an autoregressive form, and the embedded features of the last decoded description word are used as the input of the next decoding process, which can improve the continuity of each description word in time sequence and the semantic dependency. This method improves the coherence and accuracy of the text content of the video description text spliced ​​by multiple description words.

[0235] See also Figure 8 The embodiment of the present application further provides a video description generating device, which can implement the above-mentioned video description generating method, and the device includes:

[0236] An acquisition unit 801 is used to acquire a sample video;

[0237] An extraction unit 802 is used to extract sample video image content, sample video audio content, and sample video subtitle content of the sample video in the sample video;

[0238] The encoding unit 803 is used to perform feature encoding on the sample video image content based on a preset image mode encoder to obtain sample image features, perform feature encoding on the sample video audio content based on a preset audio mode encoder to obtain sample audio features, and perform feature encoding on the sample video subtitle content based on a preset text mode encoder to obtain sample subtitle features;

[0239] A calculation unit 804 is used to perform similarity calculation based on sample image features, sample audio features, and sample subtitle features to obtain feature similarity data;

[0240] A training unit 805 is used to jointly train an image modality encoder, an audio modality encoder, and a text modality encoder based on feature similarity data to obtain a video description generation model;

[0241] The generating unit 806 is used to input the video image, video subtitle content and video audio content of the target video into the video description generating model to generate the video description, so as to obtain the video description text of the target video.

[0242] In some embodiments, the extraction unit 802 is used to:

[0243] Extracting video frames from the sample video to obtain multiple video frame images;

[0244] Performing image clustering on a plurality of video frame images to obtain video frame image classes and cluster center images of the video frame image classes, wherein there are at least two video frame image classes;

[0245] The cluster center images are flattened in time sequence to obtain sample video image content.

[0246] In some embodiments, performing image clustering on a plurality of video frame images to obtain video frame image classes and cluster center images of the video frame image classes includes:

[0247] For each video frame image, obtaining a video frame image fingerprint feature of the video frame image;

[0248] Based on the video frame image fingerprint features, the similarity between every two video frame images is calculated to obtain the image fingerprint similarity;

[0249] Performing image clustering on multiple video frame images based on image fingerprint similarity to obtain a video frame image class, so that the image fingerprint similarity of any two video frame images in the video frame image class is greater than a preset threshold;

[0250] For any video frame image in the video image class, based on the image fingerprint similarity between the video frame image and other video frame images in the video image class, determine the overall similarity of the video frame image;

[0251] Based on the overall similarity, a cluster center image is determined among the multiple video frame images of the video image class.

[0252] In some embodiments, the extraction unit 802 is used to:

[0253] Determine the video length of the sample video;

[0254] Based on the video length, the pitch features of the sample video are extracted to obtain the sample video audio content, wherein the audio length of the sample video audio content is equal to the video length.

[0255] In some embodiments, the extraction unit 802 is used to:

[0256] Performing audio content recognition on the sample video to obtain a first video text;

[0257] Extracting subtitle features from the sample video to obtain a second video text;

[0258] The first video text and the second video text are integrated into sample video subtitle content.

[0259] In some embodiments, the computing unit 804 is configured to:

[0260] Performing cross-similarity calculation based on sample image features and sample audio features to obtain first similarity data;

[0261] Perform cross-similarity calculation based on sample image features and sample subtitle features to obtain second similarity data;

[0262] Performing cross-similarity calculation based on the sample audio features and the sample subtitle features to obtain third similarity data;

[0263] The first similar data, the second similar data and the third similar data are integrated to obtain feature similar data.

[0264] The sample image features include global image features and local image features, and the sample audio features include global audio features and local audio features;

[0265] The cross-similarity calculation is performed based on the sample image feature and the sample audio feature to obtain the first similarity data, including:

[0266] Calculating a first similarity between the global image feature and the global audio feature, calculating a second similarity between the global image feature and the local audio feature, calculating a third similarity between the local image feature and the global audio feature, and calculating a fourth similarity between the local image feature and the local audio feature;

[0267] Performing a comparison calculation based on the third similarity to obtain a first comparison similarity, and performing a comparison calculation based on the second similarity to obtain a second comparison similarity;

[0268] Performing a comparison calculation based on the fourth similarity to obtain a third comparison similarity;

[0269] The third similarity, the first comparison similarity, the second comparison similarity, and the third comparison similarity are averaged to obtain first similarity data.

[0270] In some embodiments, the feature similarity data includes first similarity data, second similarity data, and third similarity data;

[0271] The training unit 805 is used to:

[0272] Performing loss calculation based on the first similar data to obtain a first loss value, performing loss calculation based on the second similar data to obtain a second loss value, and performing loss calculation based on the third similar data to obtain a third loss value;

[0273] An image modality encoder is trained based on the first loss value and the third loss value, an audio modality encoder is trained based on the first loss value and the second loss value, and a text modality encoder is trained based on the second loss value and the third loss value to obtain a video description generation model.

[0274] In some embodiments, the video description generation model includes an image modality encoder, an audio modality encoder, a text modality encoder, and a text decoder;

[0275] The generating unit 806 is used for:

[0276] Based on the image modality encoder, feature encoding is performed on the video image to obtain the video image feature; based on the audio modality encoder, feature encoding is performed on the video audio content to obtain the video audio feature; based on the text modality encoder, feature encoding is performed on the video subtitle content to obtain the video subtitle feature;

[0277] Performing splicing processing on video image features, video audio features, and video subtitle features to obtain video splicing features;

[0278] The video splicing features are decoded based on the text decoder to obtain the video description text.

[0279] In some embodiments, feature decoding is performed on the video splicing feature based on a text decoder to obtain a video description text, including:

[0280] Decode the video splicing features based on the text decoder to obtain the first description word;

[0281] Initialize the target vector to the word embedding feature corresponding to the first description word;

[0282] Decode the target vector based on the text decoder to obtain the next description word;

[0283] The target vector is updated with the word embedding feature of the next description word, and the target vector is decoded based on the text decoder to obtain the step of the next description word until a stop symbol is generated, and the obtained multiple description words are concatenated into a video description text.

[0284] The specific implementation of the video description generating device is substantially the same as the specific implementation of the above-mentioned video description generating method, and will not be described in detail here.

[0285] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned video description generation method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0286] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0287] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0288] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the video description generation method of the embodiment of this application;

[0289] Input / output interface 903, used to implement information input and output;

[0290] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0291] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0292] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0293] The embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the above-mentioned video description generation method.

[0294] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0295] The video description generation method, video description generation device, electronic device and computer-readable storage medium provided in the embodiments of the present application obtain a sample video; in the sample video, extract the sample video image content, sample video audio content, and sample video subtitle content of the sample video, so as to realize the extraction of content information of multiple modes of the sample video, which is conducive to improving the comprehensiveness of feature extraction. Further, based on the preset image modality encoder, the sample video image content is feature encoded to obtain sample image features, based on the preset audio modality encoder, the sample video audio content is feature encoded to obtain sample audio features, and based on the preset text modality encoder, the sample video subtitle content is feature encoded to obtain sample subtitle features, so as to realize the unified encoding of content information of multiple modes of the sample video, and convert the feature content of the image modality, audio modality and text modality into the same feature dimension. Furthermore, similarity calculation is performed based on sample image features, sample audio features, and sample subtitle features to obtain feature similarity data, and based on the feature similarity data, the image modality encoder, audio modality encoder, and text modality encoder are jointly trained to obtain a video description generation model. This method realizes cross-similarity calculation of feature content of multiple modalities, and takes into account feature alignment of feature content of multiple modalities, so that the video description generation model can learn more about the correlation of feature information of different modalities during training, which is conducive to training a video description generation model with better model performance. Finally, the video image, video subtitle content, and video audio content of the target video are input into the video description generation model for video description generation to obtain the video description text of the target video, which can improve the accuracy and efficiency of the video description text of the generated video.

[0296] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0297] It can be understood by those skilled in the art that Figure 1-7 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0298] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0299] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0300] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0301] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0302] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0303] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0304] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0305] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0306] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A video description generation method, characterized in that: The method comprises: Get sample video; Extracting sample video image content, sample video audio content, and sample video subtitle content of the sample video from the sample video; Based on a preset image modality encoder, feature encoding is performed on the sample video image content to obtain sample image features, based on a preset audio modality encoder, feature encoding is performed on the sample video audio content to obtain sample audio features, and based on a preset text modality encoder, feature encoding is performed on the sample video subtitle content to obtain sample subtitle features; Calculating similarity based on the sample image features, the sample audio features, and the sample subtitle features to obtain feature similarity data; Based on the feature similarity data, jointly training the image modality encoder, the audio modality encoder, and the text modality encoder to obtain a video description generation model; The video image, video subtitle content and video audio content of the target video are input into the video description generation model to generate a video description, thereby obtaining a video description text of the target video.

2. The method according to claim 1, characterized in that Extracting sample video image content of the sample video in the sample video includes: Extracting video frames from the sample video to obtain multiple video frame images; Performing image clustering on the multiple video frame images to obtain video frame image classes and cluster center images of the video frame image classes, wherein there are at least two video frame image classes; The cluster center image is flattened in time sequence to obtain the sample video image content.

3. The method according to claim 2, characterized in that The performing image clustering on the plurality of video frame images to obtain a video frame image class and a cluster center image of the video frame image class includes: For each of the video frame images, obtaining a video frame image fingerprint feature of the video frame image; Based on the video frame image fingerprint features, similarity calculation is performed on every two video frame images to obtain image fingerprint similarity; Performing image clustering on the plurality of video frame images based on the image fingerprint similarity to obtain the video frame image class, so that the image fingerprint similarity of any two of the video frame images in the video frame image class is greater than a preset threshold; For any video frame image in the video image class, determining the overall similarity of the video frame image based on the image fingerprint similarity between the video frame image and other video frame images in the video image class; Based on the overall similarity, the cluster center image is determined among the multiple video frame images of the video image class.

4. The method according to claim 1, characterized in that: The similarity calculation based on the sample image feature, the sample audio feature, and the sample subtitle feature to obtain feature similarity data includes: Performing cross-similarity calculation based on the sample image feature and the sample audio feature to obtain first similarity data; Performing cross-similarity calculation based on the sample image features and the sample subtitle features to obtain second similarity data; Performing cross-similarity calculation based on the sample audio feature and the sample subtitle feature to obtain third similarity data; The first similarity data, the second similarity data and the third similarity data are integrated to obtain the feature similarity data.

5. The method according to claim 4, characterized in that The sample image features include global image features and local image features, and the sample audio features include global audio features and local audio features; The performing cross-similarity calculation based on the sample image feature and the sample audio feature to obtain first similarity data includes: Calculating a first similarity between the global image feature and the global audio feature, calculating a second similarity between the global image feature and the local audio feature, calculating a third similarity between the local image feature and the global audio feature, and calculating a fourth similarity between the local image feature and the local audio feature; Performing a comparison calculation based on the third similarity to obtain a first comparison similarity, and performing a comparison calculation based on the second similarity to obtain a second comparison similarity; Performing a comparison calculation based on the fourth similarity to obtain a third comparison similarity; The third similarity, the first comparison similarity, the second comparison similarity, and the third comparison similarity are averaged to obtain the first similarity data.

6. The method according to claim 1, characterized in that The feature similarity data includes first similarity data, second similarity data and third similarity data; The step of jointly training the image modality encoder, the audio modality encoder, and the text modality encoder based on the feature similarity data to obtain a video description generation model includes: Performing loss calculation based on the first similar data to obtain a first loss value, performing loss calculation based on the second similar data to obtain a second loss value, and performing loss calculation based on the third similar data to obtain a third loss value; The image modality encoder is trained based on the first loss value and the third loss value, the audio modality encoder is trained based on the first loss value and the second loss value, and the text modality encoder is trained based on the second loss value and the third loss value to obtain the video description generation model.

7. The method according to any one of claims 1 to 6, characterized in that: The video description generation model includes an image modality encoder, an audio modality encoder, a text modality encoder, and a text decoder; The step of inputting the video image, video subtitle content, and video audio content of the target video into the video description generation model to generate a video description to obtain a video description text of the target video includes: Based on the image modality encoder, feature encoding is performed on the video image to obtain video image features; based on the audio modality encoder, feature encoding is performed on the video audio content to obtain video audio features; based on the text modality encoder, feature encoding is performed on the video subtitle content to obtain video subtitle features; Performing splicing processing on the video image feature, the video audio feature, and the video subtitle feature to obtain a video splicing feature; The video splicing features are decoded based on the text decoder to obtain the video description text.

8. A video description generating device, characterized in that: The video description generating device comprises: An acquisition unit, used for acquiring a sample video; An extraction unit, configured to extract, from the sample video, sample video image content, sample video audio content, and sample video subtitle content of the sample video; an encoding unit, configured to perform feature encoding on the sample video image content based on a preset image mode encoder to obtain sample image features, perform feature encoding on the sample video audio content based on a preset audio mode encoder to obtain sample audio features, and perform feature encoding on the sample video subtitle content based on a preset text mode encoder to obtain sample subtitle features; A calculation unit, configured to perform similarity calculation based on the sample image feature, the sample audio feature, and the sample subtitle feature to obtain feature similarity data; A training unit, configured to jointly train the image modality encoder, the audio modality encoder, and the text modality encoder based on the feature similarity data to obtain a video description generation model; The generating unit is used to input the video image, video subtitle content and video audio content of the target video into the video description generating model to generate the video description, so as to obtain the video description text of the target video.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the video description generating method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the video description generating method according to any one of claims 1 to 7 is implemented.

Images