Video description generation method and device, electronic device, and storage medium

Abstract

The application provides a video description generation method and device, electronic equipment and a storage medium, and belongs to the technical field of financial technology. The method comprises the following steps: encoding sample video image content of a sample video into sample image features based on an image modal encoder, encoding sample video audio content of the sample video into sample audio features based on an audio modal encoder, and encoding sample video subtitle content of the sample video into sample subtitle features based on a text modal encoder; calculating feature similarity data based on the sample image features, the sample audio features and the sample subtitle features; jointly training the image modal encoder, the audio modal encoder and the text modal encoder based on the feature similarity data to obtain a video description generation model; and inputting video image content, video subtitle content and video audio content of a target video into the video description generation model to generate video description text. The application can improve the generation accuracy of video description.

Description

Technical Field

[0001] This application relates to the field of video processing technology, and is applicable to the financial technology field, and particularly to a video description generation method and apparatus, electronic device and storage medium. Background Technology

[0002] In the field of video content processing, especially in the insurance industry, when sales personnel need to obtain video descriptions for insurance videos, they often need to watch all the video content from beginning to end (background introduction, business introduction, product introduction, etc.) and manually summarize the video content to generate corresponding insurance copy (e.g., promotional or product copy). This method often suffers from low efficiency in generating insurance descriptions and the inability of the insurance descriptions to accurately reflect the main content of the insurance video.

[0003] In response, related technologies have considered using image description generation technology. This involves extracting frames from the insurance video to generate video frame images, and then using an image encoder to extract semantics from the video frame images to generate video description text corresponding to the insurance video. While this method improves the efficiency of insurance description text generation to some extent, the generated insurance description text may still be inaccurate because the image encoder cannot fully understand the semantic feature information of the video frame images. Summary of the Invention

[0004] The main objective of this application is to provide a video description generation method, apparatus, electronic device, and storage medium, aiming to improve the accuracy of video description generation.

[0005] To achieve the above objectives, a first aspect of this application proposes a video description generation method, the method comprising:

[0006] Obtain sample videos;

[0007] In the sample video, the sample video image content, sample video audio content, and sample video subtitle content are extracted;

[0008] The sample video image content is feature-encoded based on a preset image modal encoder to obtain sample image features; the sample video audio content is feature-encoded based on a preset audio modal encoder to obtain sample audio features; and the sample video subtitle content is feature-encoded based on a preset text modal encoder to obtain sample subtitle features.

[0009] Similarity calculations are performed based on the sample image features, the sample audio features, and the sample subtitle features to obtain feature similarity data;

[0010] Based on the feature similarity data, the image modal encoder, the audio modal encoder, and the text modal encoder are jointly trained to obtain a video description generation model;

[0011] The video image, video subtitles, and video audio of the target video are input into the video description generation model to generate a video description, resulting in the video description text of the target video.

[0012] In some embodiments, extracting sample video image content from the sample video includes:

[0013] The sample video is subjected to video frame extraction to obtain multiple video frame images;

[0014] The plurality of video frame images are subjected to image clustering to obtain video frame image classes and cluster center images of the video frame image classes, wherein there are at least two video frame image classes;

[0015] The cluster center image is flattened in time sequence to obtain the sample video image content.

[0016] In some embodiments, the step of performing image clustering on the plurality of video frame images to obtain video frame image classes and cluster center images of the video frame image classes includes:

[0017] For each of the video frame images, obtain the video frame image fingerprint feature of the video frame image;

[0018] Based on the image fingerprint features of the video frames, the similarity between every two video frames is calculated to obtain the image fingerprint similarity.

[0019] Based on the image fingerprint similarity, the multiple video frame images are clustered to obtain the video frame image class, so that the image fingerprint similarity of any two video frame images in the video frame image class is greater than a preset threshold.

[0020] For any video frame image in the video image class, the overall similarity of the video frame image is determined based on the image fingerprint similarity between the video frame image and other video frame images in the video image class;

[0021] Based on the overall similarity, the cluster center image is determined among the plurality of video frame images of the video image class.

[0022] In some embodiments, the similarity calculation based on the sample image features, the sample audio features, and the sample subtitle features to obtain feature similarity data includes:

[0023] Based on the sample image features and the sample audio features, cross-similarity calculation is performed to obtain the first similarity data;

[0024] Based on the sample image features and the sample caption features, cross-similarity calculation is performed to obtain the second similarity data;

[0025] Based on the sample audio features and the sample subtitle features, a cross-similarity calculation is performed to obtain the third similarity data;

[0026] The first similar data, the second similar data, and the third similar data are integrated to obtain the feature similar data.

[0027] In some embodiments, the sample image features include global image features and local image features, and the sample audio features include global audio features and local audio features;

[0028] The first similarity data is obtained by performing cross-similarity calculation based on the sample image features and the sample audio features, including:

[0029] Calculate the first similarity between the global image features and the global audio features, calculate the second similarity between the global image features and the local audio features, calculate the third similarity between the local image features and the global audio features, and calculate the fourth similarity between the local image features and the local audio features;

[0030] A comparison calculation is performed based on the third similarity to obtain a first comparison similarity, and a comparison calculation is performed based on the second similarity to obtain a second comparison similarity;

[0031] Based on the fourth similarity, a comparison calculation is performed to obtain the third comparison similarity.

[0032] The first similarity data is obtained by averaging the third similarity, the first comparative similarity, the second comparative similarity, and the third comparative similarity.

[0033] In some embodiments, the feature similarity data includes first similarity data, second similarity data, and third similarity data;

[0034] The video description generation model is obtained by jointly training the image modal encoder, the audio modal encoder, and the text modal encoder based on the feature similarity data, including:

[0035] A first loss value is obtained by performing loss calculation based on the first similar data, a second loss value is obtained by performing loss calculation based on the second similar data, and a third loss value is obtained by performing loss calculation based on the third similar data.

[0036] The image modal encoder is trained based on the first loss value and the third loss value, the audio modal encoder is trained based on the first loss value and the second loss value, and the text modal encoder is trained based on the second loss value and the third loss value to obtain the video description generation model.

[0037] In some embodiments, the video description generation model includes an image modal encoder, an audio modal encoder, a text modal encoder, and a text decoder;

[0038] The step of inputting the video image, video subtitle content, and video audio content of the target video into the video description generation model to generate a video description and obtain the video description text of the target video includes:

[0039] The video image is feature-encoded using the image modality encoder to obtain video image features; the video audio content is feature-encoded using the audio modality encoder to obtain video audio features; and the video subtitle content is feature-encoded using the text modality encoder to obtain video subtitle features.

[0040] The video image features, the video audio features, and the video subtitle features are spliced ​​together to obtain video splicing features;

[0041] Based on the text decoder, the video splicing features are decoded to obtain the video description text.

[0042] In some embodiments, extracting the sample video audio content from the sample video includes:

[0043] Determine the video length of the sample video;

[0044] Based on the video length, pitch features are extracted from the sample video to obtain the audio content of the sample video, wherein the audio length of the audio content of the sample video is equal to the video length.

[0045] In some embodiments, extracting the sample video subtitle content from the sample video includes:

[0046] The sample video is subjected to audio content recognition to obtain the first video text;

[0047] Subtitle features are extracted from the sample video to obtain the second video text;

[0048] The first video text and the second video text are integrated into the sample video subtitle content.

[0049] In some embodiments, the step of performing feature decoding on the video splicing features based on the text decoder to obtain the video description text includes:

[0050] The video splicing features are decoded based on the text decoder to obtain the first descriptive word;

[0051] The target vector is initialized with the word embedding feature corresponding to the first descriptive word;

[0052] The target vector is decoded based on the text decoder to obtain the next descriptive word;

[0053] The target vector is updated with the word embedding features of the next descriptor, and the step of decoding the target vector based on the text decoder to obtain the next descriptor is returned until a stop character is generated. The multiple descriptors obtained are then concatenated into the video description text.

[0054] To achieve the above objectives, a second aspect of this application provides a video description generation apparatus, the apparatus comprising:

[0055] The acquisition unit is used to acquire sample videos;

[0056] The extraction unit is used to extract sample video image content, sample video audio content, and sample video subtitle content from the sample video.

[0057] The encoding unit is used to perform feature encoding on the sample video image content based on a preset image modality encoder to obtain sample image features, to perform feature encoding on the sample video audio content based on a preset audio modality encoder to obtain sample audio features, and to perform feature encoding on the sample video subtitle content based on a preset text modality encoder to obtain sample subtitle features.

[0058] The calculation unit is used to perform similarity calculation based on the sample image features, the sample audio features, and the sample subtitle features to obtain feature similarity data;

[0059] The training unit is used to jointly train the image modal encoder, the audio modal encoder, and the text modal encoder based on the feature similarity data to obtain a video description generation model;

[0060] The generation unit is used to input the video image, video subtitle content and video audio content of the target video into the video description generation model to generate a video description and obtain the video description text of the target video.

[0061] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the video description generation method described in the first aspect.

[0062] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the video description generation method described in the first aspect.

[0063] The video description generation method, apparatus, electronic device, and storage medium proposed in this application acquire sample videos; from the sample videos, they extract sample video image content, sample video audio content, and sample video subtitle content, enabling the extraction of content information from multiple modalities of the sample video, which is beneficial to improving the comprehensiveness of feature extraction. Furthermore, based on a preset image modality encoder, feature encoding is performed on the sample video image content to obtain sample image features; based on a preset audio modality encoder, feature encoding is performed on the sample video audio content to obtain sample audio features; and based on a preset text modality encoder, feature encoding is performed on the sample video subtitle content to obtain sample subtitle features. This enables unified encoding of content information from multiple modalities of the sample video, converting the feature content of image modality, audio modality, and text modality into a single feature dimension. Furthermore, similarity calculations are performed based on sample image features, sample audio features, and sample caption features to obtain feature similarity data. Based on this feature similarity data, the image modality encoder, audio modality encoder, and text modality encoder are jointly trained to obtain a video description generation model. This approach achieves cross-similarity calculation of feature content across multiple modalities and considers feature alignment across these modalities. This allows the video description generation model to learn more about the correlation of feature information from different modalities during training, which is beneficial for training a video description generation model with better performance. Finally, the video image, video caption content, and video audio content of the target video are input into the video description generation model to generate video description text for the target video, improving the accuracy and efficiency of the generated video description text. Attached Figure Description

[0064] Figure 1 This is a flowchart of the video description generation method provided in the embodiments of this application;

[0065] Figure 2 yes Figure 1 The flowchart of step S102 in the document;

[0066] Figure 3 yes Figure 1 The flowchart of step S102 in the document;

[0067] Figure 4 yes Figure 1 The flowchart of step S104 in the process;

[0068] Figure 5 yes Figure 4 The flowchart of step S401 in the text;

[0069] Figure 6 yes Figure 1 The flowchart of step S105 in the process;

[0070] Figure 7 yes Figure 1 The flowchart of step S106 in the process;

[0071] Figure 8 This is a schematic diagram of the structure of the video description generation device provided in the embodiments of this application;

[0072] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0074] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0076] The video description generation method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the video description generation method in this application is described.

[0077] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0078] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0079] The video description generation method provided in this application relates to the fields of financial technology and artificial intelligence technology. The video description generation method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the video description generation method, but is not limited to the above forms.

[0080] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0081] It should be noted that in various specific embodiments of this application, when processing data related to the identity or characteristics of an object, such as object information, object behavior data, object historical data, and object location information, the object's permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require obtaining the object's personal information, separate permission or consent from the object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the object's separate permission or consent will the necessary object-related data for the proper functioning of the embodiments of this application be obtained.

[0082] Figure 1 This is an optional flowchart of the video description generation method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S106.

[0083] Step S101: Obtain sample video;

[0084] Step S102: Extract the sample video image content, sample video audio content, and sample video subtitle content from the sample video.

[0085] Step S103: Based on a preset image modal encoder, feature encoding is performed on the sample video image content to obtain sample image features; based on a preset audio modal encoder, feature encoding is performed on the sample video audio content to obtain sample audio features; and based on a preset text modal encoder, feature encoding is performed on the sample video subtitle content to obtain sample subtitle features.

[0086] Step S104: Calculate similarity based on sample image features, sample audio features, and sample subtitle features to obtain feature similarity data;

[0087] Step S105: Based on feature similarity data, jointly train the image modal encoder, audio modal encoder, and text modal encoder to obtain the video description generation model;

[0088] Step S106: Input the video image, video subtitle content, and video audio content of the target video into the video description generation model to generate a video description and obtain the video description text of the target video.

[0089] Steps S101 to S106, as illustrated in this embodiment, involve acquiring a sample video. From the sample video, the sample video image content, sample video audio content, and sample video subtitle content are extracted. This enables the extraction of content information from multiple modalities of the sample video, improving the comprehensiveness of feature extraction. Furthermore, based on a preset image modality encoder, feature encoding is performed on the sample video image content to obtain sample image features. Based on a preset audio modality encoder, feature encoding is performed on the sample video audio content to obtain sample audio features. Finally, based on a preset text modality encoder, feature encoding is performed on the sample video subtitle content to obtain sample subtitle features. This enables unified encoding of content information from multiple modalities of the sample video, converting the feature content of image modality, audio modality, and text modality into the same feature dimension. Furthermore, similarity calculations are performed based on sample image features, sample audio features, and sample caption features to obtain feature similarity data. Based on this feature similarity data, the image modality encoder, audio modality encoder, and text modality encoder are jointly trained to obtain a video description generation model. This approach achieves cross-similarity calculation of feature content across multiple modalities and considers feature alignment across these modalities. This allows the video description generation model to learn more about the correlation of feature information from different modalities during training, which is beneficial for training a video description generation model with better performance. Finally, the video image, video caption content, and video audio content of the target video are input into the video description generation model to generate video description text for the target video, improving the accuracy and efficiency of generating video description text for risky videos.

[0090] In step S101 of some embodiments, a sample video is acquired.

[0091] Sample videos refer to insurance video data that includes background information on the insurance industry, business introductions, product introductions, and insurance sales script training.

[0092] In the specific implementation of this embodiment, when training the video description generation model, with authorization, insurance-related video materials can be obtained from various insurance business platforms or public websites. Video materials containing background introductions, business introductions, product introductions, and various insurance-related popular science videos can be used as sample videos.

[0093] In step S102 of some embodiments, sample video image content, sample video audio content, and sample video subtitle content are extracted from the sample video.

[0094] The sample video image content is used to indicate the image information contained in the sample video.

[0095] The sample video audio content is used to indicate the audio information contained in the sample video.

[0096] The sample video caption content is used to indicate the caption information contained in the sample video.

[0097] Please see Figure 2 In some embodiments, the specific process of extracting the sample video image content from the sample video may include, but is not limited to, steps S201 to S203:

[0098] Step S201: Extract video frames from the sample video to obtain multiple video frame images;

[0099] Step S202: Perform image clustering on multiple video frame images to obtain video frame image classes and cluster center images of video frame image classes;

[0100] Step S203: Flatten the cluster center image in time sequence to obtain the sample video image content.

[0101] Steps S201 to S203 are described in detail below.

[0102] In step S201 of some embodiments, the video frame image refers to the video image corresponding to each frame in the sample video.

[0103] In the specific implementation of this embodiment, a video frame extraction tool or video editing software can be used to extract video frames from the sample video, dividing the sample video into video images frame by frame in time sequence, thereby obtaining multiple video frame images.

[0104] In step S202 of some embodiments, a video frame image class refers to a large class composed of multiple video frame images with relatively similar video content. There are at least two video frame image classes. The cluster center image refers to a video frame image within a video frame image class that has image features that are relatively similar to all the other video frame images.

[0105] In a specific implementation of this embodiment, step S202 includes, but is not limited to, the following steps:

[0106] For each video frame image, obtain the video frame image fingerprint feature of the video frame image;

[0107] Based on the fingerprint features of video frame images, the similarity between every two video frame images is calculated to obtain the image fingerprint similarity.

[0108] Based on image fingerprint similarity, multiple video frame images are clustered to obtain video frame image classes, so that the image fingerprint similarity between any two video frame images in the video frame image class is greater than a preset threshold.

[0109] For any video frame image in the video image class, the overall similarity of the video frame image is determined based on the image fingerprint similarity between the video frame image and other video frame images in the video image class;

[0110] Based on overall similarity, cluster center images are determined among multiple video frame images of the video image class.

[0111] In this context, video frame image fingerprint features refer to the feature information that can be used to uniquely identify or distinguish video frame images. Image fingerprint similarity indicates the degree of similarity between the image features of two video frame images. A preset threshold is used to measure whether two video frame images can be classified into the same broad category. Overall similarity indicates the overall similarity of the image features of a video frame image with those of other video frame images in its class.

[0112] Specifically, for each video frame image, firstly, feature extraction is performed on the video frame image to obtain its image feature information, resulting in a video frame image fingerprint feature. Next, for each video frame image, the cosine similarity is used to calculate the similarity between the video frame image fingerprint feature of that video frame image and the video frame image fingerprint features of other video frames, obtaining the image fingerprint similarity between each pair of video frames. Further, based on the image fingerprint similarity between each video frame image and other video frames, image clustering is performed on multiple video frame images, dividing them into at least two video frame image classes. This ensures that the image fingerprint similarity between any two video frames within the same video frame image class is greater than a preset threshold, and that the image fingerprint similarity between any two video frames in different video frame image classes is less than or equal to a preset threshold. Finally, for each video frame image within a video frame image class, the image fingerprint similarity between that video frame image and other video frames within that class is summed to obtain the overall similarity of that video frame image. Furthermore, within the video frame image class, the video frame image with the highest overall similarity is determined as the cluster center image of the video image class.

[0113] In step S203 of some embodiments, firstly, the video frame time points of each cluster center image are obtained. Then, based on the video frame time points, the cluster center images are flattened in time sequence, and multiple cluster center images are sequentially stitched together according to the video frame time points from earliest to latest to obtain the sample video image content.

[0114] The advantage of this embodiment is that, when extracting sample video image content from sample videos, the sample video is first segmented into multiple video frame images. Using video frame images as the basic unit, image clustering is achieved based on the similarity of the fingerprint features of every two video frame images, grouping video frame images with similar image fingerprint feature information into the same major category. Since the cluster center images of each video frame image category can represent the key salient information of the storyline in the sample video, the cluster center images of each video frame image category are used as the basic images selected to construct the sample video image content. The cluster center images are then stitched together to form the sample video image content. This method can capture fewer video frame images corresponding to significant scenes with significant changes to represent the image content of the entire sample video while retaining fewer static video frames and more dynamic scene frames, thereby improving the accuracy and conciseness of the extracted sample video image content.

[0115] Please see Figure 3 In some embodiments, the specific process of extracting the sample video audio content from the sample video may include, but is not limited to, steps S301 to S302:

[0116] Step S301: Determine the video length of the sample video;

[0117] Step S302: Based on the video length, extract the pitch features of the sample video to obtain the audio content of the sample video.

[0118] Steps S301 to S302 are described in detail below.

[0119] In step S301 of some embodiments, the video length is used to indicate the total duration of the sample video.

[0120] In the specific implementation of this embodiment, a timer or other time counting tool can be used to collect the total video duration of the sample video to obtain the video length of the sample video.

[0121] In step S302 of some embodiments, the audio length of the sample video audio content is equal to the video length.

[0122] In this specific implementation, the multimedia framework FFmpeg can be used to extract audio information from the sample video. Specifically, the multimedia framework is invoked, and the input file is specified as the sample video FFmpeg during instruction parameter settings. The settings are configured to not retain the video stream of the sample video, but to copy the audio stream, thereby obtaining preliminary video and audio content. The audio length of this preliminary video and audio content is equal to the video length. Furthermore, the audio source separation tool Bytesep can be used to decouple the pitch data in the preliminary video and audio content from the mixed audio containing various sounds. The result of decoupling the sound can be considered as audio data sharing the content plot with the video segments in the sample video without any language information. Based on this, the result of decoupling the sound is used as the audio content of the sample video.

[0123] It should be noted that the audio of the sample video is separated into speech because the video subtitles already contain speech information, so the separated speech does not need speech information, only the pitch intensity information needs to be retained.

[0124] The advantage of this embodiment is that, considering the significant overlap between the language information in the audio content and the subtitle information in the sample video, when extracting the sample audio content of the sample video, the audio stream of the sample video is first obtained using a multimedia framework, and then a speech separation operation is performed on the obtained audio stream to separate the part of the audio stream that does not contain language information but contains tone information as the sample video audio content. This method can improve the accuracy and conciseness of the extracted sample video audio content.

[0125] In some embodiments, the specific process of extracting sample video captions from sample videos may include, but is not limited to, the following steps:

[0126] Audio content recognition is performed on the sample video to obtain the first video text;

[0127] Subtitle features are extracted from the sample video to obtain the second video text;

[0128] The first video text and the second video text are integrated into sample video subtitle content.

[0129] The first video text indicates the text content generated based on the audio information of the sample video. The second video text indicates the text content generated based on the subtitle information of the sample video.

[0130] In this specific implementation, firstly, when recognizing the audio content of the sample video, Automatic Speech Recognition (ASR) technology can be used to convert the audio content in the sample video into text, obtaining the first video text. Simultaneously, when extracting subtitle features from the sample video to obtain the second video text, Optical Character Recognition (OCR) technology is used to recognize text in each video frame of the sample video to extract the text content, obtaining the second video text. Finally, based on the first and second video texts, the text content of the first and second video texts is integrated to obtain the sample video subtitle content.

[0131] It should be noted that if the text content in the sample video is dynamic or synchronized with the audio, it is easier to extract more accurate sample video subtitles using Automatic Speech Recognition (ASR) technology; if the text content in the sample video is static or presented in image form, it is easier to extract more accurate sample video subtitles using Optical Character Recognition (OCR) technology.

[0132] Furthermore, when the sample video contains human voices or subtitles, the aforementioned Automatic Speech Recognition (ASR) or Optical Character Recognition (OCR) technologies can be directly used. However, when the sample video lacks human voices and subtitles, a fusion of command detection and a Long Short-Term Memory (LSTM) deep convolutional network is required to generate the sample video subtitles. Specifically, firstly, command detection technology is used to extract command features from the audio of the sample video. Simultaneously, an LSM deep convolutional network is used to extract features from the sample video, capturing its depth features to obtain video depth features. Next, the command features and video depth features are directly added or multiplied to achieve feature fusion, resulting in video fused features. Further, the video fused features are input into a text generation model to generate text information, which is then used as the sample video subtitles. This text generation model can be an LSTM model or a transformer model, etc.

[0133] The advantage of this embodiment is that, when extracting sample video subtitles, it takes into account whether the sample video contains human voices and subtitles. It provides multiple methods for generating sample video subtitles, such as automatic speech recognition technology, optical character recognition technology, or a combination of command detection and long short-term memory deep convolutional networks. This can improve the universality of the generated sample video subtitles and meet the needs of extracting sample video subtitles in various situations.

[0134] In step S103 of some embodiments, the sample video image content is feature-encoded based on a preset image modal encoder to obtain sample image features, the sample video audio content is feature-encoded based on a preset audio modal encoder to obtain sample audio features, and the sample video subtitle content is feature-encoded based on a preset text modal encoder to obtain sample subtitle features.

[0135] An image modal encoder is an encoder that converts image data into feature representations that a model can understand. Sample image features are used to indicate the image feature information contained in each video frame of the sample video image content. An audio modal encoder is an encoder that converts audio data into feature representations that a model can understand. Sample audio features are used to indicate the audio feature information contained in each audio segment of the sample video audio content. A text modal encoder is an encoder that converts text data into feature representations that a model can understand. Sample caption features are used to indicate the feature information reflected in the sample video caption content.

[0136] In this specific implementation, firstly, an image modality encoder is used to perform feature encoding on the sample video image content, mapping each video frame in the sample video image content to a latent vector space with a fixed feature dimension, thus obtaining the sample image features corresponding to the sample video image content. Next, an audio modality encoder is used to perform feature encoding on the sample video audio content, mapping each audio segment in the sample video audio content to a latent vector space with a fixed feature dimension, thus obtaining the sample audio features corresponding to the sample video audio content. Finally, a text modality encoder is used to perform feature encoding on the sample video subtitle content, mapping the sample video subtitle content to a latent vector space with a fixed feature dimension, thus obtaining the sample subtitle features corresponding to the sample video subtitle content.

[0137] It should be noted that the sample image features, sample audio features, and sample subtitle features have the same feature dimension. Further, the sample image features include the individual image features (local image features ff) of each video frame in the sample video image content, and the overall image features (global image features fv) obtained by average pooling the individual image features of each video frame. The sample audio features include the individual audio features (local audio features fas) of each audio segment in the sample video audio content, and the overall audio features (global audio features fa) obtained by average pooling the individual audio features of each audio segment. The sample subtitle features include the individual word features (local subtitle features fw) of each word in the sample video subtitle content, and the overall subtitle features (global subtitle features fs) obtained by average pooling the individual word features of each word.

[0138] In step S104 of some embodiments, similarity calculation is performed based on sample image features, sample audio features, and sample subtitle features to obtain feature similarity data.

[0139] Please see Figure 4 In some embodiments, step S104 may include, but is not limited to, steps S401 to S404:

[0140] Step S401: Calculate the cross-similarity based on the sample image features and sample audio features to obtain the first similarity data;

[0141] Step S402: Calculate the cross-similarity based on the sample image features and sample caption features to obtain the second similarity data;

[0142] Step S403: Calculate the cross-similarity based on the sample audio features and sample subtitle features to obtain the third similarity data.

[0143] Step S404: Integrate the first similar data, the second similar data, and the third similar data to obtain feature similar data.

[0144] Steps S401 to S404 are described in detail below.

[0145] In step S401 of some embodiments, the first similarity data is used to indicate the degree of similarity between sample image features and sample audio features.

[0146] To save space, the specific process of cross-similarity calculation based on sample image features and sample audio features in this application embodiment will be described in detail below, and will not be repeated here.

[0147] In step S402 of some embodiments, the second similarity data is used to indicate the degree of similarity between sample image features and sample caption features.

[0148] In the specific implementation of this embodiment, the specific process of step S402 is similar to that of step S401 described above. To save space, it will not be described again.

[0149] In step S403 of some embodiments, third similarity data is used to indicate the degree of similarity between sample audio features and sample subtitle features.

[0150] In the specific implementation of this embodiment, the specific process of step S403 is similar to that of step S401 described above. To save space, it will not be described again.

[0151] In step S404 of some embodiments, the first similar data, the second similar data, and the third similar data are integrated, and the first similar data, the second similar data, and the third similar data are incorporated into the same data set, and all the data in the data set are used as feature similar data.

[0152] The advantage of this embodiment is that it takes into account the pairwise cross-similarity calculation of the sample image features, sample subtitle features and sample audio features of the sample video, and realizes the cross-feature alignment of the feature information of the image modality, audio modality and text modality of the sample video. This enables the model to better learn the correlation between different modalities during training, thereby improving the accuracy of the trained video description generation model in generating video description text for various videos.

[0153] In this embodiment of the application, the sample image features include global image features and local image features, and the sample audio features include global audio features and local audio features.

[0154] Global image features are used to indicate the overall feature information of multiple video frames in the sample video image content.

[0155] Local image features are used to indicate the feature information of each video frame in the sample video image content.

[0156] Global audio features are used to indicate the overall feature information of multiple audio segments in the audio content of a sample video.

[0157] Local audio features are used to indicate the feature information of individual audio segments in the audio content of a sample video.

[0158] Please see Figure 5 In some embodiments, step S401 may include, but is not limited to, steps S501 to S504:

[0159] Step S501: Calculate the first similarity between global image features and global audio features, calculate the second similarity between global image features and local audio features, calculate the third similarity between local image features and global audio features, and calculate the fourth similarity between local image features and local audio features.

[0160] Step S502: Perform a comparison calculation based on the third similarity to obtain the first comparison similarity, and perform a comparison calculation based on the second similarity to obtain the second comparison similarity;

[0161] Step S503: Perform a comparison calculation based on the fourth similarity to obtain the third comparison similarity;

[0162] Step S504: The third similarity, the first comparative similarity, the second comparative similarity, and the third comparative similarity are averaged to obtain the first similarity data.

[0163] Steps S501 to S504 are described in detail below.

[0164] In step S501 of some embodiments, a first similarity is used to indicate the overall similarity between the sample video image content and the sample video audio content of the sample video. A second similarity is used to indicate the similarity of each of the multiple audio segments within the sample video image content and the sample video audio content of the sample video. A third similarity is used to indicate the similarity between each of the multiple video frame images within the sample video image content and the sample video audio content. A fourth similarity is used to indicate the feature similarity between a single video frame image and a single audio segment.

[0165] In this specific implementation, cosine similarity is used to calculate the first similarity between global image features and global audio features. Simultaneously, cosine similarity is used to calculate the second similarity between global image features and each local audio feature. Cosine similarity is then used to calculate the third similarity between each local image feature and global audio features. Finally, cosine similarity is used to calculate the fourth similarity between each local image feature and each local audio feature.

[0166] It should be noted that when there are m cluster center images in the sample video image content and n audio segments in the sample video audio content, one first similarity score, n second similarity scores, m third similarity scores, and m×n fourth similarity scores will be obtained. Here, m and n are both positive integers.

[0167] In a specific example, the first similarity S a,v It can be represented as: S a,v =(f a ) T (f v ), where f v It is a global image feature, f a It is a global audio feature, (f a ) T This refers to the transpose of global image features. First similarity S a,v It is data in a one-dimensional real number space.

[0168] In a specific example, the second similarity S as,v It can be represented as: S as,v =f as f v , where f v It is a global image feature, f asIt represents the local audio features corresponding to a single audio segment. The second similarity S formed by all audio segments... as,v It is an n-row, one-column real matrix.

[0169] In a specific example, the third similarity S a,f It can be represented as: S a,f =(f a f f ) T , where f f It is the local image feature corresponding to a single cluster center image, f a It is a global audio feature, (f a f f ) T This refers to the transpose of the product of local image features and global audio features. All cluster center images form a third similarity S. a,f It is a real matrix with one row and m columns.

[0170] In a specific example, the fourth similarity S as,f It can be represented as: S as,f =(f f ) T (f as ), where f f It is the local image feature corresponding to a single cluster center image, f as It is the local audio feature corresponding to a single audio segment, (f f ) T This refers to the transpose of the local image features of a single cluster center image. The fourth similarity S formed by all audio segments and all cluster center images... as,f It is an n x m real matrix.

[0171] In step S502 of some embodiments, the first comparison similarity is used to indicate the overall similarity between the local image features and the global audio features of all cluster center images, and the second comparison similarity is used to indicate the overall similarity between the local audio features and the global image features of all audio segments.

[0172] It should be noted that both the first and second contrastive similarities are instance-level similarities. Instance-level similarity is generally used to evaluate the semantic relevance between two specific instances in the training data.

[0173] In this specific implementation, when performing comparison calculations based on the third similarity, the third similarity of each cluster center image is normalized to obtain the first comparison similarity of the cluster center image. Similarly, when performing comparison calculations based on the second similarity, the second similarity of each audio segment is normalized to obtain the second comparison similarity of the audio segment.

[0174] It should be noted that the normalization process described above is mainly used to convert the original second and third similarities into a probability distribution.

[0175] In a specific example, the first comparison similarity can be expressed as:

[0176]

[0177] Where, S a,f It is the third similarity, S′ a,f The first comparison similarity, S a,f(1,i) It is the element in the 1st row and i-th column of the third similarity (i.e., the similarity between the local image features and the global audio features of the i-th cluster center image); exp(S a,f(1,i) ) indicates that the base is the natural constant and the digit is S. a,f(1,i) The result of performing an exponential calculation on an index; S a,f(1,j) This represents the element in the 1st row and jth column of the third similarity (i.e., the similarity between the local image features and the global audio features of the jth cluster center image); It is for m numbers with the natural constant as the base and S a,f(1,j) The result is the sum of the results of the exponential calculation. The first comparison similarity is the sum of the normalized probabilities and the original similarity matrix S. a,f The product is obtained by multiplying the values ​​to achieve probability weighting for each element in the third similarity score. Here, i and j are positive integers not greater than m.

[0178] The second comparison similarity can be expressed as:

[0179]

[0180] Among them, S as,v It is the second similarity, S′ as,v It is the second comparison similarity, S as,v(p,1) It is the element in the p-th row and 1-th column of the second similarity (i.e., the similarity between the local audio features and the global image features of the p-th audio segment); exp(S as,v(p,1) ) indicates that the base is the natural constant and the digit is S. as,v(p,1) The result of performing an exponential calculation on an index; S as,v(q,1)This represents the element in the q-th row and 1-th column of the second similarity (i.e., the similarity between the local audio features and the global image features of the q-th audio segment); For n numbers with the natural constant as the base and S as the base... ss,v(q,1) The first is the sum of the results of the exponential calculation for the exponent. The second comparison similarity is the sum of the normalized probabilities and the original similarity matrix S. as,v The product is obtained by multiplying the values ​​to achieve probability weighting for each element in the second similarity score. Here, p and q are positive integers no greater than n.

[0181] In step S503 of some embodiments, the third comparison similarity is used to indicate the overall similarity of the feature information of the sample video at the audio level and the image level.

[0182] In this specific implementation, firstly, for a single audio segment, the fourth similarity between the audio segment and all cluster center images is normalized to obtain a first normalized similarity. Next, for a single cluster center image, the fourth similarity between the cluster center image and all audio segments is normalized to obtain a second normalized similarity. Further, a third comparative similarity is obtained by weighting the first and second normalized similarities.

[0183] In a specific example, the first normalized similarity can be expressed as follows:

[0184]

[0185] Among them, S avd(1,i) This refers to the similarity value corresponding to the cluster center image in the 1st row and 1st column of the first similarity matrix; S avd(1,j) Let represent the similarity value corresponding to the cluster center image in the 1st row and jth column of the first similarity matrix. Here, i and j are positive integers not greater than m.

[0186] Furthermore, the first similarity matrix is ​​represented as follows:

[0187]

[0188] Among them, S as,f(p,*) S represents the fourth similarity between the p-th audio segment and all cluster center images; as,f(q,*) This represents the fourth similarity between the q-th audio segment and all cluster center images; S represents the sum of exponential results calculated using the fourth similarity between each of the n audio segments and all cluster center images as the exponent, with the natural base. avd S is a 1×m vector matrix. avd S represents the normalized similarity between each audio segment and all cluster center images.as,f(p,) This represents the fourth similarity between the p-th audio segment and the images of each cluster center. This represents the normalization coefficient of the p-th audio segment. Let represent the normalized similarity between the p-th audio segment and each cluster center image. This normalized similarity is a 1×m vector matrix. Based on this, The instruction specifies that for each cluster center image, the normalized similarity of all audio segments corresponding to that cluster center image is summed to obtain a similarity value. The vector matrix formed by the similarity values ​​of the M cluster center images is the first similarity matrix S. avd The first similarity matrix is ​​also a 1×m vector matrix. Here, p and q are positive integers no greater than n.

[0189] Similarly, the second normalized similarity can be expressed as follows:

[0190]

[0191] Among them, S vid(p,1) This refers to the similarity value corresponding to the audio segment in the p-th row and 1-th column of the second similarity matrix; S vid(q,1) Let p represent the similarity value of the audio segment in the q-th row and 1-th column of the second similarity matrix. Here, p and q are positive integers not greater than n.

[0192] Furthermore, the second similarity matrix is ​​represented as follows:

[0193]

[0194] Among them, S as,f(*,i) S represents the fourth similarity between the i-th cluster center image and all audio segments; as,f(*,j) This represents the fourth similarity between the j-th cluster center image and all audio segment images; S represents the sum of exponential results calculated using the fourth similarity between each of the m cluster center images and all audio segments as the exponent, with the natural base. vid S is a 1×n vector matrix. vid S represents the normalized similarity between each cluster center image and all audio segments. as , f( , i) This represents the fourth similarity between the i-th cluster center image and each audio segment. Represents the normalization coefficient of the image of the i-th cluster center. Let represent the normalized similarity between the i-th cluster center image and each audio segment. This normalized similarity is an n×1 vector matrix. Based on this, The instruction specifies that for each audio segment, the normalized similarity of all cluster center images corresponding to that audio segment is summed to obtain a similarity value. The vector matrix formed by the similarity values ​​of N audio segments is the second similarity matrix S. vid The second similarity matrix is ​​also an n×1 vector matrix, where i and j are positive integers not greater than m.

[0195] In step S504 of some embodiments, the third similarity, the first comparative similarity, the second comparative similarity, and the third comparative similarity are averaged. First, the second similarity, the first comparative similarity, the second comparative similarity, and the third comparative similarity are added together to obtain the total similarity. Then, the total similarity is divided by four to obtain the average similarity, and this average similarity is used as the first similarity data of the sample video.

[0196] In a specific example, the first similarity data Sim from the sample video AV It can be expressed as follows:

[0197] Sim AV =(S a,f +S' a,f +S′ as,v +S′ as,f ) / 4;

[0198] Among them, S a,f It is the second similarity, S' a,f It is the first comparison similarity, S′ as,v This is the second comparison similarity, S′ as,f It is the third comparison similarity.

[0199] The advantage of this embodiment is that after calculating the similarity of the feature content of the sample video in the image modality, audio modality and text modality respectively, it takes into account the feature alignment of the feature content of each modality. By using the idea of ​​cross-similarity calculation, it integrates instance-level similar feature alignment, audio-level similar feature alignment and image-level similar feature alignment in the alignment of feature content of audio modality and image modality, which can improve the accuracy and effectiveness of feature alignment.

[0200] In step S105 of some embodiments, the image modal encoder, audio modal encoder and text modal encoder are jointly trained based on feature similarity data to obtain a video description generation model.

[0201] Video description generation models are artificial intelligence models that can generate corresponding video description text based on videos. These models typically take the video as input and output the corresponding video description text.

[0202] In this embodiment of the application, the video description generation model includes an image modal encoder, an audio modal encoder, a text modal encoder, and a text decoder.

[0203] The text decoder is used to map the feature vectors generated by the video description generation model based on the input data from the latent vector space to the text data space, so as to output the text content corresponding to the input data.

[0204] In this embodiment, the text decoder can be an autoregressive model with a transformer structure.

[0205] Please see Figure 6 In some embodiments, step S105 may include, but is not limited to, steps S601 to S602:

[0206] Step S601: Calculate the loss based on the first similar data to obtain the first loss value; calculate the loss based on the second similar data to obtain the second loss value; and calculate the loss based on the third similar data to obtain the third loss value.

[0207] Step S602: Train an image modal encoder based on the first loss value and the third loss value, train an audio modal encoder based on the first loss value and the second loss value, and train a text modal encoder based on the second loss value and the third loss value to obtain a video description generation model.

[0208] Steps S601 to S602 are described in detail below.

[0209] In step S601 of some embodiments, a first loss value is used to indicate the degree of similarity between the sample image features and the sample audio features and the degree of difference from the training expectation. A second loss value is used to indicate the degree of similarity between the sample image features and the sample subtitle features and the degree of difference from the training expectation. A third loss value is used to indicate the degree of similarity between the sample audio features and the sample subtitle features and the degree of difference from the training expectation.

[0210] In the specific implementation of this embodiment, the first loss function can be calculated using the idea of ​​a contrastive loss function. Specifically, since there are generally multiple sample videos during training, the first similarity data between the sample image features and sample audio features of the same sample video can be used as positive sample values, and the first similarity data between the sample image features and sample audio features of different sample videos can be used as negative sample values. For a single sample video, there is generally one positive sample value and multiple negative sample values. Next, for a single sample video, the result with the natural parameter as the base and the positive sample value as the exponent is used as the numerator, and the sum of the results with the natural constant as the base and the negative sample value as the exponent is used as the denominator to obtain the first result. Further, the negative logarithm of the first result is taken to obtain the sub-loss value of a single sample video, and the average of the sub-loss values ​​of multiple sample videos is used to obtain the first loss value.

[0211] For example, when there are three sample videos, namely sample video A, sample video B, and sample video C, for sample video A, the first similarity data between the sample image features and sample audio features of sample video A is taken as positive sample value 1. The first similarity data between the sample audio features of sample video A and the sample image features of sample video B is taken as negative sample value 1, and the first similarity data between the sample audio features of sample video A and the sample image features of sample video C is taken as negative sample value 2. When calculating the sub-loss value of sample video A, the result with the natural parameter as the base and the positive sample value 1 as the exponent is used as the numerator, and the sum of the result with the natural constant as the base and the negative sample value 1 as the exponent and the result with the natural constant as the base and the negative sample value 2 as the exponent is used as the denominator to obtain the first result.

[0212] In step S602 of some embodiments, when training the image modality encoder based on the first loss value and the third loss value, the first loss value and the third loss value are first added together to obtain the total loss value. Then, the total loss value is compared with a preset loss threshold. If the total loss value is not less than the preset loss threshold, the parameters of the image modality encoder are adjusted, and the process returns to steps S101-S105 above. The image modality encoder is iteratively trained continuously until the total loss value is less than the preset loss threshold in a certain training round. Training then stops, and the parameters of the image modality encoder in that training round are used as the final model parameters. The image modality encoder in that training round is used as the final image modality encoder for constructing the video description generation model.

[0213] Furthermore, the specific process of training the audio modal encoder based on the first and second loss values ​​is similar to that of training the image modal encoder based on the first and third loss values. Similarly, the specific process of training the text modal encoder based on the second and third loss values ​​is similar to that of training the image modal encoder based on the first and third loss values. For the sake of brevity, these details will not be elaborated further.

[0214] Furthermore, after the image modal encoder, audio modal encoder, and text modal encoder have all been trained, they are connected in parallel, and the parallel network structure is then connected in series with the text decoder to form a complete video description generation model.

[0215] The advantage of this embodiment is that it takes into account the joint training of image modal encoder, audio modal encoder and text modal encoder based on feature similarity data. It uses the similarity data of feature content between different modalities of sample video as the basis for calculating contrast loss. Contrast loss is calculated separately for image modal encoder, audio modal encoder and text modal encoder. This can maximize the similarity of feature content of each modality of the same sample video and minimize the similarity of feature content of each model of different sample videos, thereby improving the accuracy of the trained video description generation model in generating insurance description text.

[0216] In step S106 of some embodiments, the video image, video subtitle content and video audio content of the target video are input into the video description generation model to generate a video description, thereby obtaining the video description text of the target video.

[0217] The target video refers to the video for which a video description needs to be generated.

[0218] The video description text of the target video refers to the text data output by the video description generation model based on the target video, which describes the main content of the target video.

[0219] Please see Figure 7 In some embodiments, step S106 may include, but is not limited to, steps S701 to S703:

[0220] Step S701: Perform feature encoding on the video image based on the image modal encoder to obtain video image features; perform feature encoding on the video audio content based on the audio modal encoder to obtain video audio features; perform feature encoding on the video subtitle content based on the text modal encoder to obtain video subtitle features;

[0221] Step S702: Perform splicing processing on video image features, video audio features, and video subtitle features to obtain video splicing features;

[0222] Step S703: Based on the text decoder, the video splicing features are decoded to obtain the video description text.

[0223] Steps S701 to S703 are described in detail below.

[0224] In step S701 of some embodiments, the specific implementation process is similar to step S103 described above. To save space, it will not be repeated here.

[0225] In step S702 of some embodiments, since the video image features, video audio features, and video subtitle features are feature vectors of the same feature dimension, they can be directly concatenated into a longer feature vector. This concatenated feature vector is then used as the video concatenation feature. The video concatenation feature is used to comprehensively represent the feature information of the sample video in the image modality, text modality, and audio modality.

[0226] In step S703 of some embodiments, the specific implementation of this embodiment may include, but is not limited to, the following steps:

[0227] The video splicing features are decoded using a text decoder to obtain the first descriptive word;

[0228] Initialize the target vector with the word embedding features corresponding to the first descriptor;

[0229] The target vector is decoded using a text decoder to obtain the next descriptive word;

[0230] Update the target vector with the word embedding features of the next descriptor, and return the step of decoding the target vector based on the text decoder to obtain the next descriptor, until a stop symbol is generated, and concatenate the obtained multiple descriptors into the video description text.

[0231] The first descriptor refers to the descriptor predicted by the text decoder based on video concatenation features. Word embedding features are used to indicate the semantic features of the first descriptor. The next descriptor refers to the descriptor predicted by the text decoder based on the target vector. The stop symbol is used to specify when the text decoder's decoding process stops.

[0232] Specifically, firstly, when decoding the video stitching features using a text decoder, the video stitching features are linearly projected to obtain the corresponding query features, key features, and value features. Attention is then calculated based on these features to obtain the video attention calculation result. This result is then fed forward to obtain the video feedforward features, which are mapped from the latent vector space to the data space to obtain the first descriptor. Further, the first descriptor is embedded to obtain the corresponding word embedding features, and the target vector is initialized with these features. Finally, the target vector is decoded using a text decoder to obtain the next descriptor, a process similar to decoding the video stitching features using a text decoder to obtain the first descriptor. Next, the target vector is updated with the word embedding features of the next descriptor, and the process of decoding the target vector based on the text decoder to obtain the next descriptor continues until a stop character is decoded based on the word embedding features of a descriptor. The decoding process ends, and multiple descriptors are concatenated in sequence according to their generation time to obtain the video description text.

[0233] For example, at the first moment, the text decoder generates the first word based on the video splicing features. At the second moment, it uses the vector of the first word as input to generate the second word, and so on, until a stop character is generated. Then, the words generated at all moments are spliced ​​together to form the video description text of the target video, which represents the entire text content of the target video.

[0234] The advantage of this embodiment is that by generating video descriptions using the target video's video images, video subtitles, and video audio content, and by combining the feature information of the target video across image, text, and audio modalities, the comprehensiveness and diversity of the video splicing features can be improved. Furthermore, by using an autoregressive text decoder to decode the video splicing features temporally, and using the embedding features of the previously decoded descriptive word as input for the next decoding process, the temporal continuity and semantic dependencies of each descriptive word can be improved. This approach enhances the coherence and accuracy of the text content of the video description text spliced ​​from multiple descriptive words.

[0235] Please see Figure 8 This application also provides a video description generation apparatus that can implement the above-described video description generation method. The apparatus includes:

[0236] Acquisition unit 801 is used to acquire sample videos;

[0237] Extraction unit 802 is used to extract sample video image content, sample video audio content, and sample video subtitle content from the sample video.

[0238] The encoding unit 803 is used to perform feature encoding on the sample video image content based on a preset image modal encoder to obtain sample image features, to perform feature encoding on the sample video audio content based on a preset audio modal encoder to obtain sample audio features, and to perform feature encoding on the sample video subtitle content based on a preset text modal encoder to obtain sample subtitle features.

[0239] The calculation unit 804 is used to perform similarity calculation based on sample image features, sample audio features, and sample subtitle features to obtain feature similarity data;

[0240] Training unit 805 is used to jointly train the image modal encoder, audio modal encoder and text modal encoder based on feature similarity data to obtain a video description generation model;

[0241] The generation unit 806 is used to input the video image, video subtitle content and video audio content of the target video into the video description generation model to generate a video description and obtain the video description text of the target video.

[0242] In some embodiments, the extraction unit 802 is used for:

[0243] Video frames are extracted from the sample video to obtain multiple video frame images;

[0244] Perform image clustering on multiple video frame images to obtain video frame image classes and cluster center images of video frame image classes, wherein there are at least two video frame image classes;

[0245] The cluster center images are flattened in time sequence to obtain the sample video image content.

[0246] In some embodiments, image clustering is performed on multiple video frame images to obtain video frame image classes and cluster center images of the video frame image classes, including:

[0247] For each video frame image, obtain the video frame image fingerprint feature of the video frame image;

[0248] Based on the fingerprint features of video frame images, the similarity between every two video frame images is calculated to obtain the image fingerprint similarity.

[0249] Based on image fingerprint similarity, multiple video frame images are clustered to obtain video frame image classes, so that the image fingerprint similarity between any two video frame images in the video frame image class is greater than a preset threshold.

[0250] For any video frame image in the video image class, the overall similarity of the video frame image is determined based on the image fingerprint similarity between the video frame image and other video frame images in the video image class;

[0251] Based on overall similarity, cluster center images are determined among multiple video frame images of the video image class.

[0252] In some embodiments, the extraction unit 802 is used for:

[0253] Determine the video length of the sample video;

[0254] Based on the video length, pitch features are extracted from the sample video to obtain the audio content of the sample video, where the audio length of the sample video audio content is equal to the video length.

[0255] In some embodiments, the extraction unit 802 is used for:

[0256] Audio content recognition is performed on the sample video to obtain the first video text;

[0257] Subtitle features are extracted from the sample video to obtain the second video text;

[0258] The first video text and the second video text are integrated into sample video subtitle content.

[0259] In some embodiments, the calculation unit 804 is used for:

[0260] The first similarity data is obtained by calculating the cross-similarity based on the sample image features and sample audio features;

[0261] Cross-similarity calculation is performed based on sample image features and sample caption features to obtain the second similarity data;

[0262] The third similarity data is obtained by calculating the cross-similarity based on the sample audio features and sample subtitle features;

[0263] The first, second, and third similar data are integrated to obtain feature similarity data.

[0264] Sample image features include global image features and local image features; sample audio features include global audio features and local audio features.

[0265] Based on the sample image features and sample audio features, cross-similarity calculation is performed to obtain the first similarity data, including:

[0266] Calculate the first similarity between global image features and global audio features, the second similarity between global image features and local audio features, the third similarity between local image features and global audio features, and the fourth similarity between local image features and local audio features.

[0267] The first comparison similarity is obtained by comparing based on the third similarity, and the second comparison similarity is obtained by comparing based on the second similarity.

[0268] The third similarity score is obtained by comparing the fourth similarity score.

[0269] The first similarity data is obtained by averaging the third similarity, first contrast similarity, second contrast similarity, and third contrast similarity.

[0270] In some embodiments, the feature similarity data includes first similarity data, second similarity data, and third similarity data;

[0271] Training unit 805 is used for:

[0272] Loss is calculated based on the first similar data to obtain the first loss value; loss is calculated based on the second similar data to obtain the second loss value; and loss is calculated based on the third similar data to obtain the third loss value.

[0273] The image modal encoder is trained based on the first and third loss values, the audio modal encoder is trained based on the first and second loss values, and the text modal encoder is trained based on the second and third loss values ​​to obtain the video description generation model.

[0274] In some embodiments, the video description generation model includes an image modal encoder, an audio modal encoder, a text modal encoder, and a text decoder;

[0275] The generation unit 806 is used for:

[0276] Video image features are obtained by feature encoding based on image modality encoder; video audio features are obtained by feature encoding based on audio modality encoder; and video subtitle features are obtained by feature encoding based on text modality encoder.

[0277] Video image features, video audio features, and video subtitle features are spliced ​​together to obtain video splicing features;

[0278] The video splicing features are decoded using a text decoder to obtain the video description text.

[0279] In some embodiments, video splicing features are decoded based on a text decoder to obtain video description text, including:

[0280] The video splicing features are decoded using a text decoder to obtain the first descriptive word;

[0281] Initialize the target vector with the word embedding features corresponding to the first descriptor;

[0282] The target vector is decoded using a text decoder to obtain the next descriptive word;

[0283] Update the target vector with the word embedding features of the next descriptor, and return the step of decoding the target vector based on the text decoder to obtain the next descriptor, until a stop symbol is generated, and concatenate the obtained multiple descriptors into the video description text.

[0284] The specific implementation of the video description generation device is basically the same as the specific embodiment of the video description generation method described above, and will not be repeated here.

[0285] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned video description generation method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0286] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0287] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0288] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the video description generation method of the embodiments of this application.

[0289] The input / output interface 903 is used to implement information input and output;

[0290] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0291] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0292] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0293] This application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the video description generation method described above.

[0294] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0295] The video description generation method, video description generation device, electronic device, and computer-readable storage medium provided in this application embodiment acquire sample videos; extract sample video image content, sample video audio content, and sample video subtitle content from the sample videos, enabling the extraction of content information from multiple modalities of the sample videos, which is beneficial to improving the comprehensiveness of feature extraction. Furthermore, based on a preset image modality encoder, feature encoding is performed on the sample video image content to obtain sample image features; based on a preset audio modality encoder, feature encoding is performed on the sample video audio content to obtain sample audio features; and based on a preset text modality encoder, feature encoding is performed on the sample video subtitle content to obtain sample subtitle features. This enables unified encoding of content information from multiple modalities of the sample video, converting the feature content of image modality, audio modality, and text modality into the same feature dimension. Furthermore, similarity calculations are performed based on sample image features, sample audio features, and sample caption features to obtain feature similarity data. Based on this feature similarity data, the image modality encoder, audio modality encoder, and text modality encoder are jointly trained to obtain a video description generation model. This approach achieves cross-similarity calculation of feature content across multiple modalities and considers feature alignment across these modalities. This allows the video description generation model to learn more about the correlation of feature information from different modalities during training, which is beneficial for training a video description generation model with better performance. Finally, the video image, video caption content, and video audio content of the target video are input into the video description generation model to generate video description text for the target video, improving the accuracy and efficiency of the generated video description text.

[0296] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0297] It will be understood by those skilled in the art that Figure 1-7 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0298] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0299] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0300] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0301] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0302] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0303] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0304] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0305] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0306] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A video description generation method, characterized in that, The method includes: Obtain sample videos; In the sample video, the sample video image content, sample video audio content, and sample video subtitle content are extracted; The sample video image content is feature-encoded based on a preset image modal encoder to obtain sample image features; the sample video audio content is feature-encoded based on a preset audio modal encoder to obtain sample audio features; and the sample video subtitle content is feature-encoded based on a preset text modal encoder to obtain sample subtitle features. Similarity calculations are performed based on the sample image features, the sample audio features, and the sample subtitle features to obtain feature similarity data; Based on the feature similarity data, the image modal encoder, the audio modal encoder, and the text modal encoder are jointly trained to obtain a video description generation model; The video image, video subtitles, and video audio of the target video are input into the video description generation model to generate a video description, resulting in the video description text of the target video.

2. The method according to claim 1, characterized in that, The step of extracting sample video image content from the sample video includes: The sample video is subjected to video frame extraction to obtain multiple video frame images; The plurality of video frame images are subjected to image clustering to obtain video frame image classes and cluster center images of the video frame image classes, wherein there are at least two video frame image classes; The cluster center image is flattened in time sequence to obtain the sample video image content.

3. The method according to claim 2, characterized in that, The step of performing image clustering on the multiple video frame images to obtain video frame image classes and cluster center images of the video frame image classes includes: For each of the video frame images, obtain the video frame image fingerprint feature of the video frame image; Based on the image fingerprint features of the video frames, the similarity between every two video frames is calculated to obtain the image fingerprint similarity. Based on the image fingerprint similarity, the multiple video frame images are clustered to obtain the video frame image class, so that the image fingerprint similarity of any two video frame images in the video frame image class is greater than a preset threshold. For any video frame image in the video image class, the overall similarity of the video frame image is determined based on the image fingerprint similarity between the video frame image and other video frame images in the video image class; Based on the overall similarity, the cluster center image is determined among the plurality of video frame images of the video image class.

4. The method according to claim 1, characterized in that, The similarity calculation based on the sample image features, the sample audio features, and the sample subtitle features yields feature similarity data, including: Based on the sample image features and the sample audio features, cross-similarity calculation is performed to obtain the first similarity data; Based on the sample image features and the sample caption features, cross-similarity calculation is performed to obtain the second similarity data; Based on the sample audio features and the sample subtitle features, a cross-similarity calculation is performed to obtain the third similarity data; The first similar data, the second similar data, and the third similar data are integrated to obtain the feature similar data.

5. The method according to claim 4, characterized in that, The sample image features include global image features and local image features, and the sample audio features include global audio features and local audio features; The first similarity data is obtained by performing cross-similarity calculation based on the sample image features and the sample audio features, including: Calculate the first similarity between the global image features and the global audio features, calculate the second similarity between the global image features and the local audio features, calculate the third similarity between the local image features and the global audio features, and calculate the fourth similarity between the local image features and the local audio features; A comparison calculation is performed based on the third similarity to obtain a first comparison similarity, and a comparison calculation is performed based on the second similarity to obtain a second comparison similarity; Based on the fourth similarity, a comparison calculation is performed to obtain the third comparison similarity. The first similarity data is obtained by averaging the third similarity, the first comparative similarity, the second comparative similarity, and the third comparative similarity.

6. The method according to claim 1, characterized in that, The feature similarity data includes first similarity data, second similarity data, and third similarity data; The video description generation model is obtained by jointly training the image modal encoder, the audio modal encoder, and the text modal encoder based on the feature similarity data, including: A first loss value is obtained by performing loss calculation based on the first similar data, a second loss value is obtained by performing loss calculation based on the second similar data, and a third loss value is obtained by performing loss calculation based on the third similar data. The image modal encoder is trained based on the first loss value and the third loss value, the audio modal encoder is trained based on the first loss value and the second loss value, and the text modal encoder is trained based on the second loss value and the third loss value to obtain the video description generation model.

7. The method according to any one of claims 1 to 6, characterized in that, The video description generation model includes an image modal encoder, an audio modal encoder, a text modal encoder, and a text decoder; The step of inputting the video image, video subtitle content, and video audio content of the target video into the video description generation model to generate a video description and obtain the video description text of the target video includes: The video image is feature-encoded using the image modality encoder to obtain video image features; the video audio content is feature-encoded using the audio modality encoder to obtain video audio features; and the video subtitle content is feature-encoded using the text modality encoder to obtain video subtitle features. The video image features, the video audio features, and the video subtitle features are spliced ​​together to obtain video splicing features; Based on the text decoder, the video splicing features are decoded to obtain the video description text.

8. A video description generation apparatus, characterized in that, The video description generation device includes: The acquisition unit is used to acquire sample videos; The extraction unit is used to extract sample video image content, sample video audio content, and sample video subtitle content from the sample video. The encoding unit is used to perform feature encoding on the sample video image content based on a preset image modality encoder to obtain sample image features, to perform feature encoding on the sample video audio content based on a preset audio modality encoder to obtain sample audio features, and to perform feature encoding on the sample video subtitle content based on a preset text modality encoder to obtain sample subtitle features. The calculation unit is used to perform similarity calculation based on the sample image features, the sample audio features, and the sample subtitle features to obtain feature similarity data; The training unit is used to jointly train the image modal encoder, the audio modal encoder, and the text modal encoder based on the feature similarity data to obtain a video description generation model; The generation unit is used to input the video image, video subtitle content and video audio content of the target video into the video description generation model to generate a video description and obtain the video description text of the target video.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the video description generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the video description generation method according to any one of claims 1 to 7.

Images