Multi-modal video description generation method and device, and computer readable medium

Abstract

The embodiment of the invention provides a multi-modal video description generation method and device and a computer readable medium, and in the scheme, image features extracted based on video frames are utilized, and sound features extracted based on audio waveform information of an input video are also utilized, so that the multi-modal video description generation method and device are provided. In this way, video description information can be generated by combining multi-modal feature information such as images and sounds, meanwhile, by setting a reasonable feature extraction mode and utilizing a cross attention mechanism to organically fuse feature information of different modalities, compared with a single type of image features, due to the fact that the input features are more comprehensive, the video description efficiency is improved, and the video description efficiency is improved. The video description information about the input video can be more accurately generated, so that the actual content of the input video can be more accurately described.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a multimodal video description generation method, device and computer-readable medium. Background Art

[0002] With the development of information technology, video has gradually become a major data carrier on the Internet. Automatically generate corresponding text descriptions for a given video or video clip. Video description generation technology is to help machines understand video content and convert it into text form for easy understanding and retrieval by humans. In current video description generation schemes, the image features of each video frame in the video are generally extracted, and then input into a pre-trained video description generation model to obtain description information about the video. However, since the features input into the video description generation model in this type of scheme are only image features about the video frame image information, and the feature types are relatively single, the generated video description information is often inaccurate and cannot accurately describe the actual content of the video. Summary of the invention

[0003] An object of the present application is to provide a multimodal video description generation method, device and computer-readable medium to solve the problem of insufficient accuracy of video description information generated in existing solutions.

[0004] To achieve the above object, an embodiment of the present application provides a method for generating a multimodal video description, the method comprising:

[0005] Extracting video frames from an input video and separating audio waveform information from the input video;

[0006] Adding a video start mark and a video end mark before and after the video frame, respectively, and adding an audio start mark and an audio end mark before and after the audio waveform information, respectively;

[0007] Perform feature encoding on the video start mark, each video frame, video end mark, audio start mark, audio waveform information and audio end mark respectively to obtain corresponding first feature token, second feature token, third feature token, fourth feature token, fifth feature token and sixth feature token;

[0008] Respectively optimizing the first feature token, the second feature token, the third feature token, the fourth feature token, the fifth feature token, and the sixth feature token to obtain corresponding first feature information, second feature information, third feature information, fourth feature information, fifth feature information, and sixth feature information;

[0009] Using a cross attention mechanism, the first feature information, the fourth feature information, the fifth feature information and the sixth feature information are fused to obtain head features;

[0010] The head feature and the second feature information are spliced ​​and input into a preset video description generation model to obtain video description information of the input video.

[0011] Further, extracting video frames from the input video includes:

[0012] A plurality of video frames are extracted from the input video at preset time intervals.

[0013] Further, the video start mark, each video frame, the video end mark, the audio start mark, the audio waveform information and the audio end mark are feature encoded respectively to obtain the corresponding first feature token, second feature token, third feature token, fourth feature token, fifth feature token and sixth feature token, including:

[0014] The video start mark, the video end mark, the audio start mark and the audio end mark are respectively feature encoded by linear space embedding to obtain the corresponding first feature token, third feature token, fourth feature token and sixth feature token;

[0015] Use the encoder of the same VQVAE model to perform feature encoding on each video frame to obtain a set of corresponding second feature tokens;

[0016] The audio waveform information is feature encoded using an encoder of another VQVAE model to obtain a set of corresponding fifth feature tokens.

[0017] Further, the first feature token, the second feature token, the third feature token, the fourth feature token, the fifth feature token and the sixth feature token are respectively optimized to obtain the corresponding first feature information, the second feature information, the third feature information, the fourth feature information, the fifth feature information and the sixth feature information, including:

[0018] The first feature token, the second feature token, the third feature token, the fourth feature token, the fifth feature token and the sixth feature token are flattened and arranged in order, and then input into a large language model, which performs feature optimization on them to obtain corresponding first feature information, second feature information, third feature information, fourth feature information, fifth feature information and sixth feature information.

[0019] Furthermore, before using the cross attention mechanism to perform feature fusion on the first feature information, the fourth feature information, the fifth feature information and the sixth feature information to obtain the head feature, the method further includes:

[0020] Adding Gaussian noise information to the first feature information, the second feature information, the fourth feature information, the fifth feature information, and the sixth feature information;

[0021] The first feature information, the second feature information, the fourth feature information, the fifth feature information and the sixth feature information after adding the Gaussian noise information are denoised.

[0022] Furthermore, the first feature information, the fourth feature information, the fifth feature information and the sixth feature information are fused by using a cross attention mechanism to obtain head features, including:

[0023] According to the first feature information, obtain the Q vector feature of the cross attention mechanism model;

[0024] Performing global average pooling processing on the fourth feature information, the fifth feature information, and the sixth feature information to obtain audio feature information;

[0025] Input the audio feature information into two different self-attention mechanism models respectively to obtain K vector features and V vector features of the cross-attention mechanism model;

[0026] The K vector features, V vector features and Q vector features are input into the cross attention mechanism model to obtain the head features.

[0027] Furthermore, the video description generation model is a large language model;

[0028] The head feature and the second feature information are spliced ​​and input into a preset video description generation model to obtain video description information of the input video, including:

[0029] Concatenate the head feature and the second feature information to obtain a description generation feature;

[0030] The following iterative processing is performed based on the description generation feature: the description generation feature is input into a preset large language model to obtain output information of the first position of the large language model; an output character corresponding to the output information is determined by a preset classification function; the output character is feature-encoded, and the result of the feature encoding is concatenated to the back of the head feature and the second feature information to obtain a new description generation feature, and the stopping condition of the iterative processing is that the large language model outputs a stop symbol;

[0031] The generated output characters are combined in sequence to generate video description information of the input video.

[0032] Furthermore, before the feature encoding result is spliced ​​to the head feature and the second feature information, and a new description generation feature is obtained, the method further includes:

[0033] Adding Gaussian noise information to the result of the feature encoding;

[0034] The result of feature encoding after adding Gaussian noise information is denoised.

[0035] Some embodiments of the present application also provide a multimodal video description generating device, wherein the device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the aforementioned multimodal video description generating method.

[0036] Some other embodiments of the present application further provide a computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the multimodal video description generating method.

[0037] Compared with the prior art, in a multimodal video description generation scheme provided in an embodiment of the present application, first, video frames are extracted from an input video, and audio waveform information is separated from the input video; a video start mark and a video end mark are added before and after the video frame, respectively, and an audio start mark and an audio end mark are added before and after the audio waveform information, respectively; feature encoding is performed on the video start mark, each video frame, the video end mark, the audio start mark, the audio waveform information, and the audio end mark, respectively, to obtain the corresponding first feature token, second feature token, third feature token, fourth feature token, fifth feature token, and sixth feature token; feature optimization is performed on the first feature token, the second feature token, the third feature token, the fourth feature token, the fifth feature token, and the sixth feature token, respectively, to obtain the corresponding first feature information, second feature information, third feature information, fourth feature information, fifth feature information, and sixth feature information; the first feature information, the fourth feature information, the fifth feature information, and the sixth feature information are feature fused using a cross-attention mechanism to obtain head features; the head features and the second feature information are spliced ​​and input into a preset video description generation model to obtain video description information of the input video. In this scheme, in addition to utilizing image features extracted based on video frames, sound features extracted based on the audio waveform information of the input video are also utilized. This allows the video description information to be generated by combining multimodal feature information such as images and sounds. At the same time, by setting a reasonable feature extraction method and utilizing the cross-attention mechanism to organically integrate feature information of different modalities, compared to a single type of image feature, the input features are more comprehensive and can more accurately generate video description information about the input video, thereby more accurately describing the actual content of the input video. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0039] Figure 1 A processing flow chart of a multimodal video description generation method provided in an embodiment of the present application;

[0040] Figure 2 A processing flow chart for generating a video description using the solution in an embodiment of the present application;

[0041] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0042] The present application is described in further detail below in conjunction with the accompanying drawings.

[0043] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0044] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0045] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, program devices or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0046] An embodiment of the present application provides a multimodal video description generation method. In addition to utilizing image features extracted based on video frames, the method also utilizes sound features extracted based on audio waveform information of an input video, thereby combining multimodal feature information such as images and sounds to generate video description information. At the same time, by setting a reasonable feature extraction method and utilizing a cross-attention mechanism to organically integrate feature information of different modalities, compared with a single type of image feature, since the input features are more comprehensive, video description information about the input video can be generated more accurately, thereby more accurately describing the actual content of the input video.

[0047] In actual scenarios, the execution subject of the method can be a user device, a network device, or a device formed by integrating a user device and a network device through a network, or it can also be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, and tablet computers; the network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing (Cloud Computing), where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0048] Figure 1 The processing flow of a multimodal video description generation method provided by an embodiment of the present application is shown, and the method at least includes the following processing steps:

[0049] Step S101: extracting video frames from an input video, and separating audio waveform information from the input video.

[0050] The input video refers to the video for which video description information needs to be generated this time. When extracting video frames, the input video can be deframed, and multiple video frames can be extracted from the input video at a preset time interval. The preset time interval can be set according to the needs of the actual application scenario. For example, the preset time interval in this embodiment can be set to 1 second. Therefore, if the input video is a video with a duration of 15 seconds, 1 frame is taken at a time interval of 1 second, and 15 video frames can be extracted, which can be recorded as flame in this embodiment.

[0051] While extracting the video frame, audio waveform information can be separated from the video. The length of the audio waveform information can be the same as the length of the input video. For example, if the length of the input video is 15 seconds, the length of the audio waveform information can also be 15 seconds, which can be recorded as waveform in this embodiment.

[0052] Step S102, adding a video start mark and a video end mark before and after the video frame, and adding an audio start mark and an audio end mark before and after the audio waveform information.

[0053] The video start mark, video end mark, audio start mark and audio end mark can all be preset specific characters. For example, the video start mark in this embodiment can be set to the character<V_s> , the video end mark can be set to the character<V_e> , the audio start mark can be set to the character<W_s> , the audio end mark can be set to the character<W_e> Therefore, taking the above 15 flames and 15 seconds of waveform as an example, when adding the above mark, you can add before the first video frame flame1<V_s> , add after the last video frame flame15<V_e> , and add<W_s> , add later<W_e> The starting and ending positions of each modal information are indicated by pre-set specific characters, which can deepen the understanding of the modal information of each model in the subsequent processing process and improve the accuracy of the processing results.

[0054] Step S103, respectively perform feature encoding on the video start mark, each video frame, video end mark, audio start mark, audio waveform information and audio end mark to obtain the corresponding first feature token, second feature token, third feature token, fourth feature token, fifth feature token and sixth feature token.

[0055] The first feature token refers to the preliminary feature representation information obtained after the video start marker is feature encoded, which can be recorded as token1.<V_s> Linear space embedding is used to perform feature encoding, so as to obtain the corresponding first feature token token1. The second feature token is the preliminary feature representation information obtained after the extracted video frames are feature encoded. Each video frame can correspond to a set of feature tokens, which can be recorded as token2. In this embodiment, the encoder encoder of the same VQVAE (Vector Quantised Variational AutoEncoder) model can be used to perform feature encoding on each video frame respectively, so as to obtain the second feature token token2 corresponding to the number of video frames. If the number of a group of token2 corresponding to each video frame is set to N×M in this embodiment, and the number of video frames is Q, then the total number of token2 obtained in this step is Q×N×M. The third feature token refers to the preliminary feature representation information obtained after the video end marker is feature encoded, which can be recorded as token3. In this embodiment, the video end marker can be<V_e> Linear space embedding is used for feature encoding to obtain the corresponding third feature token token3.

[0056] The fourth feature token refers to the preliminary feature representation information obtained after the audio start mark is feature encoded, which can be recorded as token4.<W_s> Linear space embedding is used to perform feature encoding to obtain the corresponding fourth feature token token4. The fifth feature token is a set of preliminary feature representation information obtained after feature encoding of the extracted audio waveform information, which can be recorded as token5. In this embodiment, the audio waveform information can be feature encoded using an encoder of another VQVAE model to obtain a set of corresponding fifth feature tokens token5. If the number of a set of token5 corresponding to the audio waveform information is set to R in this embodiment, R token5 can be obtained in this step. The sixth feature token refers to the preliminary feature representation information obtained after feature encoding of the audio end marker, which can be recorded as token6. In this embodiment, the audio end marker can be encoded.<W_e> Linear space embedding is used for feature encoding to obtain the corresponding sixth feature token token6.

[0057] Step S104, respectively optimize the first feature token, the second feature token, the third feature token, the fourth feature token, the fifth feature token and the sixth feature token to obtain corresponding first feature information, second feature information, third feature information, fourth feature information, fifth feature information and sixth feature information.

[0058] In actual scenarios, a pre-trained feature optimization model can be used to optimize the features of each feature token, so that the information contained in each feature token can better represent the corresponding content, so as to facilitate the model's understanding of the feature information in the subsequent processing process and improve the accuracy of the subsequent processing. In this embodiment, the feature optimization model can use LLM (Large Language Model). When performing feature optimization processing, the first feature token, the second feature token, the third feature token, the fourth feature token, the fifth feature token and the sixth feature token can be flattened and arranged in order. Taking the aforementioned feature tokens token1, token2, token3, tokens4, token5, and token6 as examples, 1 token1, Q×N×M token2, 1 token3, 1 token4, R token5 and 1 token6 are flattened, and the results are arranged in the order of token1 to token6. Then, it can be input into the large language model, which optimizes its features and obtains the corresponding first feature information F1, second feature information F2, third feature information F3, fourth feature information F4, fifth feature information F5 and sixth feature information F6. This can achieve feature optimization, which is beneficial to subsequent processing and improves the accuracy of the results.

[0059] Step S105, using the cross attention mechanism, the first feature information, the fourth feature information, the fifth feature information and the sixth feature information are feature-fused to obtain the head feature. In this implementation, the head feature can be recorded as headfeature, and through the poor attention mechanism, the image information in the video frame, the sound information in the audio waveform information, and the character information in the start, end and other marks can be deeply fused, so that the head feature can effectively fuse the above-mentioned different types of modal information and can be better used to generate video description information.

[0060] Specifically, when using the cross-attention mechanism for feature fusion, the Q vector feature of the cross-attention mechanism model can be obtained according to the first feature information. In this embodiment, the MLP (Multi-Layer Perceptron) model can be used to generate the Q vector feature. At the same time, the fourth feature information, the fifth feature information and the sixth feature information can be processed by global average pooling to obtain the audio feature information. In addition, the audio feature information can be input into two different self-attention mechanism models respectively to obtain the K vector feature and V vector feature of the cross-attention mechanism model. Then, the K vector feature, the V vector feature and the Q vector feature are input into the cross-attention mechanism model to obtain the head feature that deeply integrates the information contained in the video frame, audio and characters.

[0061] In some embodiments of the present application, in order to further optimize the generation result of the video description and make the generated video description information more flexible, a certain randomness can be appropriately introduced into the feature information so that the generation result of the video description information has an expected degree of random variation, avoiding the generation result being too rigid, and improving the user experience. Therefore, in this embodiment, the first feature information, the fourth feature information, the fifth feature information and the sixth feature information can be feature fused by using the cross-attention mechanism, and before obtaining the head feature, Gaussian noise information can be added to the first feature information, the second feature information, the fourth feature information, the fifth feature information and the sixth feature information, and the first feature information, the second feature information, the fourth feature information, the fifth feature information and the sixth feature information after the Gaussian noise information is added are denoised, so that a certain amount of noise information is introduced into each feature information finally obtained, but it will not affect the accurate expression of the video description, so that the generation result has an expected degree of random variation, further improving the user experience. In actual scenarios, a pre-trained Unet model can be used for denoising. By inputting the first feature information, the second feature information, the fourth feature information, the fifth feature information and the sixth feature information after adding Gaussian noise information into the corresponding Unet model respectively, each feature information can be denoised and the expected feature information F-noise can be output.

[0062] Step S106: splice the head feature and the second feature information and input them into a preset video description generation model to obtain video description information of the input video.

[0063] In addition to utilizing image features extracted based on video frames, this solution also utilizes sound features extracted based on the audio waveform information of the input video. Therefore, it is possible to combine multimodal feature information such as images and sounds to generate video description information. At the same time, by setting a reasonable feature extraction method and utilizing a cross-attention mechanism to organically integrate feature information of different modalities, compared with a single type of image features, since the input features are more comprehensive, it is possible to more accurately generate video description information about the input video, thereby more accurately describing the actual content of the input video.

[0064] In some embodiments of the present application, the video description generation model may also use a pre-trained large language model. Therefore, after the head feature and the second feature information are spliced ​​and input into the preset video description generation model, when obtaining the video description information of the input video, the head feature and the second feature information may be spliced ​​first to obtain the description generation feature. In this embodiment, the description generation feature may be named A, and then the following iterative processing may be performed based on the description generation feature:

[0065] First, the description generating feature A is input into the preset large language model to obtain the output information of the first position of the large language model. Since the input description generating feature is composed of the head feature and the second feature information, the output information of the first position of the large language model is the input information corresponding to the head feature position. Then, the output character corresponding to the output information is determined by the preset classification function. Among them, the classification function can use the softmax function, thereby completing the classification judgment of the output information, thereby obtaining the output character char corresponding to the output information. Then, the output character char is feature encoded, and the result of the feature encoding is spliced ​​to the back of the head feature and the second feature information to obtain a new description generating feature. Among them, when the output character char is feature encoded, the same linear space embedding method as mentioned above can be used to implement it, thereby obtaining the corresponding feature encoding result. If the feature encoding result obtained after the output character char is feature encoded is named B, then B is spliced ​​to the back of the original description generating feature A (head feature and the second feature information), and a new description generating feature A can be obtained for subsequent iterative processing. Among them, the stopping condition of the iterative processing is that the large language model outputs a stop symbol.

[0066] At this time, all the output characters char obtained by the large language model before outputting the stop sign can be combined in sequence to generate the video description information of the input video. In actual scenarios, the final generated video description information can be "someone doing something at a certain place", "something happened at a certain place", etc.

[0067] In some embodiments of the present application, the result of feature coding is spliced ​​to the back of the head feature and the second feature information, and before obtaining the new description generation feature, Gaussian noise information can also be added to the result of feature coding, and the result of feature coding after adding Gaussian noise information is denoised. Its purpose is the same as the purpose of adding Gaussian noise information to the first feature information, the fourth feature information, the fifth feature information, and the sixth feature information, which is to make the generated video description information more flexible, so that the generated result of the video description information has an expected degree of random changes, avoid the generated result being too rigid, and improve the user experience.

[0068] Figure 2 The processing flow for generating a video description using the solution provided in the embodiment of the present application is shown, including the following processing steps:

[0069] Step S201, deframe the input video, take 1 frame per second, and a total of Q video frames flame.

[0070] Step S202, separating audio waveform information from the input video, the duration of which is the same as the duration of the video, to obtain audio waveform information waveform.

[0071] Step S203, add before the first video frame frame1<V_s> Character, as the video start marker, added after the Qth video frame frameQ<V_e> Character, as the video end mark; add before the waveform<W_s> Characters, add later<W_e> , as the audio start marker and audio end marker respectively.

[0072] Step S204:<V_s> Perform linear space embedding and get the output, named token1. In addition,<V_e> ,<W_s> and<W_e> Linear spatial embedding is also performed to obtain outputs, named token3, token4, and token6. The encoder of the same VQVAE model is used to encode the features of each video frame to obtain a discrete token representation. Each video frame obtains N×M tokens, totaling Q×N×M, named token2. The encoder of another VQVAE model is used to encode the features of the waveform to obtain a discrete set of token representations, namely R token5.

[0073] Step S205: Flatten 1 token1, Q×N×M token2, 1 token3, 1 token4, R token5 and 1 token6, and arrange the results in the order of token1 to token6. Then, they can be input into the large language model, which performs feature optimization on them to obtain the corresponding first feature information F1, second feature information F2, third feature information F3, fourth feature information F4, fifth feature information F5 and sixth feature information F6.

[0074] Step S206, adding Gaussian noise information to F1, F2, F4, F5 and F6 to increase the diversity of the output video description information, and then using different unet models to denoise the feature information after adding the noise information.

[0075] Step S207: After denoising, F1 is encoded by MLP and used as the Q vector feature of the cross-attention model; after global average pooling, F4, F5 and F6 are input into two different self-attention models respectively, and the two outputs are used as the K vector feature and V vector feature of the cross-attention model respectively. Then, the Q vector feature, K vector feature and V vector feature are input into the cross-attention model together to obtain the head feature.

[0076] Step S208, concatenate the head feature and F2 to obtain a description generating feature, named A, and output it to the trained LLM model for processing. Take the output information corresponding to the head feature position in the output of the LLM model, and obtain the first output character char after processing with the softmax function. Then, the char can be linearly embedded in the same way as S204 to obtain a feature encoding result, and Gaussian noise information is added in the same way as mentioned above, and after denoising with the unet model, it is concatenated to the original description generating feature A to obtain a new description generating feature. At this time, the new description generating feature A can be input into the LLM again for iterative processing until the stop condition of the iterative processing is met. In this embodiment, the stop condition can be that the LLM outputs a stop symbol, and all the chars that can be obtained at this time are sequentially composed of text in the order of output as the video description information describe of the input video.

[0077] In this scheme, in addition to utilizing image features extracted based on video frames, sound features extracted based on the audio waveform information of the input video are also utilized. This allows the video description information to be generated by combining multimodal feature information such as images and sounds. At the same time, by setting a reasonable feature extraction method and utilizing the cross-attention mechanism to organically integrate feature information of different modalities, compared to a single type of image feature, the input features are more comprehensive and can more accurately generate video description information about the input video, thereby more accurately describing the actual content of the input video.

[0078] Based on another aspect of the present application, an embodiment of the present application also provides a multimodal video description generation device, which includes a memory for storing computer program instructions and a processor for executing computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the aforementioned multimodal video description generation method.

[0079] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the processing unit, the above functions defined in the method of the present application are executed.

[0080] It should be noted that the computer-readable medium described in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0081] In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0082] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0083] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0084] As another aspect, the present application further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more computer program instructions, which may be executed by a processor to implement the methods and / or technical solutions of the above multiple embodiments of the present application.

[0085] It should be noted that the present application can be implemented in software and / or a combination of software and hardware, for example, can be implemented using an application specific integrated circuit (ASIC), a general purpose computer or any other similar hardware device. In certain embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including related data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented using hardware, for example, as a circuit that cooperates with a processor to perform each step or function.

[0086] It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or basic features of the present application. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is limited by the attached claims rather than the above description, so it is intended to include all changes that fall within the meaning and scope of the equivalent elements of the claims in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to represent names, and do not represent any specific order. The numbers corresponding to the steps are used to mark and distinguish different steps, and the size of the numbers does not limit any specific execution order.

Claims

1. A multimodal video description generation method, characterized in that: The method comprises: Extracting video frames from an input video and separating audio waveform information from the input video; Adding a video start mark and a video end mark before and after the video frame, respectively, and adding an audio start mark and an audio end mark before and after the audio waveform information, respectively; Perform feature encoding on the video start mark, each video frame, video end mark, audio start mark, audio waveform information and audio end mark respectively to obtain corresponding first feature token, second feature token, third feature token, fourth feature token, fifth feature token and sixth feature token; Respectively optimizing the first feature token, the second feature token, the third feature token, the fourth feature token, the fifth feature token, and the sixth feature token to obtain corresponding first feature information, second feature information, third feature information, fourth feature information, fifth feature information, and sixth feature information; Using a cross attention mechanism, the first feature information, the fourth feature information, the fifth feature information and the sixth feature information are fused to obtain a head feature; The head feature and the second feature information are spliced ​​and input into a preset video description generation model to obtain video description information of the input video.

2. The method according to claim 1, characterized in that: Extract video frames from the input video, including: A plurality of video frames are extracted from the input video at preset time intervals.

3. The method according to claim 1, characterized in that The video start mark, each video frame, the video end mark, the audio start mark, the audio waveform information and the audio end mark are respectively feature-encoded to obtain the corresponding first feature token, second feature token, third feature token, fourth feature token, fifth feature token and sixth feature token, including: The video start mark, the video end mark, the audio start mark and the audio end mark are respectively feature encoded by linear space embedding to obtain the corresponding first feature token, third feature token, fourth feature token and sixth feature token; Use the encoder of the same VQVAE model to perform feature encoding on each video frame to obtain a set of corresponding second feature tokens; The audio waveform information is feature encoded using an encoder of another VQVAE model to obtain a set of corresponding fifth feature tokens.

4. The method according to claim 1, characterized in that: The first feature token, the second feature token, the third feature token, the fourth feature token, the fifth feature token, and the sixth feature token are respectively optimized to obtain corresponding first feature information, second feature information, third feature information, fourth feature information, fifth feature information, and sixth feature information, including: The first feature token, the second feature token, the third feature token, the fourth feature token, the fifth feature token and the sixth feature token are flattened and arranged in order, and then input into a large language model, which performs feature optimization on them to obtain corresponding first feature information, second feature information, third feature information, fourth feature information, fifth feature information and sixth feature information.

5. The method according to claim 1, characterized in that Before the cross attention mechanism is used to fuse the first feature information, the fourth feature information, the fifth feature information and the sixth feature information to obtain the head feature, the method further includes: Adding Gaussian noise information to the first feature information, the second feature information, the fourth feature information, the fifth feature information, and the sixth feature information; The first feature information, the second feature information, the fourth feature information, the fifth feature information and the sixth feature information after adding the Gaussian noise information are denoised.

6. The method according to claim 1 or 5, characterized in that: Using the cross attention mechanism, the first feature information, the fourth feature information, the fifth feature information and the sixth feature information are fused to obtain head features, including: According to the first feature information, obtain the Q vector feature of the cross attention mechanism model; Performing global average pooling processing on the fourth feature information, the fifth feature information, and the sixth feature information to obtain audio feature information; Input the audio feature information into two different self-attention mechanism models respectively to obtain K vector features and V vector features of the cross-attention mechanism model; The K vector features, V vector features and Q vector features are input into the cross attention mechanism model to obtain the head features.

7. The method according to claim 1, characterized in that The video description generation model is a large language model; The head feature and the second feature information are spliced ​​and input into a preset video description generation model to obtain video description information of the input video, including: Concatenate the head feature and the second feature information to obtain a description generation feature; The following iterative processing is performed based on the description generation feature: the description generation feature is input into a preset large language model to obtain output information of the first position of the large language model; an output character corresponding to the output information is determined by a preset classification function; the output character is feature-encoded, and the result of the feature encoding is concatenated to the back of the head feature and the second feature information to obtain a new description generation feature, and the stopping condition of the iterative processing is that the large language model outputs a stop symbol; The generated output characters are combined in sequence to generate video description information of the input video.

8. The method according to claim 7, characterized in that The feature encoding result is spliced ​​to the head feature and the second feature information, and before obtaining a new description generation feature, the method further includes: Adding Gaussian noise information to the result of the feature encoding; The result of feature encoding after adding Gaussian noise information is denoised.

9. A multimodal video description generating device, wherein: The device comprises a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the method according to any one of claims 1 to 8.

10. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 8.

Images