Video description generation method and apparatus, electronic device, and storage medium

Abstract

Embodiments of the present disclosure provide a video description generation method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring compression domain information of a video to be processed, wherein the compression domain information comprises key frame information, motion information, and residual information; on the basis of the compression domain information, acquiring a compression domain feature, wherein the compression domain feature comprises key frame features, a motion feature, and a residual feature; and fusing the key frame features, the motion feature, and the residual feature to obtain a target feature, and inputting the key frame features and the target feature into a video description decoder so as to obtain a video description corresponding to said video, wherein the target feature comprises the motion feature, the residual feature, and a feature, which is concerned by the motion feature and the residual feature, among the key frame features.

Description

Video description generation method, device, electronic device and storage medium

[0001] This application claims priority to the Chinese invention patent application entitled “A video description generation method, device, electronic device and storage medium” filed on September 20, 2023, with application number 202311220060.X. The entire contents of that application are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present disclosure relate to computer vision and natural language processing technologies, and more particularly to a method, device, electronic device, and storage medium for generating a video description. Background Art

[0003] Video descriptions are natural language sentences that describe the content of a video. Research on video descriptions can improve computer vision and language understanding capabilities and play an important role in application scenarios such as accessibility services, content search, and recommendations.

[0004] The video description generation scheme in the related art usually decodes the video first, then inputs the decoded video frames into the video description model, uses the video description model to understand the complex visual content in the video based on the decoded video frames, and converts the complex visual content into a natural and fluent language description, that is, obtaining the video description. The processing process is complex and consumes a lot of computing power resources.

[0005] Summary of the Invention

[0006] The present disclosure provides a video description generation method, device, electronic device and storage medium, which can solve the problem that the video description generation process is complex and consumes a lot of computing resources.

[0007] In a first aspect, an embodiment of the present disclosure provides a video description generation method, comprising: obtaining compressed domain information of a video to be processed, wherein the compressed domain information includes key frame information, motion information and residual information; obtaining compressed domain features based on the compressed domain information, wherein the compressed domain features include key frame features, motion features and residual features; fusing the key frame features, motion features and residual features to obtain target features, inputting the key frame features and target features into a video description decoder to obtain a video description corresponding to the video to be processed, wherein the target features include motion features, residual features and features of the key frame features that are focused on by the motion features and the residual features.

[0008] In the second aspect, the embodiments of the present disclosure also provide a video description generation device, which includes: an information acquisition module, used to obtain compressed domain information of the video to be processed, wherein the compressed domain information includes key frame information, motion information and residual information; a feature acquisition module, used to obtain compressed domain features based on the compressed domain information, wherein the compressed domain features include key frame features, motion features and residual features; a video description generation module, used to fuse the key frame features, motion features and residual features to obtain target features, input the key frame features and target features into a video description decoder, and obtain a video description corresponding to the video to be processed, wherein the target features include motion features, residual features and features of the key frame features that are focused on by motion features and residual features.

[0009] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the video description generation method as described in any embodiment of the present disclosure.

[0010] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium comprising computer-executable instructions, which, when executed by a computer processor, are used to execute the video description generation method as described in any embodiment of the present disclosure.

[0011] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the video description generating method as described in any embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

[0013] FIG1 is a flow chart of a method for generating a video description according to an embodiment of the present disclosure;

[0014] FIG2 is a schematic diagram of a compressed domain information extraction process in a video description generation method provided by an embodiment of the present disclosure;

[0015] FIG3 is a flow chart of another method for generating a video description according to an embodiment of the present disclosure;

[0016] FIG4 is a schematic diagram of the structure of a cross-attention module provided by an embodiment of the present disclosure;

[0017] FIG5 is a schematic diagram of the structure of a target network provided by an embodiment of the present disclosure;

[0018] FIG6 is a flow chart of another method for generating a video description according to an embodiment of the present disclosure;

[0019] FIG7 is a schematic structural diagram of a video description generating apparatus provided by an embodiment of the present disclosure;

[0020] FIG8 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0021] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0022] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0023] As used herein, the term "including" and its variations are open-ended, i.e., "including but not limited to." The term "based on" means "based, at least in part, on." The term "one embodiment" means "at least one embodiment," the term "another embodiment" means "at least one additional embodiment," and the term "some embodiments" means "at least some embodiments." Other terms are defined in the following description.

[0024] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0025] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, they should be understood as "one or more".

[0026] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0027] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) must comply with the requirements of relevant laws, regulations and relevant provisions.

[0028] Existing video description methods often require first decoding the video, then inputting the decoded video frames into a video description model to generate the video description. Decoded video frames typically include I frames (intra-coded frames, key frame information), B frames (bidirectional frames, bidirectionally predicted frames), and P frames (predicted frames). This approach has the following issues in terms of computing resource consumption, time consumption, quality loss, and hardware dependence:

[0029] In terms of computing resource consumption, since video decoding is a computationally intensive operation, it may consume a large amount of computing resources (CPU, GPU or memory), especially when processing high-definition, high-frame-rate videos or multi-channel video streams, which will consume more computing resources and may affect other computing tasks.

[0030] In terms of time consumption, since the video decoding process may be very slow, this is a serious problem for video content that needs to be processed in real time (such as real-time video description).

[0031] In terms of quality loss, during the video encoding process, some lossy compression techniques are often used to compress the video to achieve smaller storage space. During the video decoding process, this discarded information is usually impossible to recover, which may affect subsequent video analysis.

[0032] In terms of hardware dependency, some efficient decoders may require special hardware support (such as certain types of GPUs, etc.), which may increase hardware dependency and raise the hardware threshold.

[0033] Figure 1 is a flow chart of a method for generating a video description provided by an embodiment of the present disclosure. This embodiment of the present disclosure is applicable to generating text descriptions of videos. The method can be performed by a video description generation device, which can be implemented in software and / or hardware. Optionally, the device can be implemented by an electronic device, such as a mobile terminal, a PC, or a server.

[0034] As shown in FIG1 , the method includes:

[0035] S110: Obtain compression domain information of the video to be processed, wherein the compression domain information includes key frame information, motion information, and residual information.

[0036] In some embodiments, the video to be processed may be video data in a compressed format. For example, the video to be processed may be video data in a compressed format obtained by compressing video frames using a video compression algorithm. The compressed domain information of the video to be processed may include key frame information, motion information, and residual information.

[0037] Because adjacent video frames are relatively similar, a video compression algorithm uses this similarity to divide the video into a series of groups of pictures (GOPs). The RGB image of the first frame in each GOP serves as the key frame (I-frame), and the motion information of subsequent frames relative to the I-frame is recorded, as well as the residual information of subsequent frames relative to the I-frame.

[0038] Figure 2 is a schematic diagram of the compressed domain information extraction process in a video description generation method provided by an embodiment of the present disclosure. As shown in Figure 2, the process of extracting keyframe information, motion information, and residual information is demonstrated using compressed domain information extraction from a GOP as an example. The first frame of the GOP is obtained as an I-frame. For the remaining B-frames or P-frames in the GOP, motion information and residual information are extracted using dequantization.

[0039] Exemplarily, the first frame of each image group in the video to be processed is obtained as key frame information. Motion information is obtained based on relative displacement data of pixels between the key frame information and adjacent frames. Residual information is obtained based on color differences of pixels between the key frame information and adjacent frames.

[0040] S120. Obtain compression domain features based on the compression domain information, wherein the compression domain features include key frame features, motion features, and residual features.

[0041] In the embodiments of the present disclosure, the compressed domain features may be features of the compressed domain information extracted by a network model, wherein the network model may be an encoder constructed by a neural network, etc. The embodiments of the present disclosure do not limit the use of a network model to extract the compressed domain features of the compressed domain information.

[0042] In the disclosed embodiment, the compressed domain information includes key frame information, motion information, and residual information. Then, the key frame information is input into the first encoder to obtain the key frame features output by the first encoder; the motion information is input into the second encoder to obtain the motion features output by the second encoder; and the residual information is input into the third encoder to obtain the residual features output by the third encoder. The first encoder is a heavyweight feature extraction network, and the second and third encoders are lightweight feature extraction networks. Using a lightweight feature extraction network to extract motion features from motion information and residual features from residual information can improve the feature extraction rate and shorten the feature extraction time.

[0043] In some embodiments, the first encoder may be a heavyweight network trained on sample pairs consisting of I frames and I frame features, and the I frame features of the I frame are extracted through the visual modality branch of the trained first encoder. Similarly, the second encoder may be a lightweight network trained on sample pairs consisting of motion information and motion features, and the motion features of the motion information are extracted through the visual modality branch of the trained second encoder. Similarly, the third encoder may be a lightweight network trained on sample pairs consisting of residual information and residual features, and the residual features of the residual information are extracted through the visual modality branch of the trained third encoder.

[0044] It should be noted that the above-mentioned first to third encoders can be encoder networks in the Transformer network architecture, or other neural network architectures, and the embodiments of the present disclosure do not specifically limit this.

[0045] S130 , fusing the key frame features, motion features, and residual features to obtain target features, and inputting the key frame features and target features into a video description decoder to obtain a video description corresponding to the video to be processed.

[0046] In some embodiments, the target features include motion features, residual features, and features of the key frame features that are focused on by the motion features and the residual features. Optionally, the cross-attention module can be used to fuse the key frame features, motion features, and residual features to obtain the target features. For example, the motion features and the residual features are superimposed to obtain a query vector of the cross-attention module. Optionally, the motion features and the residual features can be fused to obtain a first fused feature. It should be noted that the method of superimposing the motion features and the residual features can be to add the two together, and the embodiment of the present disclosure does not limit the superposition method of the motion features and the residual features. The key frame features are used as the key vector and value vector of the cross-attention module. The cross-attention module is used to determine the degree of attention of the query vector to the key frame features.

[0047] As an example, the similarity between the first fused feature and the keyframe feature is calculated through a cross-attention module, and the similarity is processed using softmax to obtain the attention weight corresponding to the keyframe feature. The weight corresponding to the keyframe feature is multiplied by the keyframe and summed to obtain the second fused feature. The second fused feature can be a feature in the keyframe feature that is focused on by the motion feature and the residual feature. The first fused feature and the second fused feature are fused by setting a neural network to generate a target feature. The set neural network can include a self-attention module and a fully connected feedforward network.

[0048] In the disclosed embodiment, the video description decoder can be a neural network that generates a video description based on key frame features and target features of key frame information in an undecoded video. For example, the video description decoder can be a neural network trained on a video-text pair consisting of a video and a video description. The key frame features and target features corresponding to the video to be processed are input into the video description decoder. Since the target features integrate motion features, residual features, and features of the key frame features that are focused on by the motion features and residual features, the target features contain rich video frame information. The video description decoder can generate a video description corresponding to the video to be processed based on the key frame features and the target features.

[0049] The technical solution of the disclosed embodiments uses compressed domain information from the undecoded video to obtain keyframe features, motion features, and residual features. These features are then fused to generate target features. The keyframe and target features are then input into a video description decoder to generate a video description corresponding to the video being processed. This allows for the generation of video descriptions using compressed domain information, thereby resolving issues such as high computational resource consumption, time consumption, quality loss, and hardware dependency associated with video decoding.

[0050] FIG3 is a flow chart of another method for generating video descriptions provided by an embodiment of the present disclosure. Based on the above embodiment, the present disclosure embodiment specifically defines the steps of fusing the key frame features, motion features, and residual features to obtain target features. As shown in FIG3 , the method includes:

[0051] S310: Obtain compression domain information of the video to be processed, wherein the compression domain information includes key frame information, motion information, and residual information.

[0052] S320. Obtain compression domain features based on the compression domain information, wherein the compression domain features include key frame features, motion features, and residual features.

[0053] S330: Fusing the motion feature and the residual feature to obtain a first fused feature.

[0054] For example, if the motion feature and the residual feature are vectors of the same dimension, fusing the motion feature and the residual feature is to calculate the vector sum of the motion feature and the residual feature and use this vector sum as the first fused feature. If the motion feature and the residual feature are vectors of different dimensions, dimensionality reduction techniques are used to make the dimensions of the motion feature and the residual feature the same.

[0055] S340: Using the key frame feature and the first fusion feature as input to a cross-attention module to obtain a second fusion feature output by the cross-attention module.

[0056] In some embodiments, the keyframe features and the first fused features have the same dimension.

[0057] Exemplarily, the key frame features and the first fused features are used as inputs of a cross-attention module, and the attention weights of the key frame features are calculated by the cross-attention module; and the second fused features are generated based on the attention weights and the key frame features by the cross-attention module.

[0058] Optionally, the key frame feature and the first fused feature are used as inputs of a cross-attention module, and the attention weight of the key frame feature is calculated by the cross-attention module, including: inputting the key frame feature and the first fused feature into the cross-attention module, calculating the attention score of the key frame feature and the first fused feature by the cross-attention module. The attention weight of the key frame feature is obtained by performing a normalization operation on the attention score by the cross-attention module.

[0059] Figure 4 is a structural diagram of a cross-attention module provided by an embodiment of the present disclosure. As shown in Figure 4, the input of the cross-attention module includes key frame information I and a first fusion feature, and the key frame information I is used as the key vector V and the value vector K, and the first fusion feature is used as the query vector Q. The cross-attention module calculates the dot product of the key frame information I and the first fusion feature to obtain the attention score of the key frame feature, where the attention score is expressed as K·Q. The attention weight W corresponding to the attention score is calculated by the normalization layer in the cross-attention module. The normalization layer can be a softmax layer, etc. Then, the key frame information is multiplied by the corresponding attention weight through the cross-attention module and the sum is obtained, where the second fusion feature is expressed as ∑V·W.

[0060] S350: Generate a target feature based on the first fusion feature and the second fusion feature.

[0061] Exemplarily, the first fusion feature and the second fusion feature are added and input into the self-attention module to obtain a third fusion feature output by the self-attention module; the third fusion feature is input into a fully connected feedforward network, and a target feature is generated according to the output feature of the fully connected feedforward network.

[0062] In some embodiments, the third fused feature is an enhanced feature calculated by the self-attention module based on the first fused feature and the second fused feature.

[0063] Exemplarily, the target feature can be obtained by fusing key frame features, motion features and residual features through a target network with a set number of layers. Figure 5 is a structural diagram of a target network provided by an embodiment of the present disclosure. As shown in Figure 5, the target network includes a cross-attention module, a self-attention module and a fully connected feedforward network. The first fusion feature corresponding to the motion feature and the residual feature is input into the cross-attention module, and the second fusion feature is output through the cross-attention module. Then, the first fusion feature is connected to the output end of the cross-attention module through a residual connection, and is added to the second fusion feature output by the cross-attention module as the input of the self-attention module. The first fusion feature and the second fusion feature are weightedly fused through the self-attention module to obtain a third fusion feature. The target feature is determined based on the third fusion feature through a fully connected feedforward network.

[0064] It should be noted that Figure 5 only shows the process of obtaining target features by fusing keyframe features, motion features, and residual features through a target network layer. However, the present disclosure does not limit the number of target network layers, and the number of target network layers can be flexibly selected according to the specific task. If a multi-layer target network is set, the output of the fully connected feedforward network of the previous target network layer is used as the input of the cross-attention module of the next target network layer.

[0065] In some embodiments, the target feature can simulate B-frame features and P-frame features. B-frame is a bidirectional prediction frame, which contains the change in the image from the previous frame and the next frame. P-frame is a prediction frame, which contains the difference data of the image (i.e., the change from the previous frame). While storing image information, the P-frame also records the change in the image. Since the target feature is a feature obtained by fusing the second fusion feature with the first fusion feature, and the first fusion feature is a feature obtained by fusing the motion feature with the residual feature, the second fusion feature represents the feature of the key frame feature that is focused on by the motion feature and the residual feature, therefore, the target feature fuses the motion feature, the residual feature, and the feature of the key frame feature that is focused on by the motion feature and the residual feature. Based on the definition of B-frame and P-frame, it can be seen that the B-frame feature and the P-frame feature can be simulated by the target feature.

[0066] Optionally, the number of target networks in the above example may be increased so that the target features contain rich feature information, or the number of target networks in the above example may be reduced, which is not specifically limited in the embodiments of the present disclosure.

[0067] S360: Input the key frame features and target features into a video description decoder to obtain a video description corresponding to the video to be processed.

[0068] The technical solution of the embodiment of the present disclosure inputs the first fusion feature and the key frame feature after the motion feature and the residual feature are fused into the cross-attention module, and generates the target feature based on the second fusion feature output by the cross-attention module and the first fusion feature. In this way, the key frame feature and the first fusion feature are fused together to enhance the features of the key frame features that are focused on by the motion feature and the residual feature, and obtain the second fusion feature. The second fusion feature represents the features of the key frame features that are focused on by the motion feature and the residual feature, and the target feature is generated based on the second fusion feature and the first fusion feature, so as to simulate the B frame feature and the P frame feature through the target feature. Therefore, there is no need to decode the video to be processed. By inputting the key frame information and the target feature into the video description decoder, the video description corresponding to the video to be processed can be obtained.

[0069] FIG6 is a flow chart of another method for generating a video description provided by an embodiment of the present disclosure. Based on the above embodiment, the present disclosure embodiment provides a flow chart of a specific example. As shown in FIG6 , the method includes:

[0070] S601, start.

[0071] S602: Input the video to be processed.

[0072] The video to be processed is a video to be processed. In the disclosed embodiment, the video to be processed can be a compressed video downloaded from the network. Alternatively, the video to be processed can be a compressed video shot by a user, or a compressed video generated by video editing software.

[0073] S603: Obtain compression domain information.

[0074] Exemplarily, compressed domain information of the video to be processed is obtained, wherein the compressed domain information includes key frame information, motion information, and residual information.

[0075] S604: Acquire key frame features.

[0076] S605: Obtain motion features.

[0077] S606: Obtain residual features.

[0078] S607: Fusing the key frame features, motion features, and residual features to obtain target features.

[0079] First, the motion features and residual features are fused to obtain a first fused feature. The keyframe features and the first fused feature are then used as inputs to the cross-attention module. The cross-attention module fuses the keyframe features, motion features, and residual features to obtain a target feature. This target feature contains motion features, residual features, and features in the keyframe features that are focused on by the motion and residual features. This target feature can simulate the B-frame and P-frame features of the video to be processed.

[0080] S608: Input the key frame features and the target features into the video description decoder.

[0081] S609: The video description decoder generates a video description.

[0082] Exemplarily, the I frame features, the simulated B frame features, and the P frame features are input into a video description decoder, and the video description corresponding to the video to be processed is generated by the video description decoder.

[0083] S610, end.

[0084] The video description method provided by the embodiment of the present disclosure does not require video decoding and can directly generate video description using compressed domain information, thereby avoiding the problems of large computing resource consumption, high time consumption, quality loss, and hardware dependence caused by video decoding.

[0085] Figure 7 is a schematic diagram of the structure of a video description generation device provided in an embodiment of the present disclosure. This device is used to execute the video description generation method described in any embodiment of the present disclosure. The device is implemented in software and / or hardware. Optionally, it is implemented in an electronic device, such as a mobile terminal, a PC, or a server. As shown in Figure 7, the device includes an information acquisition module 710, a feature acquisition module 720, and a video description generation module 730.

[0086] The information acquisition module 710 is configured to acquire compressed domain information of the video to be processed, wherein the compressed domain information includes key frame information, motion information, and residual information.

[0087] The feature acquisition module 720 is configured to acquire compressed domain features based on the compressed domain information, wherein the compressed domain features include key frame features, motion features, and residual features.

[0088] The video description generation module 730 is used to fuse the key frame features, motion features and residual features to obtain target features, input the key frame features and target features into a video description decoder, and obtain a video description corresponding to the video to be processed, wherein the target features include motion features, residual features and features in the key frame features that are focused on by the motion features and residual features.

[0089] The technical solution provided by the disclosed embodiments uses compressed domain information from the undecoded video to obtain keyframe features, motion features, and residual features, and fuses these features to generate target features. The keyframe features and target features are then input into a video description decoder to generate a video description corresponding to the video being processed. This allows for the generation of video descriptions using compressed domain information, resolving issues such as high computational resource consumption, time consumption, quality loss, and hardware dependency associated with video decoding.

[0090] Optionally, the video description generating module 730 includes:

[0091] The first fusion feature determination unit is configured to fuse the motion feature and the residual feature to obtain a first fusion feature.

[0092] The second fusion feature determination unit is used to take the key frame feature and the first fusion feature as input of the cross-attention module to obtain the second fusion feature output by the cross-attention module, wherein the second fusion feature is the feature in the key frame feature that is focused on by the motion feature and the residual feature.

[0093] A target feature generating unit is configured to generate a target feature based on the first fusion feature and the second fusion feature.

[0094] Furthermore, the second fusion feature determination unit is specifically used to: take the key frame feature and the first fusion feature as input of the cross-attention module, calculate the attention weight of the key frame feature through the cross-attention module; and generate the second fusion feature based on the attention weight and the key frame feature through the cross-attention module.

[0095] Optionally, the key frame feature and the first fusion feature are used as inputs of a cross-attention module, and the attention weight of the key frame feature is calculated through the cross-attention module, including: inputting the key frame feature and the first fusion feature into the cross-attention module, and calculating the attention score of the key frame feature and the first fusion feature through the cross-attention module; and performing a normalization operation on the attention score through the cross-attention module to obtain the attention weight of the key frame feature.

[0096] Optionally, generating a second fused feature based on the attention weight and the key frame feature through the cross-attention module includes: weighting the key frame feature based on the attention weight of the key frame feature through the cross-attention module to obtain the second fused feature.

[0097] Optionally, the target feature generation unit is specifically used to: add the first fusion feature and the second fusion feature and input them into the self-attention module to obtain a third fusion feature output by the self-attention module; input the third fusion feature into the fully connected feedforward network, and generate the target feature according to the output feature of the fully connected feedforward network.

[0098] Optionally, the feature acquisition module 720 is specifically used to: input the key frame information into the first encoder to obtain the key frame features output by the first encoder; input the motion information into the second encoder to obtain the motion features output by the second encoder; input the residual information into the third encoder to obtain the residual features output by the third encoder; wherein the first encoder is a heavyweight feature extraction network, and the second encoder and the third encoder are lightweight feature extraction networks.

[0099] The video description generation device provided in the embodiments of the present disclosure can execute the video description generation method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0100] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0101] FIG8 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Referring to FIG8 , a schematic diagram of the structure of an electronic device (such as a terminal device or server in FIG8 ) 800 suitable for implementing an embodiment of the present disclosure is shown below. The terminal device in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. The electronic device shown in FIG8 is merely an example and should not impose any limitations on the functions and scope of use of the embodiments of the present disclosure.

[0102] As shown in FIG8 , the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An edit / output (I / O) interface 805 is also connected to the bus 804.

[0103] Typically, the following devices may be connected to the I / O interface 805: an input device 806 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 808 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although FIG8 shows the electronic device 800 with various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0104] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 809, or installed from the storage device 808, or installed from the ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0105] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0106] The electronic device provided by the embodiment of the present disclosure and the video description generation method provided by the above embodiment belong to the same inventive concept. For technical details not fully described in this embodiment, please refer to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0107] An embodiment of the present disclosure provides a computer storage medium having a computer program stored thereon. When the program is executed by a processor, the video description generation method provided in the above embodiment is implemented.

[0108] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0109] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0110] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0111] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0112] Obtaining compression domain information of the video to be processed, wherein the compression domain information includes key frame information, motion information, and residual information;

[0113] Acquire compressed domain features based on the compressed domain information, wherein the compressed domain features include key frame features, motion features, and residual features;

[0114] The key frame features, motion features and residual features are fused to obtain target features, and the key frame features and target features are input into a video description decoder to obtain a video description corresponding to the video to be processed, wherein the target features include motion features, residual features and features of the key frame features that are focused on by motion features and residual features.

[0115] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0117] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

[0118] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0119] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0121] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0122] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A method for generating a video description, comprising: Acquire compression domain information of the video to be processed, wherein the compression domain information includes key frame information, motion information and residual information; Acquire compression domain features based on the compression domain information, wherein the compression domain features include key frame features, motion features, and residual features; The key frame features, motion features and residual features are fused to obtain target features, and the key frame features and target features are input into a video description decoder to obtain a video description corresponding to the video to be processed, wherein the target features include motion features, residual features and features of the key frame features that are focused on by motion features and residual features.

2. The method according to claim 1, wherein the step of fusing the key frame features, motion features and residual features to obtain target features comprises: Fusing the motion feature with the residual feature to obtain a first fused feature; The key frame feature and the first fused feature are used as inputs of a cross-attention module to obtain a second fused feature output by the cross-attention module, wherein the second fused feature is a feature of the key frame feature that is focused on by the motion feature and the residual feature; A target feature is generated according to the first fused feature and the second fused feature.

3. The method according to claim 2, wherein the step of using the key frame feature and the first fused feature as input of a cross-attention module to obtain a second fused feature output by the cross-attention module comprises: Using the key frame feature and the first fusion feature as inputs of a cross-attention module, and calculating the attention weight of the key frame feature through the cross-attention module; The cross-attention module generates a second fusion feature based on the attention weight and the key frame feature Conquest.

4. The method according to claim 3, wherein the key frame feature and the first fusion feature are used as inputs of a cross-attention module, and the attention weight of the key frame feature is calculated by the cross-attention module, comprising: Inputting the key frame feature and the first fused feature into a cross-attention module, and calculating the attention scores of the key frame feature and the first fused feature through the cross-attention module; The attention score is normalized by the cross-attention module to obtain the attention weight of the key frame feature.

5. The method according to claim 3, wherein generating a second fusion feature based on the attention weight and the key frame feature by the cross-attention module comprises: The key frame features are weighted based on the attention weights of the key frame features through the cross-attention module to obtain a second fusion feature.

6. The method according to claim 2, wherein generating a target feature according to the first fusion feature and the second fusion feature comprises: Adding the first fused feature and the second fused feature and inputting the resultant fused feature into a self-attention module to obtain a third fused feature output by the self-attention module; The third fusion feature is input into a fully connected feed-forward network, and a target feature is generated according to an output feature of the fully connected feed-forward network.

7. The method according to claim 1, wherein the obtaining the compressed domain feature based on the compressed domain information comprises: Inputting the key frame information into a first encoder to obtain key frame features output by the first encoder; Inputting the motion information into a second encoder to obtain motion features output by the second encoder; Inputting the residual information into a third encoder to obtain residual features output by the third encoder; Among them, the first encoder is a heavyweight feature extraction network, and the second encoder and the third encoder are lightweight feature extraction networks.

8. A video description generating device, comprising: An information acquisition module, used to acquire compression domain information of the video to be processed, wherein the compression domain information includes key frame information, motion information and residual information; A feature acquisition module, used for acquiring compressed domain features based on the compressed domain information, wherein the compressed domain features include key frame features, motion features and residual features; A video description generation module is used to fuse the key frame features, motion features and residual features to obtain target features, and input the key frame features and target features into a video description decoder to obtain a video description corresponding to the video to be processed, wherein the target features include motion features, residual features and features in the key frame features that are focused on by motion features and residual features.

9. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the video description generating method according to any one of claims 1 to 7.

10. A storage medium comprising computer executable instructions, wherein the computer executable instructions are used to perform the video description generation method according to any one of claims 1 to 7 when executed by a computer processor.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method for generating a video description as claimed in any one of claims 1 to 7 is implemented.

Images