Video description generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a video description generation method and device, electronic equipment and a storage medium. The method comprises the steps that compressed domain information of a video to be processed is acquired, and the compressed domain information comprises key frame information, motion information and residual error information; compressed domain features are obtained based on the compressed domain information, and the compressed domain features comprise key frame features, motion features and residual features; the key frame features, the motion features and the residual features are fused to obtain target features, the key frame features and the target features are input into a video description decoder to obtain video description corresponding to the video to be processed, and the target features comprise the motion features, the residual features and features concerned by the motion features and the residual features in the key frame features. According to the embodiment of the invention, the video description can be generated by using the compressed domain information, and the problems of high computing resource consumption, high time consumption, quality loss, hardware dependence and the like caused by video decoding are solved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to computer vision and natural language processing technologies, and more particularly to a method, device, electronic device, and storage medium for generating a video description. Background Art

[0002] Video description is a natural language sentence that describes the content of a video. The research on video description can improve the computer's visual and language understanding capabilities, and can also play an important role in application scenarios such as barrier-free services, content search and recommendation.

[0003] The video description generation scheme in the related art usually decodes the video first, and then inputs the decoded video frames into a video description model. The video description model is used to understand the complex visual content in the video based on the decoded video frames, and converts the complex visual content into a natural and fluent language description, that is, to obtain a video description. The processing process is complex and consumes a lot of computing resources. Summary of the invention

[0004] The present disclosure provides a video description generation method, device, electronic device and storage medium, which can solve the problem that the video description generation process is complicated and consumes a lot of computing resources.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for generating a video description, comprising:

[0006] Acquire compression domain information of the video to be processed, wherein the compression domain information includes key frame information, motion information and residual information;

[0007] Acquire compression domain features based on the compression domain information, wherein the compression domain features include key frame features, motion features, and residual features;

[0008] The key frame features, motion features and residual features are fused to obtain target features, and the key frame features and target features are input into a video description decoder to obtain a video description corresponding to the video to be processed, wherein the target features include motion features, residual features and features of the key frame features that are focused on by motion features and residual features.

[0009] In a second aspect, an embodiment of the present disclosure further provides a video description generating device, the device comprising:

[0010] An information acquisition module, used to acquire compression domain information of the video to be processed, wherein the compression domain information includes key frame information, motion information and residual information;

[0011] A feature acquisition module, used for acquiring compressed domain features based on the compressed domain information, wherein the compressed domain features include key frame features, motion features and residual features;

[0012] A video description generation module is used to fuse the key frame features, motion features and residual features to obtain target features, and input the key frame features and target features into a video description decoder to obtain a video description corresponding to the video to be processed, wherein the target features include motion features, residual features and features in the key frame features that are focused on by motion features and residual features.

[0013] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0014] one or more processors;

[0015] a storage device for storing one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the video description generating method as described in any embodiment of the present disclosure.

[0017] In a fourth aspect, an embodiment of the present disclosure further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the video description generating method as described in any embodiment of the present disclosure.

[0018] The disclosed embodiments provide a method, device, electronic device and storage medium for generating a video description. The method obtains key frame features, motion features and residual features through compressed domain information of a video to be processed, fuses the key frame features, motion features and residual features to obtain target features, and then inputs the key frame features and target features into a video description decoder to obtain a video description corresponding to the video to be processed. The method can generate a video description using compressed domain information, and solve the problems of large consumption of computing resources, high time consumption, quality loss and hardware dependence caused by video decoding. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0020] Figure 1 A schematic diagram of a flow chart of a method for generating a video description provided by an embodiment of the present disclosure;

[0021] Figure 2 A schematic diagram of a compressed domain information extraction process in a video description generation method provided by an embodiment of the present disclosure;

[0022] Figure 3A flowchart of another method for generating video description provided by an embodiment of the present disclosure;

[0023] Figure 4 A schematic diagram of the structure of a cross-attention module provided in an embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of the structure of a target network provided by an embodiment of the present disclosure;

[0025] Figure 6 A flowchart of another method for generating a video description provided by an embodiment of the present disclosure;

[0026] Figure 7 A schematic diagram of the structure of a video description generating device provided by an embodiment of the present disclosure;

[0027] Figure 8 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0029] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0030] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0031] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0032] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0033] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0034] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0035] Existing video description methods often require decoding the video first, and then input the decoded video frames into the video description model to generate the video description. The decoded video frames generally include I frame (Intra-coded frame, key frame information), B frame (Bidirectional frame, bidirectional prediction frame) and P frame (Predicted frame, prediction frame). This method has the following problems in terms of computing resource consumption, time consumption, quality loss and hardware dependence:

[0036] In terms of computing resource consumption, since video decoding is a computationally intensive operation, it may consume a large amount of computing resources (CPU, GPU or memory), especially when processing high-definition, high-frame-rate videos or multi-channel video streams, which will consume more computing resources and may affect other computing tasks.

[0037] In terms of time consumption, since the video decoding process may be very slow, this is a serious problem for video content that requires real-time processing (such as real-time video description).

[0038] In terms of quality loss, during the video encoding process, some quality loss compression techniques are often used to compress the video to store the video in a smaller space. During the video decoding process, the discarded information is usually unrecoverable, which may affect the subsequent video analysis.

[0039] In terms of hardware dependency, some efficient decoders may require special hardware support (such as certain types of GPUs, etc.), which may increase hardware dependency and raise the hardware threshold.

[0040] Figure 1A flow chart of a method for generating a video description provided in an embodiment of the present disclosure is provided. The embodiment of the present disclosure is applicable to the case of generating a text description of a video. The method can be executed by a video description generating device, which can be implemented in the form of software and / or hardware. Optionally, it can be implemented by an electronic device, which can be a mobile terminal, a PC or a server, etc.

[0041] like Figure 1 As shown, the method includes:

[0042] S110. Obtain compression domain information of the video to be processed, wherein the compression domain information includes key frame information, motion information, and residual information.

[0043] The video to be processed may be video data in a compressed format. For example, the video to be processed may be video data in a compressed format obtained by compressing video frames using a video compression algorithm. The compression domain information of the video to be processed may include key frame information, motion information, residual information, and the like.

[0044] Since adjacent video frames in a video are similar, a video compression algorithm is used to divide the video into a series of image groups (Group of Pictures, GOP) based on the similarity between video frames. The RGB image of the first frame in each GOP is used as the key frame information (I frame), and the motion information of subsequent frames relative to the I frame is recorded, and the residual information of subsequent frames relative to the I frame is recorded.

[0045] Figure 2 Schematic diagram of a compressed domain information extraction process in a video description generation method provided by an embodiment of the present disclosure. Figure 2 As shown in the figure, the extraction process of key frame information, motion information and residual information is shown by taking the extraction of compressed domain information from a GOP as an example. The first frame image of the GOP is obtained as an I frame. For the remaining B frames or P frames in the GOP, the motion information and residual information are extracted by dequantization.

[0046] Exemplarily, the first frame of each image group in the video to be processed is obtained as key frame information. Motion information is obtained based on the relative displacement data of pixels between the key frame information and adjacent frames. Residual information is obtained based on the color difference of pixels between the key frame information and adjacent frames.

[0047] S120. Obtain compression domain features based on the compression domain information, wherein the compression domain features include key frame features, motion features, and residual features.

[0048] In the disclosed embodiment, the compressed domain feature may be a feature of compressed domain information extracted by a network model, wherein the network model may be an encoder constructed by a neural network, etc. The disclosed embodiment does not limit which network model is used to extract the compressed domain feature of compressed domain information.

[0049] In the disclosed embodiment, the compressed domain information includes key frame information, motion information and residual information. Then, the key frame information is input into the first encoder to obtain the key frame features output by the first encoder; the motion information is input into the second encoder to obtain the motion features output by the second encoder; the residual information is input into the third encoder to obtain the residual features output by the third encoder. Among them, the first encoder is a heavyweight feature extraction network, and the second encoder and the third encoder are lightweight feature extraction networks. Using a lightweight feature extraction network to extract motion features from motion information and residual features from residual information can improve the feature extraction rate and shorten the feature extraction time.

[0050] The first encoder may be a heavyweight network trained on sample pairs consisting of I frames and I frame features, and the I frame features of the I frame are extracted through the visual modality branch of the trained first encoder. Similarly, the second encoder may be a lightweight network trained on sample pairs consisting of motion information and motion features, and the motion features of the motion information are extracted through the visual modality branch of the trained second encoder. Similarly, the third encoder may be a lightweight network trained on sample pairs consisting of residual information and residual features, and the residual features of the residual information are extracted through the visual modality branch of the trained third encoder.

[0051] It should be noted that the above-mentioned first encoder to third encoder can be an encoder network in the Transformer network architecture, or can be other neural network architectures, and the embodiments of the present disclosure do not make specific limitations.

[0052] S130, fusing the key frame features, motion features and residual features to obtain target features, inputting the key frame features and target features into a video description decoder to obtain a video description corresponding to the video to be processed.

[0053] Among them, the target features include motion features, residual features and features of the key frame features that are focused on by the motion features and the residual features. Optionally, the cross-attention module can be used to fuse the key frame features, motion features and residual features to obtain the target features. For example, the motion features and the residual features are superimposed to obtain a query vector of the cross-attention module. Optionally, the motion features and the residual features can be fused to obtain a first fused feature. It should be noted that the method of superimposing the motion features and the residual features can be to add the two, and the embodiment of the present disclosure does not limit the superposition method of the motion features and the residual features. The key frame features are used as the key vector and value vector of the cross-attention module. The cross-attention module is used to determine the degree of attention of the query vector to the key frame features.

[0054] For example, the similarity between the first fused feature and the key frame feature is calculated through the cross-attention module, and the similarity is processed by softmax to obtain the attention weight corresponding to the key frame feature. The weight corresponding to the key frame feature is multiplied and summed with the key frame to obtain the second fused feature. The second fused feature can be a feature of the key frame feature that is focused on by the motion feature and the residual feature. The target feature is generated by setting a neural network to fuse the first fused feature and the second fused feature. The setting of the neural network can include a self-attention module and a fully connected feedforward network, etc.

[0055] In the disclosed embodiment, the video description decoder can be a neural network that generates a video description based on key frame features and target features of key frame information in an undecoded video. For example, the video description decoder can be a neural network trained on a video-text pair consisting of a video and a video description. The key frame features and target features corresponding to the video to be processed are input into the video description decoder. Since the target features integrate motion features, residual features, and features in the key frame features that are focused on by the motion features and residual features, the target features contain rich video frame information. The video description decoder can generate a video description corresponding to the video to be processed based on the key frame features and the target features.

[0056] The technical solution of the disclosed embodiment obtains key frame features, motion features and residual features through the compressed domain information of the undecoded video to be processed, fuses the key frame features, motion features and residual features to obtain the target features, and then inputs the key frame features and the target features into a video description decoder to obtain the video description corresponding to the video to be processed. It can generate the video description by using the compressed domain information, and solve the problems of large computing resource consumption, high time consumption, quality loss and hardware dependence caused by video decoding.

[0057] Figure 3This is a flow chart of another method for generating video description provided by an embodiment of the present disclosure. Based on the above embodiment, the embodiment of the present disclosure specifically defines the step of fusing the key frame features, motion features and residual features to obtain the target features. Figure 3 As shown, the method includes:

[0058] S310: Obtain compression domain information of the video to be processed, wherein the compression domain information includes key frame information, motion information, and residual information.

[0059] S320. Obtain compression domain features based on the compression domain information, wherein the compression domain features include key frame features, motion features, and residual features.

[0060] S330, fusing the motion feature and the residual feature to obtain a first fused feature.

[0061] Exemplarily, if the motion feature and the residual feature are vectors of the same dimension, fusing the motion feature and the residual feature is to calculate the vector sum of the motion feature and the residual feature, and use the vector sum as the first fused feature. If the motion feature and the residual feature are vectors of different dimensions, the dimensions of the motion feature and the residual feature are made the same by dimensionality reduction technology.

[0062] S340: Use the key frame feature and the first fusion feature as input of a cross-attention module to obtain a second fusion feature output by the cross-attention module.

[0063] Among them, the key frame feature and the first fusion feature have the same dimension.

[0064] Exemplarily, the key frame features and the first fused features are used as inputs of a cross-attention module, and the attention weights of the key frame features are calculated by the cross-attention module; and the second fused features are generated based on the attention weights and the key frame features by the cross-attention module.

[0065] Optionally, the key frame feature and the first fused feature are used as inputs of a cross-attention module, and the attention weight of the key frame feature is calculated by the cross-attention module, including: inputting the key frame feature and the first fused feature into the cross-attention module, and calculating the attention score of the key frame feature and the first fused feature by the cross-attention module. The attention weight of the key frame feature is obtained by normalizing the attention score by the cross-attention module.

[0066] Figure 4 A schematic diagram of the structure of a cross-attention module provided in an embodiment of the present disclosure is shown in FIG. Figure 4As shown, the input of the cross-attention module includes key frame information I and the first fused feature, and the key frame information I is used as the key vector V and the value vector K, and the first fused feature is used as the query vector Q. The cross-attention module calculates the dot product of the key frame information I and the first fused feature to obtain the attention score of the key frame feature, where the attention score is expressed as K·Q. The attention weight W corresponding to the attention score is calculated by the normalization layer in the cross-attention module. The normalization layer can be a softmax layer, etc. Then, the key frame information is multiplied and summed by the cross-attention module to obtain the second fused feature, where the second fused feature is expressed as ∑V·W.

[0067] S350: Generate a target feature according to the first fusion feature and the second fusion feature.

[0068] Exemplarily, the first fusion feature and the second fusion feature are added and input into a self-attention module to obtain a third fusion feature output by the self-attention module; the third fusion feature is input into a fully connected feedforward network, and a target feature is generated according to the output feature of the fully connected feedforward network.

[0069] Among them, the third fusion feature is an enhanced feature calculated by the self-attention module based on the first fusion feature and the second fusion feature.

[0070] Exemplarily, the target features can be obtained by fusing key frame features, motion features and residual features through a target network with a set number of layers. Figure 5 A schematic diagram of a target network structure provided by an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the target network includes a cross-attention module, a self-attention module and a fully connected feedforward network. The first fusion feature corresponding to the motion feature and the residual feature is input into the cross-attention module, and the second fusion feature is output through the cross-attention module. Then, the first fusion feature is connected to the output end of the cross-attention module through a residual connection, and is added to the second fusion feature output by the cross-attention module as the input of the self-attention module. The first fusion feature and the second fusion feature are weightedly fused through the self-attention module to obtain a third fusion feature. The target feature is determined based on the third fusion feature through the fully connected feedforward network.

[0071] It should be noted that Figure 5 Only the process of fusing key frame features, motion features and residual features through a layer of target network to obtain target features is shown, but the disclosure does not limit the number of layers of the target network, and the number of layers of the target network can be flexibly selected according to the specific task. If a multi-layer target network is set, the output of the fully connected feedforward network of the previous layer of the target network is used as the input of the cross-attention module of the next layer of the target network.

[0072] Among them, the target feature can simulate the B frame feature and the P frame feature. The B frame is a bidirectional prediction frame, which contains the change of the image with the previous frame and the next frame. The P frame is a prediction frame, which contains the difference data of the image (that is, the change with the previous frame). While storing image information, the P frame also records the change of the image. Since the target feature is a feature obtained by fusing the second fusion feature with the first fusion feature, and the first fusion feature is a feature obtained by fusing the motion feature with the residual feature, the second fusion feature represents the feature of the key frame feature that is concerned by the motion feature and the residual feature. Therefore, the target feature fuses the motion feature, the residual feature, and the feature of the key frame feature that is concerned by the motion feature and the residual feature. Based on the definition of B frame and P frame, it can be seen that the B frame feature and the P frame feature can be simulated by the target feature.

[0073] Optionally, the number of target networks in the above example may be increased so that the target feature contains rich feature information, or the number of target networks in the above example may be reduced, which is not specifically limited in the embodiments of the present disclosure.

[0074] S360: Input the key frame features and the target features into a video description decoder to obtain a video description corresponding to the video to be processed.

[0075] The technical solution of the disclosed embodiment inputs the first fused feature and the key frame feature after the motion feature and the residual feature are fused into the cross-attention module, and the target feature is generated based on the second fused feature output by the cross-attention module and the first fused feature, so as to fuse the key frame feature with the first fused feature to enhance the feature in the key frame feature that is focused on by the motion feature and the residual feature, and obtain the second fused feature. Since the second fused feature represents the feature in the key frame feature that is focused on by the motion feature and the residual feature, the target feature is generated based on the second fused feature and the first fused feature, so that the B frame feature and the P frame feature can be simulated by the target feature, so that there is no need to decode the video to be processed. By inputting the key frame information and the target feature into the video description decoder, the video description corresponding to the video to be processed can be obtained.

[0076] Figure 6 FIG. 2 is a flowchart of another method for generating a video description provided by an embodiment of the present disclosure. Based on the above embodiment, an embodiment of the present disclosure provides a flowchart of a specific example. Figure 6 As shown, the method includes:

[0077] S601. Start.

[0078] S602: Input the video to be processed.

[0079] The video is a video to be processed. In the disclosed embodiment, the video to be processed may be a video in a compressed format downloaded from the network. Alternatively, the video to be processed may also be a video in a compressed format shot by a user, or the video to be processed may also be a video in a compressed format generated by video editing software, etc.

[0080] S603: Obtain compression domain information.

[0081] Exemplarily, compression domain information of the video to be processed is obtained, wherein the compression domain information includes key frame information, motion information and residual information.

[0082] S604: Acquire key frame features.

[0083] S605: Acquire motion features.

[0084] S606: Obtain residual features.

[0085] S607: Fusing the key frame features, motion features and residual features to obtain target features.

[0086] First, the motion feature and the residual feature are fused to obtain the first fused feature, and the key frame feature and the first fused feature are used as the input of the cross-attention module. The cross-attention module is used to fuse the key frame feature, the motion feature and the residual feature to obtain the target feature. The target feature includes the motion feature, the residual feature, and the features in the key frame feature that are concerned by the motion feature and the residual feature, which can simulate the B frame feature and the P frame feature of the video to be processed.

[0087] S608: Input the key frame features and the target features into the video description decoder.

[0088] S609: The video description decoder generates a video description.

[0089] Exemplarily, I frame features, simulated generated B frame features and P frame features are input into a video description decoder, and a video description corresponding to the video to be processed is generated by the video description decoder.

[0090] S610, end.

[0091] The video description method provided by the embodiment of the present disclosure does not require video decoding and can directly generate video description using compressed domain information, thereby avoiding the problems of large computing resource consumption, high time consumption, quality loss, and hardware dependence caused by video decoding.

[0092] Figure 7This is a schematic diagram of the structure of a video description generating device provided in an embodiment of the present disclosure. The device is used to execute the video description generating method described in any embodiment of the present disclosure, and the device is implemented in the form of software and / or hardware. Optionally, it is implemented by an electronic device, which can be a mobile terminal, a PC or a server. Figure 7 As shown, the device includes: an information acquisition module 710, a feature acquisition module 720 and a video description generation module 730.

[0093] The information acquisition module 710 is used to acquire the compression domain information of the video to be processed, wherein the compression domain information includes key frame information, motion information and residual information;

[0094] A feature acquisition module 720, configured to acquire compressed domain features based on the compressed domain information, wherein the compressed domain features include key frame features, motion features, and residual features;

[0095] The video description generation module 730 is used to fuse the key frame features, motion features and residual features to obtain target features, input the key frame features and target features into a video description decoder, and obtain a video description corresponding to the video to be processed, wherein the target features include motion features, residual features and features in the key frame features that are focused on by motion features and residual features.

[0096] The technical solution provided by the embodiment of the present disclosure obtains key frame features, motion features and residual features through the compressed domain information of the undecoded video to be processed, fuses the key frame features, motion features and residual features to obtain the target features, and then inputs the key frame features and the target features into a video description decoder to obtain the video description corresponding to the video to be processed. It can generate the video description by using the compressed domain information, and solve the problems of large computing resource consumption, high time consumption, quality loss and hardware dependence caused by video decoding.

[0097] Optionally, the video description generating module 730 includes:

[0098] A first fusion feature determination unit, used for fusing the motion feature and the residual feature to obtain a first fusion feature;

[0099] A second fusion feature determination unit is used to use the key frame feature and the first fusion feature as inputs of a cross-attention module to obtain a second fusion feature output by the cross-attention module, wherein the second fusion feature is a feature of the key frame feature that is focused on by the motion feature and the residual feature;

[0100] A target feature generating unit is used to generate a target feature according to the first fusion feature and the second fusion feature.

[0101] Furthermore, the second fusion feature determination unit is specifically used for:

[0102] Using the key frame feature and the first fusion feature as inputs of a cross-attention module, and calculating the attention weight of the key frame feature through the cross-attention module;

[0103] A second fused feature is generated based on the attention weight and the key frame feature through the cross-attention module.

[0104] Optionally, taking the key frame feature and the first fusion feature as inputs of a cross-attention module, and calculating the attention weight of the key frame feature through the cross-attention module, includes:

[0105] Inputting the key frame feature and the first fused feature into a cross-attention module, and calculating the attention scores of the key frame feature and the first fused feature through the cross-attention module;

[0106] The attention score is normalized by the cross-attention module to obtain the attention weight of the key frame feature.

[0107] Optionally, generating a second fusion feature based on the attention weight and the key frame feature through the cross-attention module includes:

[0108] The key frame features are weighted based on the attention weights of the key frame features through the cross-attention module to obtain a second fusion feature.

[0109] Optionally, the target feature generating unit is specifically used for:

[0110] Adding the first fused feature and the second fused feature and inputting the resultant feature into a self-attention module to obtain a third fused feature output by the self-attention module;

[0111] The third fusion feature is input into a fully connected feed-forward network, and a target feature is generated according to an output feature of the fully connected feed-forward network.

[0112] Optionally, the feature acquisition module 720 is specifically used for:

[0113] Inputting the key frame information into a first encoder to obtain key frame features output by the first encoder;

[0114] Inputting the motion information into a second encoder to obtain motion features output by the second encoder;

[0115] Inputting the residual information into a third encoder to obtain residual features output by the third encoder;

[0116] Among them, the first encoder is a heavyweight feature extraction network, and the second encoder and the third encoder are lightweight feature extraction networks.

[0117] The video description generating device provided in the embodiments of the present disclosure can execute the video description generating method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0118] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the protection scope of the embodiments of the present disclosure.

[0119] Figure 8 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 8 , which shows an electronic device (eg, Figure 8 The terminal device in the embodiment of the present disclosure may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 8 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0120] like Figure 8 As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An edit / output (I / O) interface 805 is also connected to the bus 804.

[0121] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0122] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0123] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0124] The electronic device provided in the embodiment of the present disclosure and the video description generation method provided in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0125] The embodiment of the present disclosure provides a computer storage medium on which a computer program is stored. When the program is executed by a processor, the video description generating method provided in the above embodiment is implemented.

[0126] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0127] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0128] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0129] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0130] Acquire compression domain information of the video to be processed, wherein the compression domain information includes key frame information, motion information and residual information;

[0131] Acquire compression domain features based on the compression domain information, wherein the compression domain features include key frame features, motion features, and residual features;

[0132] The key frame features, motion features and residual features are fused to obtain target features, and the key frame features and target features are input into a video description decoder to obtain a video description corresponding to the video to be processed, wherein the target features include motion features, residual features and features of the key frame features that are focused on by motion features and residual features.

[0133] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0134] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0135] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, constitute a limitation on the unit itself.

[0136] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0137] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0138] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0139] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0140] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A video description generation method, characterized in that: include: Acquire compression domain information of the video to be processed, wherein the compression domain information includes key frame information, motion information and residual information; Acquire compression domain features based on the compression domain information, wherein the compression domain features include key frame features, motion features, and residual features; The key frame features, motion features and residual features are fused to obtain target features, and the key frame features and target features are input into a video description decoder to obtain a video description corresponding to the video to be processed, wherein the target features include motion features, residual features and features of the key frame features that are focused on by motion features and residual features.

2. The method according to claim 1, characterized in that: The step of fusing the key frame features, motion features and residual features to obtain target features includes: Fusing the motion feature with the residual feature to obtain a first fused feature; The key frame feature and the first fused feature are used as inputs of a cross-attention module to obtain a second fused feature output by the cross-attention module, wherein the second fused feature is a feature of the key frame feature that is focused on by the motion feature and the residual feature; A target feature is generated according to the first fused feature and the second fused feature.

3. The method according to claim 2, characterized in that The step of using the key frame feature and the first fused feature as inputs of a cross-attention module to obtain a second fused feature output by the cross-attention module includes: Using the key frame feature and the first fusion feature as inputs of a cross-attention module, and calculating the attention weight of the key frame feature through the cross-attention module; A second fused feature is generated based on the attention weight and the key frame feature through the cross-attention module.

4. The method according to claim 3, characterized in that The step of taking the key frame feature and the first fusion feature as inputs of a cross-attention module and calculating the attention weight of the key frame feature through the cross-attention module includes: Inputting the key frame feature and the first fused feature into a cross-attention module, and calculating the attention scores of the key frame feature and the first fused feature through the cross-attention module; The attention score is normalized by the cross-attention module to obtain the attention weight of the key frame feature.

5. The method according to claim 3, characterized in that: The generating a second fusion feature based on the attention weight and the key frame feature by the cross-attention module includes: The key frame features are weighted based on the attention weights of the key frame features through the cross-attention module to obtain a second fusion feature.

6. The method according to claim 2, characterized in that Generating a target feature according to the first fused feature and the second fused feature includes: Adding the first fused feature and the second fused feature and inputting the resultant feature into a self-attention module to obtain a third fused feature output by the self-attention module; The third fusion feature is input into a fully connected feed-forward network, and a target feature is generated according to an output feature of the fully connected feed-forward network.

7. The method according to claim 1, characterized in that The obtaining of the compressed domain feature based on the compressed domain information includes: Inputting the key frame information into a first encoder to obtain key frame features output by the first encoder; Inputting the motion information into a second encoder to obtain motion features output by the second encoder; Inputting the residual information into a third encoder to obtain residual features output by the third encoder; Among them, the first encoder is a heavyweight feature extraction network, and the second encoder and the third encoder are lightweight feature extraction networks.

8. A video description generating device, characterized in that: include: An information acquisition module, used to acquire compression domain information of the video to be processed, wherein the compression domain information includes key frame information, motion information and residual information; A feature acquisition module, used for acquiring compressed domain features based on the compressed domain information, wherein the compressed domain features include key frame features, motion features and residual features; A video description generation module is used to fuse the key frame features, motion features and residual features to obtain target features, and input the key frame features and target features into a video description decoder to obtain a video description corresponding to the video to be processed, wherein the target features include motion features, residual features and features in the key frame features that are focused on by motion features and residual features.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the video description generating method according to any one of claims 1 to 7.

10. A storage medium containing computer executable instructions, characterized in that: When the computer executable instructions are executed by a computer processor, they are used to perform the video description generation method according to any one of claims 1 to 7.

Images