Video description method, apparatus, and storage medium

Abstract

The application discloses a video description method, device and storage medium. The video description device extracts features of a frame sequence of a to-be-processed video to obtain a feature sequence; determines a causal key frame sequence corresponding to the feature sequence based on a preset causal event chain extractor; determines scene stream features corresponding to the feature sequence based on a preset scene stream encoder; determines causal context information corresponding to the causal key frame sequence and scene context information corresponding to the scene stream features based on a preset decoder, and determines description information corresponding to the to-be-processed video based on the causal context information and the scene context information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a video description method and device and storage medium. BACKGROUND

[0002] With the rapid development of smart logistics and digital transactions, the warehouse management, transaction supervision and other links in the bulk commodity circulation field have gradually introduced intelligent analysis systems based on video content. Video description generation technology as the core support aims to automatically analyze the monitoring video and generate natural language description, which is of great significance to improve the supervision efficiency and realize the automation of the process.

[0003] At present, the related video description methods generally stay in the surface description of the visible objects and behaviors in the picture, and for the scenes of commodity warehouse logistics, there are often causal relationships between events. Therefore, based on the current video description method, it is difficult to describe the causal relationship content other than the visual appearance. SUMMARY

[0004] The present application provides a video description method, device and storage medium, which can describe the causal relationship content other than the visual appearance in the video, and improve the accuracy and applicability of the video description.

[0005] In order to achieve the above purpose, the present application provides the following technical solutions:

[0006] In a first aspect, the present application provides a video description method, comprising:

[0007] performing feature extraction on a frame sequence of a to-be-processed video to obtain a feature sequence;

[0008] determining a causal key frame sequence corresponding to the feature sequence based on a preset causal event chain extractor;

[0009] determining a scene flow feature corresponding to the feature sequence based on a preset scene flow encoder;

[0010] determining causal context information corresponding to the causal key frame sequence and scene context information corresponding to the scene flow feature based on a preset decoder, and determining description information corresponding to the to-be-processed video based on the causal context information and the scene context information.

[0011] In some embodiments of the present application, the preset causal event chain extractor includes a prediction network;

[0012] determining the causal key frame sequence corresponding to the feature sequence based on the preset causal event chain extractor, comprising:

[0013] inputting the feature sequence into the prediction network to utilize the prediction network to predict the subsequent sequence of the feature sequence to obtain a first prediction result;

[0014] zero each frame sequence in the feature sequence respectively, and input the feature sequence after the zero operation into the prediction network to utilize the prediction network to predict the subsequent sequence of the feature sequence after the zero operation, to obtain a second prediction result;

[0015] determine a causal key frame sequence based on the first prediction result and the second prediction result.

[0016] In some embodiments of the present application, determining a causal key frame sequence based on the first prediction result and the second prediction result comprises:

[0017] determine a respective causal contribution degree of each frame sequence based on difference information between the first prediction result and the second prediction result; wherein the causal contribution degree represents an influence degree of each frame sequence on the prediction network to predict the subsequent sequence;

[0018] sort the respective causal contribution degree of each frame sequence in descending order to obtain a sorting result;

[0019] select a preset number of first causal contribution degrees from the sorting result in a forward order, and determine a sequence corresponding to each of the preset number of first causal contribution degrees as the causal key frame sequence.

[0020] In some embodiments of the present application, based on the preset decoder, determine the causal context information corresponding to the causal key frame sequence and the scene context information corresponding to the scene stream feature, and based on the causal context information and the scene context information, determine the description information corresponding to the video to be processed, comprising:

[0021] determine the current demand information based on the preset decoder; wherein the current demand information represents the demand for generating the current word;

[0022] determine the causal stream feature corresponding to the causal key frame sequence based on the preset causal stream encoder, take the causal stream feature as a first key vector and a first value vector, utilize the cross-attention layer in the preset decoder to determine a first similarity weight between the current demand information and the first key vector, and utilize the first similarity weight to perform weighted summation on the first value vector to obtain the causal context information corresponding to the current word;

[0023] take the scene stream feature as a second key vector and a second value vector, utilize the cross-attention layer to determine a second similarity weight between the current demand information and the second key vector, and utilize the second similarity weight to perform weighted summation on the second value vector to obtain the scene context information corresponding to the current word;

[0024] based on the causal context information and the scene context information corresponding to the current word, determine the current word generation result, and based on the current word generation result of the entire video to be processed, determine the description information.

[0025] In some embodiments of the present application, the current word generation result is determined based on the causal context information and the scene context information corresponding to the current word, including:

[0026] A gating value is determined based on a gating network in the preset decoder and the current demand information;

[0027] The causal context information and the scene context information are weighted based on the gating value to obtain a current vector;

[0028] The current vector is subjected to word table probability distribution calculation based on the preset decoder to determine the current word generation result.

[0029] In some embodiments of the present application, the current vector is subjected to word table probability distribution calculation based on the preset decoder to determine the current word generation result, including:

[0030] The current vector is subjected to feature conversion based on a feedforward network in the preset decoder to obtain a feature mapping vector;

[0031] The feature mapping vector is subjected to word table probability distribution calculation to obtain a word table probability distribution;

[0032] The current word generation result is obtained based on the word table probability distribution through autoregressive word screening processing.

[0033] In some embodiments of the present application, the feature sequence is obtained by performing feature extraction on the frame sequence of the video to be processed, including:

[0034] The frame sequence is obtained by performing non-overlapping uniform sampling on the video to be processed;

[0035] The feature sequence is obtained by performing feature extraction on the frame sequence based on a pre-trained convolutional neural network.

[0036] In some embodiments of the present application, the method further includes:

[0037] The preset part of the video frame feature sequence is subjected to mask processing to obtain a masked frame feature;

[0038] The preset part of the masked frame feature is predicted using a preset initial network to obtain prediction information;

[0039] The preset initial network is trained based on a loss value between the prediction information and the feature sequence of the preset part to obtain a prediction network.

[0040] In a second aspect, the embodiments of the present application provide a video description device, including a processor and a memory storing processor-executable instructions; when the instructions are executed by the processor, the above-mentioned video description method is implemented.

[0041] In a third aspect, the embodiments of the present application provide a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the video description method. BRIEF DESCRIPTION OF DRAWINGS

[0042] To more intuitively illustrate the prior art and the present application, several exemplary drawings are given below. It should be understood that the specific shapes, structures, etc. shown in the drawings should not be considered as limiting conditions in the implementation of the present application; for example, based on the technical concepts disclosed in the present application and the exemplary drawings, those skilled in the art can easily make routine adjustments or further optimizations to the increase / decrease / assignment of certain units (components), specific shapes, positional relationships, connection modes, size ratio relationships, etc.

[0043] Figure 1 Implementation flowchart of the video description method provided by the embodiments of the present application Figure One ;

[0044] Figure 2 Implementation flowchart of the video description method provided by the embodiments of the present application Figure Two ;

[0045] Figure 3 Implementation flowchart of the video description method provided by the embodiments of the present application

[0046] Figure 4 Implementation flowchart of the video description method provided by the embodiments of the present application

[0047] Figure 5 Implementation flowchart of the video description method provided by the embodiments of the present application

[0048] Figure 6 Implementation flowchart of the video description method provided by the embodiments of the present application

[0049] Figure 7 Comparison of experimental results of the model proposed by the present application and the conventional model Figure One ;

[0050] Figure 8 Comparison of experimental results of the model proposed by the present application and the conventional model Figure Two ;

[0051] Figure 9 Comparison of experimental results of the model proposed by the present application and the conventional model Figure Three ;

[0052] Figure 10 Comparison of experimental results of the model proposed by the present application and the conventional model Figure Four ;

[0053] Figure 11 Comparison of the model proposed in the present application with experimental results of a conventional model Figure Five

[0054] Figure 12 Structure diagram of a video description device proposed in the present application. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application. The different embodiments can be combined arbitrarily.

[0056] In the description of the present application: unless otherwise specified, the meaning of "multiple" is two or more. The terms "first", "second", "third" and the like in the present application are intended to distinguish the objects referred to, and do not have special meanings in the technical connotation aspect (for example, should not be understood as emphasizing importance or order, etc.). The expressions "include", "contain", "have" and the like also mean "not limited to" (some units, components, materials, steps, etc.).

[0057] With the rapid development of smart logistics and digital transactions, the warehouse management, transaction supervision and other links in the field of bulk commodity circulation are gradually introducing intelligent analysis systems based on video content. Video description generation technology aims to automatically analyze surveillance videos and generate natural language descriptions, which has important application value in improving supervision efficiency and realizing process automation.

[0058] However, the existing video description technology has obvious limitations in practical application. Most current methods are limited to surface description of visible objects and behaviors in the picture, such as "person moving" and "vehicle loading and unloading", and cannot further reveal the causal motivation and business logic behind the behavior. In the bulk commodity warehouse logistics and other scenarios, there is a strong causal relationship between events, such as "starting forklift operation due to system receiving delivery instruction" and "suspending delivery due to failed cargo information verification". Video description that only describes the visual appearance and ignores the internal causal relationship is difficult to meet the actual needs of enterprises for behavior analysis, risk warning and process tracing.

[0059] ​Currently, the common video description method is mainly based on an encoder-decoder architecture, which uses a convolutional neural network (CNN) to extract frame features, and then uses a long short-term memory (LSTM) network or a Transformer model to generate a sentence. Although these methods have achieved certain results in general scenarios, they still have the following shortcomings when processing business videos with strong logicality: on the one hand, the model usually uniformly samples and encodes video frames, and fails to highlight key event frames with causal pivotal roles, resulting in descriptions that cannot reflect the key path of event deduction; on the other hand, existing methods mostly use a single information flow for end-to-end learning, and fail to effectively separate and fuse scene visual information and event logical information, making it difficult for the model to understand and express complex business causal relationships, thereby limiting its practical value in high-demand business scenarios.

[0060] To solve the problems existing in the current video description method, the embodiments of the present application provide a video description method, device and storage medium. The video description device extracts features from a frame sequence of a to-be-processed video to obtain a feature sequence; determines a causal key frame sequence corresponding to the feature sequence based on a preset causal event chain extractor; determines a scene flow feature corresponding to the feature sequence based on a preset scene flow encoder; determines causal context information corresponding to the causal key frame sequence and scene context information corresponding to the scene flow feature based on a preset decoder, and determines description information corresponding to the to-be-processed video based on the causal context information and the scene context information. Based on the above scheme, the causal relationship content other than the visual appearance in the video can be described, and the accuracy and applicability of the video description are improved.

[0061] The embodiments of the present application provide a video description method, as shown in the following Figure 1 The video description method of the video description device can include the following steps:

[0062] Step 101, extracting features from a frame sequence of a to-be-processed video to obtain a feature sequence.

[0063] In the embodiments of the present application, the video description device can first extract features from a frame sequence of a to-be-processed video to obtain a feature sequence.

[0064] In the embodiments of the present application, the video description device is an electronic device with data processing, feature calculation and natural language generation capabilities, such as a computer, an industrial control server, an intelligent monitoring terminal, an edge computing device, or any one of hardware devices with computing functions, which are not limited in the present application.

[0065] In the embodiments of the present application, the video to be processed refers to a target video that needs to generate a description text, such as a bulk commodity warehouse logistics monitoring video, a transaction scene monitoring video, and the like.

[0066] In some embodiments of the present application, when performing feature extraction on the frame sequence of the video to be processed to obtain a feature sequence, the video description device can perform non-overlapping uniform sampling on the video to be processed to obtain the frame sequence; and perform feature extraction on the frame sequence based on a pre-trained convolutional neural network to obtain the feature sequence.

[0067] In the embodiments of the present application, the feature sequence can be understood as a sequence formed by extracting high-dimensional visual semantic features from the frame sequence of the video, representing the visual information of the video.

[0068] For example, the original video to be processed is , by performing non-overlapping uniform sampling on , a frame sequence is obtained; wherein T represents the number of frames; the sampling rate of the sampling can be determined according to the video length of the video to be processed and the required time sequence granularity.

[0069] For example, the pre-trained convolutional neural network can be used to perform feature extraction on each frame in the frame sequence to obtain a high-dimensional feature vector corresponding to each frame, so that the obtained feature sequence can be represented as the frame sequence , wherein T represents the number of frames, and D represents the feature dimension.

[0070] In the embodiments of the present application, non-overlapping uniform sampling refers to a sampling method of extracting a frame sequence from an original video at a fixed time interval, and adjacent sampling frames do not overlap, which is used to reduce the computational load on the premise of retaining key visual information.

[0071] In the embodiments of the present application, the pre-trained convolutional neural network can be a CNN model (such as ResNet, VGG, etc.) trained on a large-scale image data set, which has strong visual feature extraction capability and does not need to be retrained for a target scene.

[0072] For example, non-overlapping uniform sampling is performed on the video to be processed, the sampling rate is adjusted according to the video length and the required time sequence granularity, and an original frame sequence is obtained; then each frame in the frame sequence is scaled to a fixed size, input into the pre-trained convolutional neural network, and the activation value before the last fully connected layer or the global pooling layer of each frame is extracted through forward propagation as a high-dimensional feature vector, and the feature vectors of all frames constitute a feature sequence .

[0073] In the embodiments of the present application, the high quality and stability of the feature sequence are ensured through a standardized feature extraction process. The non-overlapping uniform sampling maximizes the preservation of the timing structure and key visual information of the video while controlling the computational load, avoiding redundant calculations caused by oversampling or information loss caused by undersampling. The use of a pre-trained convolutional neural network fully utilizes the advantages of transfer learning, reduces the model training cost, and ensures the effectiveness and generalization ability of feature extraction, which can accurately capture the visual semantic information of the frame. The high-quality feature sequence provides a reliable input basis for subsequent causal key frame selection, dual-channel encoding and other steps, and is a prerequisite for the realization of high-precision description generation by the entire model.

[0074] In step 102, a feature sequence corresponding causal key frame sequence is determined based on a preset causal event chain extractor.

[0075] In the embodiments of the present application, after the video description device extracts the feature sequence from the frame sequence of the video to be processed, the feature sequence corresponding causal key frame sequence is determined based on the preset causal event chain extractor.

[0076] In the embodiments of the present application, the preset causal event chain extractor is a module for selecting key frames with causal pivotal role from the feature sequence, and its core function is to learn the event causal dynamic rule and quantify the causal contribution degree of the frame.

[0077] In the embodiments of the present application, the causal key frame sequence can be understood as the “causal skeleton” driving the development of the video event stream, and can be composed of the top K frames with the highest causal contribution degree in the feature sequence. The specific value of K is not limited in the present application.

[0078] In some embodiments of the present application, the preset causal event chain extractor includes a prediction network.

[0079] In the embodiments of the present application, the prediction network is a core component of the preset causal event chain extractor, and is a lightweight network structure used to learn the event causal dynamic rule through a frame feature prediction task.

[0080] In some embodiments of the present application, the video description device can perform mask processing on a preset part of the video frame feature sequence to obtain a masked frame feature; use a preset initial network to predict the masked preset part in the masked frame feature to obtain prediction information; train the preset initial network based on the loss value between the prediction information and the feature sequence of the preset part to obtain the prediction network.

[0081] In the embodiments of the present application, the video frame feature sequence is a sample feature sequence used to train the prediction network, and is consistent with the feature sequence extraction method of the video to be processed.

[0082] In the embodiments of the present application, the preset part can be a second half of the video frame feature sequence, and is a preset to-be-masked and to-be-predicted region.

[0083] In the embodiments of the present application, the mask processing is an operation of randomly occluding the frame features of the preset part, so that the prediction network cannot directly obtain the information of this part and is forced to predict through the previous frame features.

[0084] In the embodiments of the present application, the masked frame features represent the complete feature sequence after the mask processing, and include the previous frame features that are not masked and the preset part features that are masked.

[0085] In the embodiments of the present application, the preset initial network represents an initial model structure of the prediction network, which is not trained and has random initial values of parameters.

[0086] In the embodiments of the present application, the prediction information represents a prediction result of the preset initial network for the masked preset part features.

[0087] In the embodiments of the present application, the loss value is a difference quantization value between the prediction information and the real feature sequence of the preset part, such as a mean squared error loss (MSE), which is used to guide the update of network parameters.

[0088] In the embodiments of the present application, through the self-supervised training manner, the prediction network can autonomously learn the causal dynamic law of the event, and the problem of requiring expensive causal annotation data in the prior art is solved. The design of the mask processing builds a self-supervised task of “predicting the subsequent frame from the previous frame”, which forces the network to capture the time sequence dependence and causal correlation of the frame sequence, and enables the causal law learning without manual annotation of the causal relationship. The parameter update based on the loss value ensures the continuous improvement of the prediction accuracy of the network, so that the trained prediction network can accurately quantify the causal contribution degree of each frame. The training scheme not only reduces the data annotation cost, but also improves the generalization ability of the prediction network, which can adapt to video data in different scenarios and provide core technical support for accurate screening of the causal key frame sequence.

[0089] In some embodiments of the present application, when the video description device determines the causal key frame sequence corresponding to the feature sequence based on the preset causal event chain extractor, the feature sequence can be input into the prediction network to predict the subsequent sequence of the feature sequence by using the prediction network, to obtain a first prediction result; each frame sequence in the feature sequence is set to zero, and the feature sequence after the zero setting operation is input into the prediction network to predict the subsequent sequence of the feature sequence after the zero setting operation by using the prediction network, to obtain a second prediction result; and the causal key frame sequence is determined based on the first prediction result and the second prediction result.

[0090] In the embodiments of the present application, the first prediction result represents the subsequent sequence prediction value obtained after the complete feature sequence is input into the prediction network, that is, the prediction result under normal input.

[0091] In the embodiments of the present application, the zero setting operation refers to setting the feature vector of a certain frame in the feature sequence to a zero vector, which is used to destroy the information contribution of the frame.

[0092] In the embodiments of the present application, the second prediction result represents the subsequent sequence prediction value obtained after the feature sequence subjected to the single-frame zero setting operation is input into the prediction network.

[0093] In the embodiments of the present application, through the logic of complete prediction, single-frame destruction and comparison analysis, a quantitative basis is provided for the screening of causal key frames, and the problem that the causal pivotal frame cannot be identified due to uniform sampling in the related art is solved. By comparing the prediction result difference between the complete feature sequence and the feature sequence after single-frame zero setting, the influence degree of each frame on the subsequent event development can be accurately captured, and direct data support is provided for subsequent causal contribution calculation. The present application can avoid the limitations of subjective selection of key frames, ensure that the causal key frame sequence can truly reflect the core logic chain of event development, lay a high-quality data foundation for subsequent causal context information extraction, and improve the reliability of the entire model causal reasoning capability.

[0094] In some embodiments of the present application, when determining the causal key frame sequence based on the first prediction result and the second prediction result, the video description apparatus can determine the respective causal contribution degree of each frame sequence based on the difference value information between the first prediction result and the second prediction result; wherein the causal contribution degree represents the influence degree of each frame sequence on the subsequent sequence prediction of the prediction network; the respective causal contribution degree of each frame sequence is sorted in descending order to obtain a sorting result; and a preset number of first causal contribution degrees are selected from the sorting result in a forward order, and the respective sequence of the preset number of first causal contribution degrees is determined as the causal key frame sequence.

[0095] In the embodiments of the present application, the difference value information refers to the difference between the first prediction result and the second prediction result, which can be calculated by vector norm, for example , represents the prediction value under normal input, that is, the first prediction result, represents the prediction value after destroying the frame, that is, the second prediction result.

[0096] In the embodiments of the present application, the causal contribution degree represents the influence degree of a single frame on the prediction ability of the subsequent sequence prediction network, and can be obtained by accumulation calculation of difference information; the greater the value of the causal contribution degree, the more significant the causal effect of the frame, that is, the higher the influence degree on the prediction of subsequent frames; the calculation method of the causal contribution degree can be represented by the following formula:

[0097] (1)

[0098] wherein, that is, the causal contribution degree of each frame ; and represents the first prediction result, represents the second prediction result.

[0099] In the embodiments of the present application, the sorting result refers to a sequence obtained by arranging the causal contribution degrees of all frames in descending order.

[0100] In the embodiments of the present application, the preset number refers to a preset number K of causal key frames, and K is less than the number T of frames; the specific value of the preset number is not limited in the present application.

[0101] In the embodiments of the present application, the first causal contribution degree represents the causal contribution degree values of the top preset number of the sorting result.

[0102] In the embodiments of the present application, the causal key frame sequence is accurately extracted by quantizing the causal contribution degree and filtering the key frames according to the priority. The calculation method of the causal contribution degree can comprehensively reflect the global influence of a single frame on the development of an event, avoiding the problem of misjudgment of local influence. The setting of the preset number K takes into account the calculation efficiency and key information retention, which not only reduces the calculation load of subsequent encoding through sparse processing, but also ensures that the key causal information is not lost. The present application can solve the defect that the related art cannot quantify the frame causal importance, so that the filtered causal key frame sequence can accurately represent the “causal skeleton” of the development of an event, providing a high-quality input for subsequent causal flow encoding and improving the modeling ability of the model for causal relationship.

[0103] Step 103, determining the scene stream features corresponding to the feature sequence based on a preset scene stream encoder.

[0104] In the embodiments of the present application, after the video description device extracts the feature sequence from the frame sequence of the video to be processed, the scene stream features corresponding to the feature sequence can be determined based on a preset scene stream encoder.

[0105] In the embodiments of the present application, the preset scene stream encoder is an encoder for encoding complete visual scene information of a video, and can include multiple Transformers.

[0106] In the embodiments of the present application, the scene flow feature refers to the output result of the scene flow encoder, which can represent the global visual scene details (such as object shape, environmental features, etc.) of the video.

[0107] In step 104, the causal context information corresponding to the causal key frame sequence and the scene context information corresponding to the scene flow feature are determined based on the preset decoder, and the description information corresponding to the video to be processed is determined based on the causal context information and the scene context information.

[0108] In the embodiments of the present application, after the video description device determines the causal key frame sequence corresponding to the feature sequence based on the preset causal event chain extractor, and determines the scene flow feature corresponding to the feature sequence based on the preset scene flow encoder, the causal context information corresponding to the causal key frame sequence and the scene context information corresponding to the scene flow feature can be determined based on the preset decoder, and the description information corresponding to the video to be processed is determined based on the causal context information and the scene context information.

[0109] In the embodiments of the present application, the preset decoder is a module for generating a video description text, which can internally include cross-attention layers, gated networks, feedforward networks and the like structures, and supports word-by-word autoregressive generation.

[0110] In the embodiments of the present application, the causal context information represents a context vector related to event logic extracted based on the causal key frame sequence.

[0111] In the embodiments of the present application, the scene context information represents a context vector related to visual scenes extracted based on the scene flow feature.

[0112] In the embodiments of the present application, the description information is a natural language text finally generated, which can reflect both the visual content and the event causal relationship of the video.

[0113] In the embodiments of the present application, by constructing the overall framework of feature extraction, dual-path encoding, context fusion and description generation, the present application can break through the limitation of the current single information flow; by separating the processing paths of causal logic information and visual scene information, the core pain points of the current video description that "only describes the surface, not the inside" can be solved. The present application first explicitly introduces causal event chain modeling in video description, ensuring that the generated description not only accurately restores the visual content, but also reveals the internal logical relationship of the event, greatly improving the information amount and decision support value of the description. At the same time, the design of dual-path encoding lays the foundation for subsequent dynamic fusion of context information, and takes into account the logicality and visual accuracy of the description, which is suitable for professional scenarios such as bulk commodity transaction monitoring that require high quality of description.

[0114] In some embodiments of the present application, as Figure 2As shown, when determining the causal context information corresponding to the causal key frame sequence and the scene context information corresponding to the scene stream feature based on the preset decoder, and determining the description information corresponding to the video to be processed based on the causal context information and the scene context information, step 104 can include the following steps:

[0115] Step 104a, determining current demand information based on the preset decoder; wherein the current demand information represents the demand for generating the current word.

[0116] In an embodiment of the present application, when determining the causal context information corresponding to the causal key frame sequence and the scene context information corresponding to the scene stream feature based on the preset decoder, and determining the description information corresponding to the video to be processed based on the causal context information and the scene context information, the video description device can first determine the current demand information based on the preset decoder.

[0117] In an embodiment of the present application, the current demand information represents a query vector of the demand for the preset decoder to generate the current word, which can be converted from the output of the previous layer of the decoder and dynamically updated with the generation process.

[0118] Step 104b, determining the causal stream feature corresponding to the causal key frame sequence based on the preset causal stream encoder, taking the causal stream feature as a first key vector and a first value vector, determining a first similarity weight between the current demand information and the first key vector using the cross-attention layer in the preset decoder, and performing weighted summation on the first value vector using the first similarity weight to obtain the causal context information corresponding to the current word.

[0119] In an embodiment of the present application, after determining the current demand information based on the preset decoder, the video description device can determine the causal stream feature corresponding to the causal key frame sequence based on the preset causal stream encoder, take the causal stream feature as a first key vector and a first value vector, determine a first similarity weight between the current demand information and the first key vector using the cross-attention layer in the preset decoder, and perform weighted summation on the first value vector using the first similarity weight to obtain the causal context information corresponding to the current word.

[0120] In an embodiment of the present application, the preset causal stream encoder is a module for encoding the causal key frame sequence, which can be composed of multiple layers of Transformer encoders and have the same structure as the scene stream encoder but not share parameters.

[0121] In an embodiment of the present application, the causal stream feature represents the output result of the causal stream encoder after encoding the causal key frame sequence, and represents the logical association information of the causal key frame.

[0122] In the embodiments of the present application, the first key vector and the first value vector represent vectors used for matching and weighting in the cross-attention mechanism, and are both causal flow features; for example, they can be represented as: , , 1 represents the first key vector, 1 represents the first value vector, represents the causal flow feature.

[0123] Step 104c, taking the scene flow feature as a second key vector and a second value vector, determining a second similarity weight between the current demand information and the second key vector by using the cross-attention layer, and performing weighted summation on the second value vector by using the second similarity weight to obtain the scene context information corresponding to the current word.

[0124] In the embodiments of the present application, after the video description device determines the current demand information based on the preset decoder, the scene flow feature can be taken as a second key vector and a second value vector, a second similarity weight between the current demand information and the second key vector can be determined by using the cross-attention layer, and the second value vector can be weighted and summed by using the second similarity weight to obtain the scene context information corresponding to the current word.

[0125] In the embodiments of the present application, the second key vector and the second value vector are vectors used for matching and weighting in the cross-attention mechanism, and are both scene flow features; for example, they can be represented as: , , 2 represents the first key vector, 2 represents the first value vector, represents the causal flow feature.

[0126] Exemplarily, at each layer of the preset decoder, two independent cross-attention sub-layers are used to interact with the outputs of the causal flow and the scene flow respectively: , , wherein, represents the current demand information.

[0127] In the embodiments of the present application, the second similarity weight is a similarity quantization value between the current demand information and the second key vector, and can be used to represent the importance of each part of the scene flow feature in generating the current word.

[0128] Step 104d, determining the current word generation result based on the causal context information and the scene context information corresponding to the current word, and determining the description information based on all current word generation results of the to-be-processed video.

[0129] In the embodiments of the present application, the video description apparatus determines the causal flow characteristics corresponding to the determined causal key frame sequence based on the preset causal flow encoder, takes the causal flow characteristics as the first key vector and the first value vector, determines the first similarity weight between the current demand information and the first key vector by using the cross attention layer in the preset decoder, and performs weighted summation on the first value vector by using the first similarity weight to obtain the causal context information corresponding to the current word. In addition, the scene flow characteristics are taken as the second key vector and the second value vector, the second similarity weight between the current demand information and the second key vector is determined by using the cross attention layer, and the second value vector is weighted and summed by using the second similarity weight to obtain the scene context information corresponding to the current word. Then, the current word generation result can be determined based on the causal context information and the scene context information corresponding to the current word, and the description information can be determined based on the current word generation result of the entire to-be-processed video.

[0130] In the embodiments of the present application, the current word generation result represents a single word or symbol generated by each round of autoregressive generation of the preset decoder, and the entire current word generation result represents a word sequence generated by multiple rounds of autoregressive generation of the preset decoder, which can be arranged in the generation order.

[0131] In the embodiments of the present application, the cross attention mechanism is used to realize accurate extraction of context information, solving the problem of insufficient pertinence of context information extraction in the related art. The introduction of the current demand information enables the context extraction to dynamically adapt to the demand of each round of word generation, ensuring that the extracted causal context information is closely related to the event logic and the scene context information is consistent with the visual content. The design of the double-path cross attention realizes the separate extraction of the causal logic and the visual scene information, ensuring the purity of the two types of information and providing high-quality input for subsequent dynamic fusion. The autoregressive generation ensures the logical coherence of the description text, enabling the final generated description to present the visual content and the causal relationship in the order of event development, improving the readability and practicality of the description.

[0132] In some embodiments of the present application, when the video description apparatus determines the current word generation result based on the causal context information and the scene context information corresponding to the current word, the gate value can be determined based on the gate network in the preset decoder and the current demand information; the causal context information and the scene context information are weighted based on the gate value to obtain the current vector; and the word table probability distribution of the current vector is calculated based on the preset decoder to determine the current word generation result.

[0133] In the embodiments of the present application, the gate network is a lightweight network structure in the preset decoder, which can be composed of a feedforward network and a Sigmoid activation function, and is used to dynamically adjust the weights of the two types of context information.

[0134] In an embodiment of the present application, the gating value is the output result of the gating network, which can be denoted as , and the value range is , which can be used to quantify the relative importance of the causal context information and the scene context information.

[0135] In an embodiment of the present application, the current vector represents the final context vector obtained by weighting and fusing the causal context information and the scene context information through the gating value.

[0136] In an embodiment of the present application, the word table probability distribution calculation refers to converting the feature vector into the occurrence probability of each word in the word table through the Softmax layer in the preset decoder, which is used to represent the generation priority of each word.

[0137] In an embodiment of the present application, the dynamic fusion of context information is realized by introducing the gating network, which solves the problem of poor description adaptability caused by fixed fusion ratio in the related art. The gating value can adaptively adjust the weights of the two types of context information according to the current word generation demand: when generating logical conjunction words, the gating value tends to 1, which strengthens the contribution of causal context; when generating visual nouns, the gating value tends to 0, which focuses on the role of visual context. This dynamic adaptation mechanism ensures that each generated word can get the most matched information support, which not only avoids the logical break caused by the lack of causal information, but also prevents the description from being out of touch with the picture caused by the lack of visual information. At the same time, the design of the lightweight gating network controls the computational complexity on the premise of ensuring the fusion effect, which balances the model efficiency and the generation quality.

[0138] In some embodiments of the present application, when the video description device performs word table probability distribution calculation on the current vector based on the preset decoder to determine the current word generation result, the device can perform feature conversion on the current vector based on the feedforward network in the preset decoder to obtain a feature mapping vector; perform word table probability distribution calculation on the feature mapping vector to obtain a word table probability distribution; and perform autoregressive word screening processing based on the word table probability distribution to obtain the current word generation result.

[0139] In an embodiment of the present application, the feedforward network is a fully connected network structure in the preset decoder, which is used for feature conversion and dimension adaptation of the current vector to improve the expression ability of the feature.

[0140] In an embodiment of the present application, the feature mapping vector is the output result of the feedforward network, which is the feature representation of the current vector after linear transformation and nonlinear activation, and adapts to the input requirements of the subsequent Softmax layer.

[0141] In an embodiment of the present application, the word table probability distribution is the output result of the Softmax layer, which is a probability vector with a dimension consistent with the size of the word table, and each element corresponds to the generation probability of a word.

[0142] In the embodiments of the present application, the autoregressive word screening process can be that, in the generation order, the word with the highest probability in the probability distribution is selected as the current word generation result in each round, and the result is fed back to the decoder to update the current demand information in the next round, and the cycle is repeated until the end symbol is generated.

[0143] In the embodiments of the present application, through the fine process of feature conversion, probability calculation and regression screening, the accuracy and sequence continuity of the current word generation are ensured. The feature conversion of the feedforward network can enhance the semantic expression ability of the current vector, providing better feature input for subsequent probability calculation; the probability distribution calculation of the Softmax layer realizes the quantitative decision of word generation, avoiding the randomness of subjective selection; the autoregressive word screening method dynamically adjusts each round of generation based on the historical results, ensuring the grammatical correctness and logical continuity of the description text. The problem of strong randomness and sequence breakage in the technical word generation in the related art is solved, the fluency and accuracy of the description text are improved, and the generated sentence is more in line with the natural language expression habit.

[0144] The embodiments of the present application provide a video description method. A video description device extracts features from a frame sequence of a to-be-processed video to obtain a feature sequence; determines a causal key frame sequence corresponding to the feature sequence based on a preset causal event chain extractor; determines a scene flow feature corresponding to the feature sequence based on a preset scene flow encoder; determines causal context information corresponding to the causal key frame sequence and scene context information corresponding to the scene flow feature based on a preset decoder, and determines description information corresponding to the to-be-processed video based on the causal context information and the scene context information. As can be seen, the present application constructs an overall framework of feature extraction, dual-path encoding, context fusion and description generation, breaking through the limitation of a single information flow in the related art. By separating the processing paths of causal logic information and visual scene information, the core pain point of the existing video description that "only describes the surface, not the inside" is solved. The present application first explicitly introduces causal event chain modeling in video description, ensuring that the generated description not only accurately restores the visual content, but also reveals the internal logical relationship of the event, greatly improving the information quantity and decision support value of the description. At the same time, the design of dual-path encoding lays a foundation for subsequent dynamic fusion of context information, taking into account the logicality and visual accuracy of the description, and is suitable for professional scenarios such as bulk commodity transaction monitoring that require high quality of description.

[0145] Based on the above embodiments, in another embodiment of the present application, illustratively, Figure 3As shown, a video description generation method based on causal event chain and visual scene double-flow modeling is provided, including video preprocessing and feature extraction, causal event chain extractor, double-flow Transformer encoder, dynamic fusion decoder module, which can be implemented by the following steps:

[0146] The video preprocessing and feature extraction module includes the following steps:

[0147] Step one, input the original video stream.

[0148] Step two, uniformly sample the original video stream without overlap to reduce the computational load, and adjust the sampling rate according to the video length and the required temporal granularity. After sampling, the original frame sequence is obtained: where T is the number of frames.

[0149] Step three, use a pre-trained convolutional neural network as a feature extractor to scale each frame to a fixed size, while extracting the activation values before the last fully connected layer or global pooling layer as high-dimensional feature vectors Finally, the entire video is represented as a feature sequence where D is the feature dimension.

[0150] The causal event chain extractor module includes the following steps:

[0151] Step four, build a lightweight prediction network, and let a certain number of frame features in the second half of the random mask sequence predict the features of the masked frames based on the previous unmasked frames. Through this task, the network is forced to learn the dynamic rules of how previous events lead to subsequent events.

[0152] In some embodiments of the present application, the lightweight prediction network can be built based on a multi-layer perceptron (MLP) or a Transformer; illustratively, the structure of the lightweight prediction network can include an input layer, a normalization layer, a fully connected layer, and an output layer; wherein the input layer can receive the masked frame features, the normalization layer can perform layer normalization on the input masked frame features to eliminate the numerical distribution difference between different frame features, align the feature scale, and avoid prediction bias caused by feature value fluctuations; the fully connected layer can be set to 1-2 layers of lightweight structure, which can capture the temporal dependence and causal association between frame sequences through an activation function for non-linear feature conversion, while controlling the parameter size and strengthening the feature expression capability; the output layer can map the abstract features output by the fully connected layer back to the same dimension as the original frame features, and directly output the predicted features (i.e., prediction information) of the masked part, whose output dimension is completely matched with the dimension of the masked frame features.

[0153] Step five, for each frame in the sequence , all subsequent frames are predicted by the network after zeroing its features The degree of causal contribution is quantified by calculating the degree of influence on the ability of the network to predict subsequent frames, as shown in the formula (1) above; the greater the difference, the higher the degree of causal contribution.

[0154] Step six, according to the calculated causal contribution scores of all frames, select the top K frames with the highest scores to form a sparse causal key frame sequence: where, can represent the "causal skeleton" that drives the development of the entire video event stream.

[0155] The dual-flow Transformer encoder module includes the following steps:

[0156] Step seven, build a causal flow encoder, which consists of a standard L-layer Transformer encoder, each layer contains multi-head self-attention mechanism and feedforward neural network, and can take the sparse causal key frame sequence as input.

[0157] Step eight, build a scene flow encoder, which consists of an L-layer Transformer encoder, the number of layers and the dimension of the hidden layer are the same as the causal flow encoder, but the parameters are not shared, and the original video frame sequence is taken as input.

[0158] The dynamic fusion decoder module includes the following steps:

[0159] Step nine, at each layer of the decoder, use two independent cross-attention sublayers to interact with the outputs of the causal flow and scene flow respectively: , where, represents the current demand information.

[0160] Step ten, based on a lightweight gating network, a feedforward network with Sigmoid activation function is used as the input, which takes the mean pooling result of the query vector of the current layer of the decoder as input, and outputs a dynamic scalar gating value ; where, , represents the Sigmoid function, which compresses any numerical value to between 0 and 1, represents the weight matrix, represents the "average pooling" of the query vector , ​denotes the bias term. The final fused context vector is weighted by the gating value: , denotes the causal context information, denotes the scene context information.

[0161] Step eleven, the fused context vector is sent into the feedforward network, and the probability distribution on the word table is predicted through the Softmax layer, and the words are generated one by one in an autoregressive manner to form a complete description sentence , , that is, the current word generation result generated each time can constitute the final complete description sentence of the video, such as “forklift operation is started due to receiving the outbound instruction”, “vehicle is suspended due to unqualified cargo information verification”.

[0162] Exemplarily, as shown in Figure 4 , for the training of the lightweight prediction network, the video feature sequence can be input first (step 201), the latter half of the sequence is randomly masked (step 202), the masked part is predicted by using the lightweight prediction network (step 203), the prediction error is calculated (step 204), it is judged whether the model converges (step 205), if yes, the model is saved (step 206), if not, the parameters are adjusted (step 207); further, when the causal contribution degree is calculated by using the trained prediction network, as shown in Figure 5 , the video feature sequence can be input into the prediction network (step 301), the prediction is performed under the normal sequence condition (step 302), the current frame is zeroed and then predicted (step 303), the causal contribution degree is calculated based on the two prediction values (step 304), and the K-frame composed causal key frame sequence is output (step 305).

[0163] Exemplarily, as shown in Figure 6 , after obtaining the decoder query vector, the scene stream and the causal stream can be processed respectively by using the double-flow cross attention layer, and the scene context information and the causal context information are obtained respectively, and the gating value is determined by using the gating network, and finally the scene context information, the causal context information and the gating value are fused to obtain the current description word.

[0164] Exemplarily, as shown in Figure 7As shown, the effect of the model (causal event chain + double-flow sparse Transformer) proposed in the present application and the conventional CNN + Transformer model (baseline method) is experimentally verified, and the verification indicators include 4-tuple bilingual evaluation replacement indicators (BLEU-4), ROUGE indicators based on the longest common subsequence (ROUGE-L), explicit ranking translation evaluation indicators (METEOR), and consensus-based image description evaluation indicators (CIDEr-D); as shown in Figures 8 to 11 As shown, the performance of the present application and the conventional model under different interference conditions is compared, wherein, Figure 8 The performance comparison under Gaussian noise conditions, Figure 9 The performance comparison under motion blur conditions, Figure 10 The performance comparison under resolution reduction conditions, Figure 11 The performance comparison under compression artifact conditions; through the comparison of these experimental results, it can be seen that the performance of the model of the present application is obviously better than that of the conventional model of the video description.

[0165] To sum up, the present application first explicitly introduces and models the "causal event chain" in video description, can generate descriptions that reveal the internal logical relationship of events, greatly improves the information content and decision support value of the description, and is especially suitable for monitoring and analyzing complex business processes such as bulk commodity transactions; through the sparse processing of the video frame by the causal event chain extractor, the causal flow encoder only needs to process a small number of key frames, which reduces the computational complexity of the model, and at the same time enables the model to focus more on key information, improves the description accuracy and logical consistency of long video sequences, and realizes the coexistence of efficiency and accuracy; the causal event chain extractor uses self-supervised learning, does not require expensive causal relationship labeled data, and is easy to extend to new fields. At the same time, the description generated by the model and its internal gating attention mechanism enable the causal reasoning process to have a certain explainability, which is convenient for users to understand and trust the model output, has strong generalization and explainability; solves the core pain points of "shallow description and missing logic" of general video description models in related business scenarios, and provides a powerful technical tool for automated monitoring, intelligent report generation and risk warning, and has good practicality for professional fields.

[0166] Based on the above embodiment, another embodiment of the present application provides a video description device, as shown in Figure 12 The video description device 1 proposed in the present application embodiment can include a processor 11, a memory 12 storing processor 11 executable instructions; further, the video description device 1 can also include a communication interface 13, and a bus 14 for connecting the processor 11, the memory 12 and the communication interface 13.

[0167] In the embodiments of the present application, the processor 11 can be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, or a microprocessor. It can be understood that, for different devices, the electronic device used to implement the functions of the processor can also be other devices, and the embodiments of the present application are not limited in this regard. The memory 12 can be connected to the processor 11, wherein the memory 12 is configured to store executable program codes, and the program codes include computer operation instructions. The memory 12 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least two disk memories.

[0168] In the embodiments of the present application, the bus 14 is configured to connect the communication interface 13, the processor 11, and the memory 12, so as to realize mutual communication among the devices.

[0169] In the embodiments of the present application, the memory 12 is configured to store instructions and data.

[0170] In actual applications, the memory 12 can be a volatile memory such as a Random-Access Memory (RAM), or a non-volatile memory such as a Read-Only Memory (ROM), a flash memory, a Hard Disk Drive (HDD), or a Solid-State Drive (SSD), or a combination of the above types of memories, and provide instructions and data to the processor 11.

[0171] In addition, each functional module in the embodiments can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware or in the form of a software functional module.

[0172] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer readable storage medium based on such understanding. The technical solutions of the embodiments essentially or the parts that make contributions to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) or a processor to execute all or part of the steps of the embodiments.

[0173] Specifically, the program instructions corresponding to the video description method in the embodiment can be stored on a storage medium such as an optical disc or a hard disk. When the program instructions corresponding to the video description method in the storage medium are read by a video description device or executed, the following steps are included:

[0174] Feature extraction is performed on the frame sequence of the video to be processed to obtain a feature sequence;

[0175] A causal key frame sequence corresponding to the feature sequence is determined based on a preset causal event chain extractor;

[0176] Scene flow features corresponding to the feature sequence are determined based on a preset scene flow encoder;

[0177] Causal context information corresponding to the causal key frame sequence and scene context information corresponding to the scene flow features are determined based on a preset decoder, and description information corresponding to the video to be processed is determined based on the causal context information and the scene context information.

[0178] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system or a computer program product. Therefore, the present application can be in the form of a hardware embodiment, a software embodiment or an embodiment combining software and hardware aspects. Moreover, the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer usable program code.

[0179] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks.

[0180] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart or flowsheet block or blocks. Figure One Figure One These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks.

[0181] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks. Figure One Figure One These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flowsheet block or blocks.

[0182] The above embodiments are only preferred embodiments of the present application, but the protection scope of the present application is not limited to this. Any equivalent substitution or transformation made by the skilled in the art based on the present application is within the protection scope of the present application.​​

Claims

1. A method of video description, characterized by, The method comprises: performing feature extraction on a frame sequence of a to-be-processed video to obtain a feature sequence; inputting the feature sequence into a prediction network in a preset causal event chain extractor to perform prediction on subsequent sequences of the feature sequence by using the prediction network to obtain a first prediction result; the prediction network comprises an input layer, a normalization layer, a full connection layer, and an output layer; zeroing each frame sequence in the feature sequence respectively, and inputting the feature sequence subjected to the zeroing operation into the prediction network to perform prediction on subsequent sequences of the feature sequence subjected to the zeroing operation by using the prediction network to obtain a second prediction result; determining respective causal contribution degrees of the each frame sequence based on difference information between the first prediction result and the second prediction result; wherein the causal contribution degree represents an influence degree of the each frame sequence on subsequent sequence prediction of the prediction network; sorting the respective causal contribution degrees of the each frame sequence in descending order to obtain a sorting result; selecting a preset number of first causal contribution degrees from the sorting result in a forward order, and determining respective sequences of the preset number of first causal contribution degrees as causal key frame sequences; determining scene stream features corresponding to the feature sequence based on a preset scene stream encoder; determining causal context information corresponding to the causal key frame sequences and scene context information corresponding to the scene stream features based on a preset decoder, and determining description information corresponding to the to-be-processed video based on the causal context information and the scene context information.

2. The video description method of claim 1, wherein, The determining of the causal context information corresponding to the causal key frame sequences and the scene context information corresponding to the scene stream features based on the preset decoder, and the determining of the description information corresponding to the to-be-processed video based on the causal context information and the scene context information, comprise: determining current demand information based on the preset decoder; wherein the current demand information represents a demand for generating a current word; determining causal stream features corresponding to the causal key frame sequences based on a preset causal stream encoder, taking the causal stream features as a first key vector and a first value vector, determining a first similarity weight between the current demand information and the first key vector by using a cross-attention layer in the preset decoder, and performing weighted summation on the first value vector by using the first similarity weight to obtain the causal context information corresponding to the current word; taking the scene stream features as a second key vector and a second value vector, determining a second similarity weight between the current demand information and the second key vector by using the cross-attention layer, and performing weighted summation on the second value vector by using the second similarity weight to obtain the scene context information corresponding to the current word; determining a current word generation result based on the causal context information and the scene context information corresponding to the current word, and determining the description information based on all the current word generation results of the to-be-processed video.

3. The video description method of claim 2, wherein, The current word generation result is determined based on the causal context information and the scene context information corresponding to the current word. A gating value is determined based on a gating network in the preset decoder and the current requirement information. The causal context information and the scene context information are weighted based on the gating value to obtain a current vector. The current vector is subjected to word table probability distribution calculation based on the preset decoder to determine the current word generation result.

4. The video description method of claim 3, wherein, The current vector is subjected to word table probability distribution calculation based on the preset decoder to determine the current word generation result. The current vector is subjected to feature conversion based on a feedforward network in the preset decoder to obtain a feature mapping vector. The feature mapping vector is subjected to word table probability distribution calculation to obtain a word table probability distribution. The current word generation result is obtained through autoregressive word screening processing based on the word table probability distribution.

5. The video description method of claim 1, wherein, The method further comprises: A preset part of the video frame feature sequence is subjected to mask processing to obtain a masked frame feature. The preset part of the masked frame feature is predicted based on a preset initial network to obtain prediction information.

6. The video description method of claim 1, wherein, The preset initial network is trained based on a loss value between the prediction information and the feature sequence of the preset part to obtain the prediction network. The video description device comprises a processor, a memory, and a memory storing executable instructions of the processor; when the instructions are executed by the processor, the video description method of any one of claims 1 to 6 is implemented. The computer program is executed by the processor to implement the video description method of any one of claims 1 to 6. ​ 7. A video description apparatus characterized by comprising: ​ 8. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images