A dense video description method, device, system and storage medium

Abstract

The application provides a dense video description method, device, system and storage medium, and belongs to the field of video processing.The method comprises the following steps: performing feature extraction on an original video to obtain a video feature vector; performing fusion analysis on the video feature vector to obtain a target fusion feature vector; performing feature analysis on the target fusion feature vector to obtain an event query feature vector; performing video description analysis on the event query feature vector to obtain a location confidence, a generated sentence length and an event description; summing up all the event query feature vectors to obtain a total number of event query feature vectors; and performing confidence analysis according to the location confidence, the generated sentence length and the event description to obtain a target confidence.The application can detect the usefulness of each feature vector, reduces the information redundancy in the model, and greatly improves the overall dense description accuracy.

Description

TECHNICAL FIELD

[0001] The present application mainly relates to the technical field of video processing, and particularly relates to a dense video description method, device, system and storage medium. BACKGROUND

[0002] The dense video description is divided into two sub-tasks, one is time event positioning, and the other is time description. Most of the previous methods are to position the event first and then describe. The end-to-end dense video description framework has a parallel decoding function, which can utilize the correlation between the features to make the intermediate feature vector corresponding to the event have stronger discriminability. However, the existing dense video description framework does not well select the useful features for the coding of the visual features, and other useless features are also input into the model for learning, thereby increasing the information redundancy and reducing the accuracy of the dense description. SUMMARY

[0003] The present application mainly relates to the technical field of video processing, and particularly relates to a dense video description method, device, system and storage medium.

[0004] The technical problem solved by the present application is as follows: a dense video description method, device, system and storage medium are provided to solve the technical problems of the prior art.

[0005] The original video is imported, and feature extraction is performed on the original video to obtain a plurality of video feature vectors;

[0006] Each video feature vector is subjected to fusion analysis to obtain a target fusion feature vector of each video feature vector;

[0007] The target fusion feature vector of each video feature vector is subjected to feature analysis to obtain a plurality of event query feature vectors of each video feature vector;

[0008] Each event query feature vector of each video feature vector is subjected to video description analysis to obtain a position confidence, a generated sentence length and a plurality of event descriptions of each event query feature vector;

[0009] The sum of all event query feature vectors is counted to obtain the total number of event query feature vectors;

[0010] The position confidence, the generated sentence length and the plurality of event descriptions of each event query feature vector are subjected to confidence analysis to obtain a target confidence of each event query feature vector;

[0011] The target confidences of all event query feature vectors are sorted in descending order, and the top N setsort the target confidence degrees of all the event query feature vectors in descending order, and take the event query feature vectors corresponding to the target confidence degrees after the first N set as the video description results, wherein N set is the total number of event query feature vectors.

[0012] Another technical solution of the present application to solve the above technical problems is as follows: a dense video description device, comprising:

[0013] a feature extraction module configured to import an original video, perform feature extraction on the original video, and obtain a plurality of video feature vectors;

[0014] a fusion analysis module configured to perform fusion analysis on each of the video feature vectors respectively, and obtain a target fusion feature vector of each of the video feature vectors;

[0015] a feature analysis module configured to perform feature analysis on the target fusion feature vector of each of the video feature vectors respectively, and obtain a plurality of event query feature vectors of each of the video feature vectors;

[0016] a video description analysis module configured to perform video description analysis on each of the event query feature vectors of each of the video feature vectors respectively, and obtain a location confidence degree, a generated sentence length, and a plurality of event descriptions of each of the event query feature vectors;

[0017] a statistical module configured to count the sum of all the event query feature vectors, and obtain a total number of event query feature vectors;

[0018] a confidence degree analysis module configured to perform confidence degree analysis on the location confidence degree, the generated sentence length, and the plurality of event descriptions of each of the event query feature vectors, and obtain a target confidence degree of each of the event query feature vectors;

[0019] a video description result obtaining module configured to sort the target confidence degrees of all the event query feature vectors in descending order, and take the event query feature vectors corresponding to the target confidence degrees after the first N set as the video description results, wherein N set is the total number of event query feature vectors.

[0020] Based on the above-described dense video description method, the present application further provides a dense video description system.

[0021] Another technical solution of the present application to solve the above technical problems is as follows: a dense video description system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, when the processor executes the computer program, the dense video description method as described above is implemented.

[0022] Based on the above one dense video description method, the application also provides a computer readable storage medium.

[0023] Another technical solution of the application to solve the above technical problems is as follows: a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program is executed by a processor, the dense video description method as described above is realized.

[0024] The application has the beneficial effects that: the feature extraction of the original video obtains the video feature vector, the fusion analysis of the video feature vector obtains the target fusion feature vector, the feature analysis of the target fusion feature vector obtains the event query feature vector, the video description analysis of the event query feature vector obtains the location confidence, the generated sentence length and the multiple event descriptions, the summation of the event query feature vector obtains the total number of event query feature vectors, the target confidence is analyzed according to the location confidence, the generated sentence length and the confidence of the event description, the video description result is obtained by sorting the target confidence in descending order, the usefulness of each feature vector can be detected, the information redundancy in the model is reduced, and the overall dense description accuracy is greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 A flowchart of a dense video description method provided by an embodiment of the application is shown in the figure.

[0026] Figure 2 A module block diagram of a dense video description device provided by an embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0027] The principles and characteristics of the application are described below in combination with the drawings, and the examples are only used to explain the application and not to limit the scope of the application.

[0028] Figure 1 A flowchart of a dense video description method provided by an embodiment of the application is shown in the figure.

[0029] As shown in the figure, a dense video description method includes the following steps: Figure 1

[0030] Importing an original video, performing feature extraction on the original video to obtain multiple video feature vectors;

[0031] Performing fusion analysis on each video feature vector respectively to obtain a target fusion feature vector of each video feature vector;

[0032] Performing feature analysis on the target fusion feature vector of each video feature vector respectively to obtain multiple event query feature vectors of each video feature vector; ​

[0033] video description analysis is performed on each of the event query feature vectors of each of the video feature vectors to obtain a localization confidence, a generated sentence length, and a plurality of event descriptions for each of the event query feature vectors;

[0034] a total number of event query feature vectors is obtained by counting a total of all of the event query feature vectors;

[0035] a target confidence for each of the event query feature vectors is obtained by confidence analysis according to the localization confidence, the generated sentence length, and the plurality of event descriptions for each of the event query feature vectors;

[0036] the target confidences of all of the event query feature vectors are sorted in descending order, and the event query feature vectors corresponding to the top N set target confidences after sorting are taken as a video description result, where N set is the total number of event query feature vectors.

[0037] It should be understood that, because the events (i.e., the event query feature vectors) contained in the input video (i.e., the original video) are inconsistent, in order to prevent information loss and redundant title generation caused by unreliable event counting, the module performs average pooling on the features of the N set queries and classifies them using a multi-layer perception to obtain the number of events N

[0038] Specifically, the number of events (i.e., the total number of event query feature vectors) is predicted from a global view, the boundary (i.e., the localization confidence) and the description (i.e., the event description) of each event are obtained through parallel decoding, and the final output is obtained by selecting the top N set (i.e., the total number of event query feature vectors) events (i.e., the event query feature vectors) with accurate boundaries and good titles from the N

[0039] In the above embodiment, feature extraction on the original video obtains a video feature vector, fusion analysis on the video feature vector obtains a target fusion feature vector, feature analysis on the target fusion feature vector obtains an event query feature vector, video description analysis on the event query feature vector obtains a localization confidence, a generated sentence length, and a plurality of event descriptions, counting a total of the event query feature vectors obtains a total number of event query feature vectors, confidence analysis according to the localization confidence, the generated sentence length, and the event description obtains a target confidence, sorting the target confidences in descending order obtains a video description result, which can detect the usefulness of each feature vector and reduce information redundancy in the model, greatly improving the overall dense description accuracy.

[0040] Optionally, as an embodiment of the present application, the process of feature extraction on the original video to obtain a plurality of video feature vectors comprises:

[0041] Feature extraction on the original video using an action recognition network to obtain a plurality of video feature vectors.

[0042] Preferably, the action recognition network can be a C3D network.

[0043] It should be understood that the input video (i.e. the original video) is subjected to video feature extraction using the action recognition network C3D network, each video frame corresponds to a video feature vector, and a feature map (i.e. a plurality of video feature vectors) is obtained.

[0044] In the above embodiment, the action recognition network is used to extract features from the original video to obtain a plurality of video feature vectors, which can extract useful features and provide a basis for subsequent data processing.

[0045] Optionally, as an embodiment of the present application, the plurality of video feature vectors are arranged in time sequence, and the process of respectively performing fusion analysis on each video feature vector to obtain a target fusion feature vector of each video feature vector comprises:

[0046] The first video feature vector is taken as the fusion feature vector after the first video feature vector;

[0047] Starting from the second video feature vector, fusion calculation is performed according to each video feature vector and the previous video feature vector of each video feature vector, respectively, until the last video feature vector, to obtain the fusion feature vector after each video feature vector;

[0048] Feature extraction is performed on the fusion feature vector after the first video feature vector and the fusion feature vector after each video feature vector by a plurality of one-dimensional convolution layers to obtain a target fusion feature vector of each video feature vector.

[0049] Specifically, the usefulness of each feature vector (i.e. the video feature vector) in the feature map is measured, and then the feature vectors with high usefulness are selected to pass through the gate, and then the fusion feature vector after the gate (i.e. the fusion feature vector) is input to the stacked 1D convolution layer, and finally the feature map is concatenated.

[0050] In the above embodiments, the target fusion feature vector of each video feature vector is obtained by performing fusion analysis on each video feature vector respectively, the usefulness of each feature vector can be detected frame by frame by using a gating full fusion mechanism, and the propagation of information is controlled by the gate according to the size of the usefulness, which can reduce the information redundancy in the model, so that multi-level feature fusion can be performed in a full connection manner, and features with more effective information can be selected from a large number of visual features for propagation.

[0051] Optionally, as an embodiment of the present application, the process of obtaining the fusion feature vector of each video feature vector by performing fusion calculation on each video feature vector and the previous video feature vector of each video feature vector from the second video feature vector to the last video feature vector includes:

[0052] Based on the first formula, the fusion calculation is performed on each video feature vector and the previous video feature vector of each video feature vector, and the fusion feature vector of each video feature vector is obtained, the first formula is:

[0053]

[0054] wherein, X l is the fusion feature vector of the i-th video feature vector, X i is the i-th video feature vector, X j is the previous video feature vector of the i-th video feature vector, G i , G j are threshold values, and L is the total number of video feature vectors.

[0055] It should be understood that the purpose of fusing the extracted features (i.e. the video feature vectors) is to use a gating mechanism to select and propagate useful features and filter out useless features, so that the encoding is more compact.

[0056] Specifically, the gating full fusion formula is as follows:

[0057]

[0058] The gating full fusion is based on the attention mechanism to propagate according to the size of the feature usefulness. For the visual features extracted by the action behavior recognition network C3D, there are 3 layers of feature maps. For the i-th layer feature X i (i.e. the i-th video feature vector), a threshold value G i is assigned, and G i can control the propagation of the feature usefulness, and the features higher than the threshold value can be sent, and the features lower than the threshold value can be received. The specific design is as follows:

[0059] Sender: through G i All useful information in feature maps of all layers except i layer higher than the threshold value is reserved and then added together through addition;

[0060] Receiver: through 1-G i The information sent by the sender is reversely filtered to reserve the information that i layer does not have (avoid information redundancy).

[0061] In the above embodiment, the first formula is used to respectively calculate the fusion feature vectors of the video feature vectors and the previous video feature vectors of the video feature vectors, and the gating mechanism is used to select and propagate useful features and filter out useless features, so that the coding is more compact, and the multi-level feature fusion can be performed in a full connection manner, and the features with more effective information are selected and propagated in a large number of visual features.

[0062] Optionally, as an embodiment of the present application, the process of respectively performing feature analysis on the target fusion feature vectors of the video feature vectors to obtain a plurality of event query feature vectors of each video feature vector includes:

[0063] The target fusion feature vectors of each video feature vector are encoded by an encoder to obtain an encoded feature vector of each video feature vector;

[0064] The encoded feature vectors of each video feature vector are decoded by a decoder to obtain a plurality of event query feature vectors of each video feature vector.

[0065] It should be understood that the feature vectors (i.e. the video feature vectors) propagated by the gate are input to the encoder to capture the correlation between frames at different distances.

[0066] Specifically, the feature map (i.e. the encoded feature vector) obtained by the encoder is input to the decoder for decoding, and each token in the output N event queries represents each possible event (i.e. the event query feature vector), and all tokens predict an event set as a representation of N events.

[0067] In the above embodiment, the encoding of the target fusion feature vectors by the encoder obtains the encoded feature vectors, and the decoding of the encoded feature vectors by the decoder obtains the event query feature vectors, which can capture the correlation between frames at different distances, reduce the information redundancy in the model, and greatly improve the overall dense description accuracy.

[0068] Optionally, as one embodiment of the present application, the process of performing video description analysis on each of the event query feature vectors of each of the video feature vectors to obtain the localization confidence of each of the event query feature vectors, the generated sentence length and the plurality of event descriptions comprises:

[0069] The multi-head attention mechanism algorithm is used to localize each of the event query feature vectors of each of the video feature vectors to obtain the localization confidence of each of the event query feature vectors.

[0070] The LSTM network is used to perform video description on each of the event query feature vectors of each of the video feature vectors to obtain the generated sentence length and the plurality of event descriptions of each of the event query feature vectors.

[0071] Specifically, the prediction of the event boundary by the multi-layer perception machine gives an event localization confidence (i.e., the localization confidence). Since the boundary of the event cannot be directly obtained, the deformable attention mechanism is used to limit the attention weight of each incoming envent query in a small area around the reference point as the event center point. A plurality of sampling points are sampled around the center point, and the positions of the sampling points can be adaptively learned. Then, the attention weight on the sampling points is predicted. Based on this attention mechanism method, the visual area with strong relevance to the word is focused at each time, which ensures that the generated word sequence describes the same event. The LSTM takes the context feature, the event query feature and the previous word as input, and the probability of the word is obtained by using the FC layer of Softmax. A sentence is output by the iteration of the LSTM. The queries are directly decoded into a set of events with positions and corresponding descriptions.

[0072] In the above embodiment, the localization of the event query feature vector by the multi-head attention mechanism algorithm obtains the localization confidence, and the video description of the event query feature vector by the LSTM network obtains the generated sentence length and the event description, which reduces the information redundancy in the model and greatly improves the overall dense description accuracy.

[0073] Optionally, as one embodiment of the present application, the process of performing confidence analysis on the localization confidence, the generated sentence length and the plurality of event descriptions of each of the event query feature vectors to obtain the target confidence of each of the event query feature vectors comprises:

[0074] Based on the second formula, the confidence of each event query feature vector is calculated according to the positional confidence, the generated sentence length and the confidence of the plurality of event descriptions, and a target confidence of each event query feature vector is obtained, the second formula being:

[0075]

[0076] wherein C j is the target confidence of the jth event query feature vector, is the positional confidence of the jth event query feature vector, is the tth event description of the jth event query feature vector, M j is the generated sentence length, μ is a balance factor, and y is a modulation factor.

[0077] Specifically, the calculation method of the confidence of each event query (i.e., the target confidence) is as follows:

[0078]

[0079] wherein is the positional confidence, is the probability of the generated word (i.e., the event description), M j is the length of the generated sentence (i.e., the generated sentence length), and t is the length of the current sentence. However, the confidence of each word cannot reflect the confidence of the whole sentence description, because the event description has a higher confidence for a short sentence. Therefore, a modulation factor y is added to correct the influence of the title length. μ is a balance factor.

[0080] In the above embodiment, the target confidence is calculated based on the second formula according to the positional confidence, the generated sentence length and the confidence of the event description, which can detect the usefulness of each feature vector and reduce the information redundancy in the model, greatly improving the overall dense description accuracy.

[0081] Optionally, as another embodiment of the present application, the present application uses a transformer framework combined with a gated all-fusion module to solve the problem of selecting useful features for learning, so that the whole framework can generate event descriptions under the overall understanding of the video, and more accurate description results are obtained.

[0082] Optionally, as another embodiment of the present application, it comprises: a feature extraction module, a gated all fusion module, an encoder module, a decoder module, an event counting module, an event positioning module, and an event description module. The feature extraction module extracts features from the input video using a C3D network, and inputs the extracted features into the gated all fusion module to select useful features for propagation to the encoder module according to the size of the feature usefulness, and then the decoder module and the event positioning, event counting, and event description modules jointly constitute a decoding network. Given the position, description, and number of events of a learning event query, the attention mechanism is used to capture the interaction between frames and events to generate queryable features of the event, and then the event with high confidence is selected to obtain the final dense description result. The present application can use a gated all fusion mechanism to detect the usefulness of each feature vector frame by frame, and control the propagation of information through the gate according to the size of the usefulness. Based on this gating mechanism, the information redundancy in the model can be reduced, so that multi-level feature fusion can be performed in a fully connected manner, and features with more effective information are selected for propagation among a large number of visual features. The feature vector effectiveness of the encoder module is improved, thereby greatly improving the dense description accuracy of the entire system.

[0083] Optionally, as another embodiment of the present application, it comprises: (1) a feature extraction module, (2) a gated all fusion module, (3) an encoder module, (4) a decoder module, (5) an event counting module, (6) an event positioning module, and (7) an event description module.

[0084] The (1) feature extraction module is used for visual feature extraction of a video, to obtain frame-level features, and the position encoding is combined to select propagation through the gating module and input into the encoding module;

[0085] The (2) gated all fusion module is used for fusing the extracted features, and the purpose is to use the gating mechanism to select useful features for propagation, thereby filtering out useless features to make the encoding module more compact;

[0086] The (3) encoder module is used for encoding and transmitting the useful features propagated in (2) to the (4) decoder module.

[0087] The (4) decoder module, together with the (5) event counting module, the (6) event positioning module, and the (7) event description module, jointly constitutes a decoding network, and takes the dense video description task as a set prediction problem, and directly decodes the feature vector obtained from the encoder into an event with specific positioning and description.

[0088] Figure 2 A module block diagram of a dense video description device provided for an embodiment of the present application.

[0089] Optionally, as another embodiment of the present application, as shown inFigure 2 As shown in the figure, a dense video description device comprises:

[0090] a feature extraction module configured to import an original video, perform feature extraction on the original video, and obtain a plurality of video feature vectors;

[0091] a fusion analysis module configured to perform fusion analysis on each of the video feature vectors, and obtain a target fusion feature vector of each of the video feature vectors;

[0092] a feature analysis module configured to perform feature analysis on the target fusion feature vector of each of the video feature vectors, and obtain a plurality of event query feature vectors of each of the video feature vectors;

[0093] a video description analysis module configured to perform video description analysis on each of the event query feature vectors of each of the video feature vectors, and obtain a location confidence, a generated sentence length, and a plurality of event descriptions of each of the event query feature vectors;

[0094] a statistics module configured to count a total of all the event query feature vectors, and obtain a total number of event query feature vectors;

[0095] a confidence analysis module configured to perform confidence analysis on the location confidence, the generated sentence length, and the plurality of event descriptions of each of the event query feature vectors, and obtain a target confidence of each of the event query feature vectors;

[0096] a video description result obtaining module configured to sort the target confidences of all the event query feature vectors in descending order, and take the event query feature vectors corresponding to the top N set target confidences as video description results, wherein N set is the total number of event query feature vectors.

[0097] Optionally, another embodiment of the present application provides a dense video description system comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, when the processor executes the computer program, the dense video description method as described above is implemented. The system can be a computer or the like.

[0098] Optionally, another embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, when the computer program is executed by a processor, the dense video description method as described above is implemented.

[0099] It should be noted that, in the present document, the terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprising", "containing" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a list of elements does not only include those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article or apparatus.

[0100] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0101] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative, for example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0102] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0103] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0104] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art, or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0105] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A dense video description method, characterized in that, Includes the following steps: Import the original video, extract features from the original video, and obtain multiple video feature vectors; Each of the video feature vectors is fused and analyzed to obtain the target fused feature vector of each of the video feature vectors. Feature analysis is performed on the target fused feature vectors of each video feature vector to obtain multiple event query feature vectors for each video feature vector; Video description analysis is performed on each of the event query feature vectors of each video feature vector to obtain the location confidence, generated sentence length, and multiple event descriptions of each event query feature vector. The total number of event query feature vectors is obtained by summing all the event query feature vectors. Confidence analysis is performed on the location confidence of each event query feature vector, the length of the generated sentence, and multiple event descriptions to obtain the target confidence of each event query feature vector; Sort the target confidence scores of all event query feature vectors in descending order, and then... The event query feature vectors corresponding to the sorted target confidence scores are used as the video description results, where... This represents the total number of event query feature vectors. The multiple video feature vectors are arranged in chronological order. The process of fusing and analyzing each of the video feature vectors to obtain the target fused feature vector for each video feature vector includes: The first video feature vector is used as the fused feature vector of the first video feature vector; Starting from the second video feature vector, the fusion calculation is performed based on each video feature vector and the previous video feature vector of each video feature vector, until the last video feature vector is obtained, thus obtaining the fused feature vector of each video feature vector. By using multiple one-dimensional convolutional layers to extract features from the fused feature vector of the first video feature vector and the fused feature vectors of each video feature vector, the target fused feature vector of each video feature vector is obtained. The process of performing confidence analysis based on the location confidence of each event query feature vector, the generated sentence length, and multiple event descriptions to obtain the target confidence of each event query feature vector includes: Based on the second formula, the confidence level is calculated according to the location confidence of each event query feature vector, the generated sentence length, and multiple event descriptions to obtain the target confidence level of each event query feature vector. The second formula is: ， in, For the first The target confidence score of the feature vector for each event query. For the first Locational reliability of the feature vector of an event query. For the first The first event query feature vector Event description, To generate sentence length, As a balance factor, is the modulation factor.

2. The dense video description method according to claim 1, characterized in that, The process of extracting features from the original video to obtain multiple video feature vectors includes: The original video is used to extract features using an action recognition network, resulting in multiple video feature vectors.

3. The dense video description method according to claim 1, characterized in that, The process of fusing and calculating the fused feature vectors of each video feature vector, starting from the second video feature vector and the preceding video feature vector of each video feature vector, until the last video feature vector is obtained, includes: Based on the first formula, fusion calculations are performed on each video feature vector and the preceding video feature vector of each video feature vector to obtain the fused feature vector of each video feature vector. The first formula is: ， in, For the first The fused feature vector of individual video feature vectors For the first Each video feature vector For the first The previous video feature vector of each video feature vector. , All are threshold values. This represents the total number of video feature vectors.

4. The dense video description method according to claim 1, characterized in that, The process of performing feature analysis on the target fused feature vectors of each video feature vector to obtain multiple event query feature vectors for each video feature vector includes: The target fusion feature vector of each video feature vector is encoded by an encoder to obtain the encoded feature vector of each video feature vector. The encoded feature vectors of each video feature vector are decoded by the decoder to obtain multiple event query feature vectors of each video feature vector.

5. The dense video description method according to claim 1, characterized in that, The process of performing video description analysis on each of the event query feature vectors of each of the video feature vectors to obtain the location confidence, generated sentence length, and multiple event descriptions of each event query feature vector includes: The multi-head attention mechanism algorithm is used to locate each event query feature vector of each video feature vector to obtain the location confidence of each event query feature vector. The LSTM network is used to perform video descriptions on each of the event query feature vectors of each video feature vector, thereby obtaining the generated sentence length and multiple event descriptions for each event query feature vector.

6. A dense video description device, characterized in that, include: The feature extraction module is used to import the original video, extract features from the original video, and obtain multiple video feature vectors. The fusion analysis module is used to perform fusion analysis on each of the video feature vectors to obtain the target fused feature vector of each of the video feature vectors. The feature analysis module is used to perform feature analysis on the target fused feature vectors of each of the video feature vectors to obtain multiple event query feature vectors for each of the video feature vectors; The video description analysis module is used to perform video description analysis on each of the event query feature vectors of each video feature vector to obtain the location confidence, generated sentence length, and multiple event descriptions of each event query feature vector. The statistics module is used to sum all the event query feature vectors to obtain the total number of event query feature vectors; The confidence analysis module is used to perform confidence analysis based on the positional confidence of each event query feature vector, the length of the generated sentence, and multiple event descriptions to obtain the target confidence of each event query feature vector. The video description result acquisition module is used to sort the target confidence scores of all event query feature vectors in descending order, and then... The event query feature vectors corresponding to the sorted target confidence scores are used as the video description results, where... This represents the total number of event query feature vectors. The multiple video feature vectors are arranged in chronological order, and the fusion analysis module is specifically used for: The first video feature vector is used as the fused feature vector of the first video feature vector; Starting from the second video feature vector, the fusion calculation is performed based on each video feature vector and the previous video feature vector of each video feature vector, until the last video feature vector is obtained, thus obtaining the fused feature vector of each video feature vector. By using multiple one-dimensional convolutional layers to extract features from the fused feature vector of the first video feature vector and the fused feature vectors of each video feature vector, the target fused feature vector of each video feature vector is obtained. The confidence analysis module is specifically used for: Based on the second formula, the confidence level is calculated according to the location confidence of each event query feature vector, the generated sentence length, and multiple event descriptions to obtain the target confidence level of each event query feature vector. The second formula is: ， in, For the first The target confidence score of the feature vector for each event query. For the first Locational reliability of the feature vector of an event query. For the first The first event query feature vector Event description, To generate sentence length, As a balance factor, is the modulation factor.

7. A dense video description system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the dense video description method as described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the dense video description method as described in any one of claims 1 to 5.

Images