A method and system for end-to-end dense description of shipborne surveillance video based on dynamic feature memory

By using a dynamic feature memory network to automatically describe events in shipborne surveillance videos, the problem of understanding jittery and blurry videos is solved, enabling fast and automated video content analysis.

CN117746328BActive Publication Date: 2026-11-10SHANGHAI MARITIME UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202311766684.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-21
Publication Date
2026-11-10
Estimated Expiration
2043-12-21

AI Technical Summary

Technical Problem

When faced with shaky and blurry video, existing shipborne surveillance videos are difficult to automatically record and understand events using traditional manual viewing and static image recognition methods, resulting in time-consuming and labor-intensive video understanding and analysis.

Method used

An end-to-end approach based on dynamic feature memory is adopted. Multi-scale features are extracted through a pre-trained feature extraction network, and combined with position encoding and dynamic feature memory networks. Multi-scale deformable self-attention layers and decoding networks are used to generate event proposals, descriptions and counts, thereby achieving automated understanding of video content.

Benefits of technology

It enables automated event description of jittery and blurry shipborne monitoring videos, reducing manual processing costs and improving the efficiency of video event understanding.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117746328B_ABST
    Figure CN117746328B_ABST
Patent Text Reader

Abstract

This invention discloses an end-to-end dense description method for shipborne surveillance video based on dynamic feature memory, comprising: Step 1, extracting features from the surveillance video using a pre-trained feature extraction network and obtaining multi-scale features through convolution processing; Step 2, adding positional encoding to the multi-scale features and inputting them into an encoding network for encoding, outputting the encoded multi-scale features; Step 3, inputting the encoded multi-scale features into a dynamic feature memory network to update the memory storage in the dynamic feature memory network, obtaining more detailed features; Step 4, inputting N learnable embeddings and the more detailed features together into a decoding network, converting the N learnable embeddings into a single feature output; Step 5, inputting the feature output into a localization module, a description module, and an event counting module, predicting event proposals, descriptions, and event counts according to these three parallel prediction modules.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of deep learning, specifically to an end-to-end dense description method and system for shipborne surveillance video based on dynamic feature memory. Background Technology

[0002] In the shipping industry, shipborne surveillance video plays a crucial role. Ships, as important means of transportation, bear the heavy responsibility of cargo transport, personnel safety, and navigation supervision. However, ships face various potential risks and challenges during navigation, such as collisions, fires, and cargo damage. Therefore, to ensure the safe and smooth operation of ships, ship monitoring systems are widely used. Shipborne surveillance video is a vital component of these systems. By installing cameras on ships and recording real-time video during navigation, ship managers and relevant departments can monitor the ship's status, navigation conditions, and cargo transport processes in real time. This surveillance video provides important evidence and data to assist ship managers in ship operation management, safety monitoring, and accident investigation. However, shipborne surveillance video also faces some challenges and problems. One of these is video jitter and blurring, which can be caused by multiple factors, including ship vibration, camera position and fixation, environmental factors, and the performance of the camera itself. These problems reduce the visual quality of the video, negatively impacting subsequent video understanding, analysis, and processing when relying on traditional manual viewing methods, and also making rapid video retrieval difficult.

[0003] To date, the main solution involves acquiring images from videos and running relevant object recognition algorithms on them to identify objects in the video in a static way. In most cases, techniques such as computer vision and machine learning can be used to identify specific objects in the images. Object recognition algorithms determine the category or label of an object by extracting its feature representation and matching it with predefined categories. Common object recognition algorithms include feature descriptor-based methods, such as Scale Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF), as well as deep learning-based methods, such as Convolutional Neural Networks (CNN) and Residual Neural Networks (ResNet). However, the results obtained from these methods still require post-processing human interpretation and analysis, a process that often consumes significant time and human effort.

[0004] In today's shipborne surveillance videos, what people want to see are the events recorded in the video, such as the process of accidents like ship collisions and grounding, the ignition point of fires and explosions, the spread and extinguishing of fires, and the loading and abnormal stacking of cargo. Therefore, simple target recognition methods cannot solve this problem. They cannot help people automatically record complete events when the surveillance video is shaky or blurry. The identified targets still need to be connected by people and carefully analyzed to understand the events. Summary of the Invention

[0005] This invention is made to solve the above-mentioned problems, and aims to provide an end-to-end dense description method and system for shipborne monitoring video based on dynamic feature memory.

[0006] This invention provides an end-to-end dense description method for shipborne surveillance video based on dynamic feature memory, comprising the following steps: Step 1, extracting features from the surveillance video using a pre-trained feature extraction network and obtaining multi-scale features through convolution processing; Step 2, adding positional encoding to the multi-scale features and inputting them into an encoding network for encoding, outputting the encoded multi-scale features; Step 3, inputting the encoded multi-scale features into a dynamic feature memory network to update the memory storage in the dynamic feature memory network, obtaining more detailed features; Step 4, inputting N learnable embeddings and the more detailed features together into a decoding network, converting the N learnable embeddings into a single feature output; Step 5, inputting the feature output into three parallel prediction modules, predicting event proposals, descriptions, and event counts according to the three parallel prediction modules, wherein the three parallel prediction modules are a localization module, a description module, and an event counting module.

[0007] The end-to-end shipborne monitoring video dense description method based on dynamic feature memory provided by this invention may also have the following features: Step 1 specifically includes the following steps: Step 1-1, dividing the monitoring video into non-overlapping segments, each segment having 64 frames; Step 1-2, extracting video features from the segments using pre-trained C3D, TSN, and I3D feature extraction networks to obtain segment-level video feature sequences; Step 1-3, scaling the temporal dimension of the feature map to a fixed length F using an interpolation algorithm to obtain a new segment-level video feature sequence; Step 1-4, passing the new segment-level video feature sequence through L convolutional layers to obtain multi-scale feature sequences with multiple resolutions from F to F / 2L.

[0008] The end-to-end shipborne monitoring video dense description method based on dynamic feature memory provided by this invention may also have the following features: Step 2 specifically includes the following steps: Step 2-1, the spatial location encoding and multi-scale features are processed through a multi-scale deformable self-attention layer to obtain the attention feature vector corresponding to the multi-scale feature map; Step 2-2, the attention feature vector is subjected to residual connection and normalization operations to obtain a normalized feature vector; Step 2-3, the normalized feature vector is input into a feedforward network for weighted processing to obtain a feature vector with the same dimension as the input vector; Step 2-4, the feature vector with the same dimension as the input vector is subjected to residual and normalization processing to obtain a feature vector representing the relationship between features of multi-scale video segments.

[0009] The end-to-end shipborne monitoring video dense description method based on dynamic feature memory provided by the present invention may also have the following features: wherein step 3 specifically includes the following steps: step 3-1, inputting the encoded multi-scale features into the dynamic feature memory network; step 3-2, the dynamic feature memory network updating the memory storage according to the newly input features.

[0010] The end-to-end shipborne monitoring video dense description method based on dynamic feature memory provided by the present invention may also have the following features: in step 3-2, the memory storage is updated in the dynamic feature memory network through two mechanisms, namely the "add" operation of the feature memory addition mechanism and the "delete" operation of the over-access decay mechanism.

[0011] The end-to-end shipborne monitoring video dense description method based on dynamic feature memory provided by this invention may also have the following features: Step 4 specifically includes the following steps: Step 4-1, the learnable embeddings are set to 100 vectors, and randomly initialized using the "0+position encoding" method to obtain the initialized learnable embedding vectors; Step 4-2, the initialized learnable embedding vectors are passed through a self-attention layer to optimize the position and range of the learnable embeddings, so that the range of each learnable embedding is as non-repetitive as possible, to obtain the optimized learnable embeddings; Step 4-3, the optimized learnable embeddings and more detailed features of the dynamic feature memory network are input together into a multi-scale deformable cross-attention layer. After residual normalization and feedforward network layer processing, attention weights and hidden states are obtained respectively, and N learnable embeddings are converted into a feature output; Step 4-4, the attention weights and hidden states are fed back into the dynamic feature memory network for training to obtain features that learn more event logic structures.

[0012] The end-to-end shipborne surveillance video dense description method based on dynamic feature memory provided by this invention may also have the following features: In step 5, the feature outputs converted from N learnable embeddings are input to the localization module, the description module, and the event counting module. In the localization module, a multilayer perceptron is used to perform bounding box prediction and binary classification operations for each learnable embedding. In the description module, a reference point is first obtained through the learnable embedding as the reference position of the event center point. Then, sampling continues around the center point, limiting the sampling to a region to ensure that the generated word sequences describe the same event. Next, the attention weights on the sampling points are predicted to assist the long short-term memory network in generating the description. In the event counting module, the most salient information of the learnable embedding is first compressed into a global feature vector, and then a fixed-size vector r is predicted, where each value represents the probability of a specific number.

[0013] The end-to-end shipborne surveillance video dense description method based on dynamic feature memory provided by the present invention may also have the following features: wherein, bounding box prediction aims to predict the relative offset of the actual event segment with respect to the reference point, specifically the offset of the center position and length, and binary classification aims to generate the foreground confidence of each learnable embedding.

[0014] The end-to-end shipborne monitoring video dense description method based on dynamic feature memory provided by the present invention may also have the following feature: in the event counting module, during the inference phase, the number of predicted events is Nset, and the final output is obtained by selecting the top Nset events with accurate boundaries and good descriptions from N learnable embeddings.

[0015] This invention provides an end-to-end shipborne surveillance video dense description system based on dynamic feature memory, comprising: a feature extraction unit that extracts features from the surveillance video using a pre-trained feature extraction network and obtains multi-scale features through convolution processing; a target prediction unit that adds positional encoding to the multi-scale features and inputs them into an encoding network for encoding, outputting encoded multi-scale features; inputting the encoded multi-scale features into a dynamic feature memory network to update the memory storage in the dynamic feature memory network to obtain more detailed features; inputting N learnable embeddings and the more detailed features together into a decoding network to convert the N learnable embeddings into a single feature output; and inputting the feature output into three parallel prediction modules that predict event proposals, descriptions, and event counts, respectively. The three parallel prediction modules are a localization module, a description module, and an event counting module.

[0016] The role and effect of invention

[0017] The end-to-end shipborne surveillance video dense description method based on dynamic feature memory according to the present invention includes the following steps: Step 1, extracting features from the surveillance video using a pre-trained feature extraction network and obtaining multi-scale features through convolution processing; Step 2, adding position encoding to the multi-scale features and inputting them into an encoding network for encoding, outputting the encoded multi-scale features; Step 3, inputting the encoded multi-scale features into a dynamic feature memory network, updating the memory in the dynamic feature memory network to obtain more detailed features; Step 4, inputting N learnable embeddings and more detailed features together into a decoding network, converting the N learnable embeddings into a single feature output; Step 5, inputting the feature output into three parallel prediction modules, predicting event proposals, descriptions, and event counts according to the three parallel prediction modules, wherein the three parallel prediction modules are a localization module, a description module, and an event counting module.

[0018] Therefore, this invention aims to generate video summary understanding from raw shipborne surveillance video using a model, thereby reducing manual costs and facilitating faster retrieval and understanding of events in the video. This method eliminates the time-consuming and labor-intensive drawbacks of traditional methods that rely on manual viewing and processing of surveillance videos and manual analysis of image recognition results, helping to quickly understand shipborne surveillance video content exhibiting jitter and blurriness.

[0019] In summary, this invention has the following advantages: First, it can generate video descriptions for events in shipborne monitoring videos that suffer from blurriness or jitter. Second, it eliminates the drawback of relying on extensive manual processing of shipborne monitoring videos, and it also eliminates the limitation of relying on static images to identify video objects, which prevents the formation of an understanding of the video content. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall structure of the end-to-end shipborne monitoring video dense description method based on dynamic feature memory in an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of the C3D network structure in an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram of the I3D network structure in an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of the Inception-v1 structure in an embodiment of the present invention;

[0024] Figure 5 This is a schematic diagram of the TSN network structure in an embodiment of the present invention;

[0025] Figure 6This is a schematic diagram of the feature memory addition mechanism in an embodiment of the present invention;

[0026] Figure 7 This is a schematic diagram of the over-access decay mechanism in an embodiment of the present invention;

[0027] Figure 8 This is a schematic diagram of network (a) described in an embodiment of the present invention;

[0028] Figure 9 This is a schematic diagram illustrating network (ii) in an embodiment of the present invention; Detailed Implementation

[0029] To make the technical means, creative features, objectives and effects of the present invention easy to understand, the following embodiments, in conjunction with the accompanying drawings, specifically illustrate the end-to-end shipborne monitoring video dense description method and system based on dynamic feature memory.

[0030] Example 1

[0031] In this embodiment, an end-to-end dense description method for shipborne monitoring video based on dynamic feature memory is provided.

[0032] Figure 1 This is a schematic diagram of the overall structure of the end-to-end shipborne monitoring video dense description method and system based on dynamic feature memory in this embodiment.

[0033] like Figure 1 As shown, the end-to-end shipborne monitoring video dense description method based on dynamic feature memory involved in this embodiment includes the following steps:

[0034] Step S1 involves extracting features from the surveillance video using a pre-trained feature extraction network and then using convolutional processing to obtain multi-scale features. The specific implementation process is as follows:

[0035] Step S1-1: Divide the surveillance video into non-overlapping segments, each segment having 64 frames.

[0036] Steps S1-2 involve extracting video features from the segmented video segments using pre-trained C3D, I3D, and TSN feature extractors, respectively, to obtain segment-level video feature sequences.

[0037]

[0038] In the formula, Len is the number of video segments.

[0039] Figure 2 This is a schematic diagram of the C3D network structure in this embodiment.

[0040] The C3D technology provided by Tran D et al. can simultaneously extract temporal and spatial information from video, such as... Figure 2 As shown, its network structure mainly consists of 8 convolutional layers, 5 pooling layers, 2 fully connected layers, and 1 softmax output layer. Except for pool1, the convolutional kernels of the pooling layers are all 2×2×2 with a stride of 2×2×2, while the convolutional kernel of pool1 is 1×2×2 with a stride of 1×2×2. The '1' in the convolutional kernel represents depth, aiming to preserve temporal and motion information early on and avoid premature fusion. In the early stages of training the C3D feature extraction network, 5 two-second segments were extracted from each video in the Sports-1M dataset, and the frame size of each segment was adjusted to 128×127. During training, multiple segments were randomly cropped to 16×112×112, with a mini-batch size of 30. The initial learning rate was 0.003, decaying tenfold every 4 epochs, and training stopped after 16 epochs.

[0041] Figure 3 This is a schematic diagram of the I3D network structure in this embodiment.

[0042] Figure 4 This is a schematic diagram of the Inception-v1 structure in this embodiment.

[0043] like Figure 3 As shown, for the initial video input, I3D uses 3D convolutional networks on both the left and right sides. The left side inputs multiple frames of RGB images, and the right side inputs optical flow. In I3D, Inception-v1 is used as the base network, as follows: Figure 4 As shown, the stride of the first convolutional layer is 2. Besides Inception (Inc.), there are four max-pooling layers with a maximum stride of 2 and one 7×7 average pooling layer. Figure 4 The right side shows the network architecture diagram of Inception (Inc.). Iashin et al. used the Kinetics dataset to pre-train the I3D feature extraction model, with 64 frames of RGB and optical flow data as input, and a video frame rate of 25. During the extraction process, the model is segmented into 64 frames, and outputs a 1024-dimensional RGB feature vector and an optical flow feature vector for each 64 frames. Finally, these two vectors are weighted and averaged to obtain a 1024-dimensional output vector for each 64-frame video segment.

[0044] Figure 5 This is a schematic diagram of the TSN network structure in this embodiment.

[0045] like Figure 5As shown, TSN consists of two stages: segmentation and feature extraction. In the segmentation stage, the input video sequence is divided into multiple equal-length segments, each containing multiple consecutive video frames. In the feature extraction stage, for each segment, TSN uses a pre-trained convolutional neural network to extract features from each video frame. Finally, average or max pooling is applied to the feature vectors of all segments to obtain the feature representation of the entire video sequence. During training, the video is downsampled every 0.5 seconds, and one-dimensional appearance and optical flow features are extracted for each frame. For appearance features, the output of the "Flatten673" layer in ResNet-200 is used. For optical flow features, BN-Inception is chosen as a building block, and the output of the pooling layer is obtained. Then, the two feature vectors obtained from the ResNet-200 and BN-Inception networks are concatenated and further encoded using a linear layer.

[0046] Steps S1-3 involve scaling the temporal dimension of the feature map to a fixed length F using an interpolation algorithm to obtain a new segment-level video feature sequence for batch processing.

[0047] Steps S1-4: Use L convolutional layers (stride 2, kernel size 3) to obtain values ​​from F to F / 2. L This involves multiple resolution feature sequences, i.e., multi-scale features. During processing, information about the relative or absolute positions of the sequences is injected.

[0048] Step S2 involves adding positional encoding to the multi-scale features and inputting it into an encoding network for encoding, outputting the encoded multi-scale features. The specific process is as follows:

[0049] Step S2-1: In this embodiment, the encoding network has M encoding modules, each with the same structure: a multi-scale deformable self-attention layer and a feedforward network (FNN). The FFN consists of fully connected layers (FC) and the ReLU activation function. The addition of the ReLU function adds non-linear components to the model, further enhancing its learning ability. Because the transformer architecture lacks positional information, positional information needs to be added to each attention layer to supplement it. For example... Figure 1 As shown, Figure 1 Only one encoding module is shown, which includes a multi-scale deformable self-attention layer and a feedforward network FFN. After each processing by the multi-scale deformable self-attention layer or the feedforward network, residual connections and normalization operations are performed to prevent network degradation and gradient vanishing, making the network easy to train.

[0050] The obtained features and location information are processed by a multi-scale deformable self-attention layer to obtain the attention feature vector corresponding to the multi-scale feature map.

[0051] Step S2-2: The attention feature vector is subjected to residual connection and normalization operations through the add&normalize layer to obtain the normalized feature vector.

[0052] Step S2-3: Input the normalized feature vector into the feedforward network for weighted processing to obtain a feature vector with the same dimension as the input vector.

[0053] Steps S2-4 involve performing residual and normalization processing on the feature vectors with the same dimension as the input vectors, and then inputting the output features into the dynamic feature memory network. For example... Figure 1 As shown, the output of the encoding network is the relationship between features of video segments at multiple scales. The output feature representation is:

[0054]

[0055] In the formula, M is the number of coding modules in the coding network, Len is the number of video segments, and L is the number of feature scale layers.

[0056] Step S3 involves inputting the encoded multi-scale features into the dynamic feature memory network, updating the memory storage within the network, and obtaining more detailed features. The specific process is as follows:

[0057] Step S3-1: Input the encoded multi-scale features into the dynamic feature memory network. The dynamic feature memory network is the core of this embodiment. It enhances the attention mechanism in the decoding network, facilitating the model's learning of the logical structure of video events. The encoded feature sequence is input into the dynamic feature memory network. The decoder enhanced by the feature memory module has T layers, where T = {1, ..., t} is the number of video segments. The position and flow of the dynamic feature memory network in the model are as follows... Figure 1 As shown.

[0058] Step S3-2: The dynamic feature memory network updates the memory storage based on the newly input features.

[0059] The dynamic feature memory network does not use the same feature f at every decoding step, but rather uses an updated feature memory M at every step t. t ={m t,1 ,…,m t,Len},m t,1 This represents the memory features of the first video segment in the t-th step. By dynamically updating the features in memory, visual attention can move coherently across different events, enhancing event coherence.

[0060] "Add" operation: This embodiment utilizes historical attention weights {α} t-W ,…,α t} and history window W will M t Support updated to M t+1 α t M represents the feature memory at step t. t The attention weights are averaged across multiple attention heads. The attention history is then aggregated. This also makes the attention weight of the most recent step more important:

[0061]

[0062]

[0063] In the formula, w j It is the weight of historical decay. Used to update feature memory.

[0064] Figure 6 This is a schematic diagram of the feature memory addition mechanism in this embodiment.

[0065] Figure 7 This is a schematic diagram of the over-access decay mechanism in this embodiment.

[0066] In dynamic feature memory networks, updates to the memory storage are primarily achieved through two mechanisms: the "addition" operation of the feature memory addition mechanism and the "deletion" operation of the over-access decay mechanism. For example... Figure 6 As shown, the "Add" operation gradually adds more video clips to the memory storage. Figure 7 As shown, the "Delete" operation is used to delete fragment features that have already been described.

[0067] For the feature memory addition mechanism of the dynamic feature memory network: in order to ensure that the event description follows the chronological order, the encoded video features f enc The features are gradually revealed to the participating feature memory storage M. M0 is first initialized as follows:

[0068] M0=u0·f enc

[0069]

[0070] In the formula, This indicates the exposure state at step t, and records the proportion of each video segment feature added to the feature memory. t,i ∈[0,1] is continuously updated, where u t,i =1 indicates that features of the i-th segment should not be added. S is the initial window length, which can be equal to u.t The collaboration allows the decoder to focus on the beginning of the video first, rather than randomly focusing on the video.

[0071] To determine whether a new feature should be "added" to the memory storage at event t, an add gate was first proposed. When newly added video features are highly correlated with video features in memory, the update of feature memory should be relatively infrequent. The calculation is as follows:

[0072]

[0073] In the formula, σ is the sigmoid function, and h t Let f be the output hidden state at step t. add It is caused by θ add A parameterized fully connected network. Then, based on the features and the previous context... The context is used to determine whether to add features to the feature memory storage based on their relevance. This allows the model to maintain consistency between events and descriptions, which is computed based on the aggregated attention history:

[0074]

[0075] Therefore, the probability of adding each video segment feature to the feature memory storage is calculated as follows:

[0076]

[0077] In the formula, f vis Is with f add A similar fully connected network.

[0078] Add gates based on features of each video segment Sum of probabilities This embodiment can gradually add video features to the memory storage device, that is:

[0079]

[0080]

[0081] In the formula, These are intermediate memory features, which may be further processed later through a "deletion" operation.

[0082] "Deletion" operation: To prevent the decoder from focusing on only a few prominent frames, this embodiment encourages the model to focus on more video frames by weakening already accessed video features. Similar to the feature memory addition mechanism, this embodiment uses... As a deletion gate, it determines whether to delete video features at step t, and its calculation method is as follows:

[0083]

[0084] Considering that attention weights can represent the access intensity of each video segment feature, this embodiment updates the feature memory storage based on attention weights, and the calculation method is as follows:

[0085]

[0086]

[0087] In the formula, It is the historical hidden state of computation, f vis Is with f del Similar to a fully connected network, del represents deletion. Finally, the feature memory is updated to M using the deletion probability and deletion gate. t+1 The calculation method is as follows:

[0088]

[0089] Step S4: Input the N event queries along with more detailed features into the decoding network, and transform the N event queries into a single feature output. The specific process is as follows:

[0090] Step S4-1: Set the event queries to 100 vectors and randomly initialize them using the "0 + position encoding" method to obtain the initialized event queries.

[0091] The decoding network module in this embodiment includes a self-attention layer, a multi-scale deformable cross-attention layer, and an FFN layer. Each layer is followed by a residual connection and a normalization operation, such as... Figure 1 As shown, this module is called the deformable transformer decoder. The multi-scale deformable cross-attention layer needs to work in conjunction with the aforementioned dynamic feature memory network. The features in the dynamic feature memory network need to be input into the multi-scale deformable cross-attention layer to help learn the logic of events.

[0092] The decoder is designed to extract from N event queries Given the video features and their corresponding scalar reference points p j You can directly query event-level features. Note that p... j By q j A linear layer with a sigmoid activation function is used for prediction. The output query features and reference points are denoted as...

[0093] Similar to the encoding network, the decoder in this embodiment also has M decoding modules, each containing a multi-scale deformable cross-attention layer and a self-attention layer. Since the decoder is permutation-invariant, it requires N different input features to produce N different output results. These input features are learned positional encoding information, referred to as event queries. The event queries are initially set to 100 vectors, randomly initialized using a "0 + positional encoding" method.

[0094] Step S4-2: The initialized event query vector is passed through a self-attention layer to optimize the position and range of the event queries, so that the range of each event query is as unique as possible, resulting in optimized event queries.

[0095] Step S4-3: The optimized event queries and the more detailed features from the dynamic feature memory network are input into the multi-scale deformable cross-attention layer. After residual normalization and processing by the feedforward network layer (FFN layer), the attention weights α are obtained. t and hidden state h t This transforms N event queries into a single feature output.

[0096] Step S4-4, adjust the attention weight α t and the hidden state h t The feedback is fed into a dynamic feature memory network for training, which learns more features of the logical structure of events.

[0097] Step S5: The feature outputs converted from the N event queries are input into the localization module, the description module, and the event counting module, respectively, to predict the event proposal, description, and number of events.

[0098] The localization module in this embodiment aims to predict event boundaries using confidence scores to obtain event proposals. Two operations are performed on each event query in the localization module: boundary prediction and binary classification. Boundary prediction aims to predict the two-dimensional relative offset (i.e., center and length) with respect to the ground-truth proposal based on a reference point. Binary classification aims to generate the localization confidence score for each event query. Both boundary prediction and binary classification are implemented using a multilayer perceptron. These two operations result in a set of tuples. To represent the detected event, where It is the location reliability of the event query. and This indicates the start and end of event j.

[0099] Figure 8 This is a schematic diagram of network (I) in this embodiment.

[0100] The description module in this embodiment does not generate proposals in advance before prediction, unlike the traditional "localization-description" method. Therefore, it cannot access event boundaries, which increases the difficulty of learning the relationship between language words and video features.

[0101] To address this issue, this embodiment employs a deformable soft attention (DSA) mechanism to force the soft attention weights to concentrate on a small region around the reference point. Soft attention is a widely used module in video description, used to dynamically determine the importance of each frame when generating words.

[0102] Specifically, when the t-th word w is generated t At that time, firstly, based on the language query h jt and event query From each multi-scale feature f l K sampling points are generated, note that t here is different from T in the dynamic feature memory network. Where h... jt This represents the hidden state in the LSTM. Then, K×L sampling points are used as key / value pairs. As a query in DSA. Because the sampling points are distributed around the reference point. The surrounding area, therefore the output feature z of DSA jt It is confined to a relatively small area.

[0103] like Figure 8 As shown, DSA-LSTM (LSTM with DSA) will use context features z jt Event query characteristics And the preceding word w j,t-1 The concatenated result is used as input. The word w jt The probability is in h jt The above is obtained from a fully connected (FC) layer with softmax. With each iteration of the DSA-LSTM, a sentence S can be obtained. j ={w j1 ,…,w j M j}. In the formula, M j Let j be the length of the sentence, where j is the j-th event query.

[0104] Figure 9 This is a schematic diagram illustrating network (II) in an embodiment of the present invention.

[0105] like Figure 9As shown, the process of describing the network is vividly expressed from different perspectives. First, reference points are obtained through event queries as reference positions for the center point of the event. Then, sampling continues around the center point, limiting the sampling to a region to ensure that the generated word sequences describe the same event. Next, attention weights on the sampling points are predicted to assist LSTM in generating descriptions.

[0106] In this embodiment, the event counting module addresses the issue that too many detected event proposals can lead to redundant descriptions with high similarity, while too few detected events result in incomplete video descriptions and potential loss of crucial information. To generate an appropriate number of proposals, this embodiment places the event counter alongside two other modules (a localization module and a description module) to detect the number of event proposals contained in the video. It consists of a max-pooling layer and a fully connected (FC) layer with a softmax activation function. This module first selects the most salient information from the event queries. Compress it into a global feature vector, and then predict a fixed-size vector r, where each value represents the probability of a particular number.

[0107] During the inference phase, the number of predicted events N set =argmax(r). The final model output is obtained by selecting the top N events with accurate boundaries and good descriptions from N event queries. set This is obtained from a number of events. The events are sorted by their confidence score, which is calculated as follows:

[0108]

[0109] In the formula, This represents the probability of the generated words. However, the confidence score of each word cannot reflect the confidence score of the entire sentence because the description network assigns higher confidence scores to shorter sentences. Therefore, a modulation factor γ is added to correct for the influence of description length, and μ is a balancing factor.

[0110] During training, the model generates a set of N event proposals and descriptions. To match the ground-truth proposals with the generated proposals, this invention employs the Hungarian algorithm, with a matching loss of:

[0111] C = α iou L iou +α cls L cls

[0112] In the formula, L iou To predict IOU on proposals and ground-truth proposals, L clsThis represents the focal loss between the predicted classification score and the ground truth label. Finally, the prediction loss is calculated by weighting the IOU loss, classification loss, counting loss, and description loss.

[0113] L=λ iou L iou +λ cls L cls +λ ec L ec +λ cap L cap

[0114] Example 2

[0115] In this embodiment, an end-to-end shipborne monitoring video dense description system based on dynamic feature memory is provided.

[0116] The end-to-end shipborne monitoring video dense description system based on dynamic feature memory in this embodiment includes a feature extraction unit and a target prediction unit. The feature extraction unit is implemented using the method in step S1 of embodiment 1, and the target prediction unit is implemented using the method in steps S2 to S5 of embodiment 1.

[0117] The role and effect of the embodiments

[0118] The end-to-end shipborne surveillance video dense description method based on dynamic feature memory involved in this embodiment includes the following steps: Step 1, extracting features from the surveillance video using a pre-trained feature extraction network and obtaining multi-scale features through convolution processing; Step 2, adding position encoding to the multi-scale features and inputting them into an encoding network for encoding, outputting the encoded multi-scale features; Step 3, inputting the encoded multi-scale features into a dynamic feature memory network, updating the memory storage in the dynamic feature memory network to obtain more detailed features; Step 4, inputting N learnable embeddings and more detailed features together into a decoding network, converting the N learnable embeddings into a single feature output; Step 5, inputting the feature output into three parallel prediction modules, predicting event proposals, descriptions, and event counts according to the three parallel prediction modules, wherein the three parallel prediction modules are a localization module, a description module, and an event counting module.

[0119] Therefore, the above embodiments aim to generate a summary understanding of the original shipborne surveillance video through a model, thereby reducing manual costs and facilitating faster retrieval and understanding of events in the video. This method eliminates the time-consuming and labor-intensive drawbacks of traditional manual viewing and processing of surveillance videos and relying on manual understanding and analysis of image recognition results, helping to quickly understand the content of shipborne surveillance videos with jitter and blurriness.

[0120] In summary, the above embodiments have the following advantages: First, they can generate video descriptions for events in shipborne monitoring videos that suffer from blurriness or jitter. Second, they eliminate the drawbacks of relying on extensive manual processing of shipborne monitoring videos, and also overcome the limitation of failing to understand video content by relying on static image recognition of video objects.

[0121] The above embodiments are preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention.

Claims

1. A method for dense description of end-to-end shipborne surveillance video based on dynamic feature memory, characterized in that, Includes the following steps: Step 1: Use a pre-trained feature extraction network to extract features from the surveillance video and use convolution to obtain multi-scale features; Step 2: Add positional encoding to the multi-scale features and input them into the encoding network for encoding, and output the encoded multi-scale features; Step 3: Input the encoded multi-scale features into the dynamic feature memory network, update the memory storage in the dynamic feature memory network, and obtain more detailed features; Step 4: Input the N learnable embeddings and the more detailed features into the decoding network, and convert the N learnable embeddings into a single feature output; Step 5: Input the feature output into three parallel prediction modules, and predict the event proposal, description, and number of events respectively based on the three parallel prediction modules. The three parallel prediction modules are a localization module, a description module, and an event counting module. Step 3 specifically includes the following steps: Step 3-1: Input the encoded multi-scale features into the dynamic feature memory network; Step 3-2: The dynamic feature memory network updates the memory storage based on the newly input features. In step 3-2, the dynamic feature memory network updates the memory storage through two mechanisms. The operations of the two mechanisms are the "add" operation of the feature memory addition mechanism and the "delete" operation of the over-access decay mechanism; In step 3, the feature memory addition mechanism of the dynamic feature memory network is as follows: the encoded video features are... Gradually revealed to the participating feature memory storage M, Add gates based on features of each video segment Sum of probabilities Gradually add video features to the memory storage, that is: In the formula, It is an intermediate memory feature. Deletion operation As a deletion gate, it determines whether to delete video features at step t, and its calculation method is as follows: The feature memory storage is updated based on attention weights, and the calculation method is as follows: In the formula, It is the hidden state of the computation history. Is with Similar to a fully connected network, del represents deletion. Finally, the feature memory is updated using the deletion probability and deletion gate. The calculation method is as follows: 。 2. The end-to-end shipborne monitoring video dense description method based on dynamic feature memory as described in claim 1, Its features are: in, Step 1 specifically includes the following steps: Step 1-1: Divide the surveillance video into non-overlapping segments, each segment having 64 frames; Steps 1-2: Extract video features from the segments using pre-trained C3D, TSN, and I3D feature extraction networks to obtain segment-level video feature sequences. Steps 1-3: The time dimension of the feature map of the segment-level video feature sequence is scaled to a fixed length F by an interpolation algorithm to obtain a new segment-level video feature sequence; Steps 1-4: The new segment-level video feature sequence is passed through L convolutional layers to obtain multi-scale feature sequences with multiple resolutions from F to F / 2L.

3. The end-to-end shipborne monitoring video dense description method based on dynamic feature memory as described in claim 1, Its features are: in, Step 2 specifically includes the following steps: Step 2-1: The spatial location encoding and the multi-scale features are processed through a multi-scale deformable self-attention layer to obtain the attention feature vector corresponding to the multi-scale feature map; Step 2-2: Perform residual connection and normalization operations on the attention feature vector to obtain the normalized feature vector; Steps 2-3: Input the normalized feature vector into the feedforward network for weighted processing to obtain a feature vector with the same dimension as the input vector; Steps 2-4 involve performing residual and normalization processing on the feature vectors with the same dimension as the input vectors to obtain feature vectors representing the relationships between features of multi-scale video segments.

4. The end-to-end shipborne monitoring video dense description method based on dynamic feature memory as described in claim 1, Its features are: in, Step 4 specifically includes the following steps: Step 4-1: The learnable embeddings are set to 100 vectors, which are randomly initialized using the "0+position encoding" method to obtain the initialized learnable embedding vectors. Step 4-2: The initialized learnable embedding vector is passed through a self-attention layer to optimize the position and range of the learnable embedding, so that the range of each learnable embedding is as unique as possible, and thus the optimized learnable embedding is obtained. Step 4-3: The optimized learnable embeddings and the more detailed features of the dynamic feature memory network are input into the multi-scale deformable cross attention layer. After residual normalization and feedforward network layer processing, attention weights and hidden states are obtained respectively, and N learnable embeddings are converted into one feature output. Step 4-4: Feed the attention weights and hidden states into the dynamic feature memory network for training to obtain features that learn more event logic structures.

5. The end-to-end shipborne monitoring video dense description method based on dynamic feature memory according to claim 1, characterized in that: in, In step 5, the feature outputs converted from the N learnable embeddings are input into the localization module, the description module, and the event counting module. In the localization module, a multilayer perceptron is used to perform bounding box prediction and binary classification for each learnable embedding. In the description module, a reference point is first obtained through learnable embeddings as the reference location for the event center. Then, sampling continues around the center point, restricting the sampling to a certain region to ensure that the generated word sequences all describe the same event. Next, attention weights at the sampling points are predicted to assist the Long Short-Term Memory network in generating the description. In the event counting module, the most salient information of the learnable embedding is first compressed into a global feature vector, and then a fixed-size vector r is predicted, where each value represents the probability of a particular number.

6. The end-to-end shipborne monitoring video dense description method based on dynamic feature memory according to claim 5, characterized in that: in, The bounding box prediction aims to predict the relative offset of the actual event fragment relative to a reference point, specifically the offset in center position and length. The binary classification aims to generate a foreground confidence score for each learnable embedding.

7. The end-to-end shipborne monitoring video dense description method based on dynamic feature memory according to claim 5, characterized in that: in, In the event counting module, during the inference phase, the number of predicted events is Nset. The final output is obtained by selecting the top N sets of events with accurate boundaries and good descriptions from N learnable embeddings.

8. An end-to-end shipborne surveillance video dense description system based on dynamic feature memory, used to implement the method as described in claim 1, characterized in that, include: Feature extraction unit: Uses a pre-trained feature extraction network to extract features from surveillance videos and uses convolution processing to obtain multi-scale features; The target prediction unit adds positional encoding to the multi-scale features and inputs them into an encoding network for encoding, outputting encoded multi-scale features. The encoded multi-scale features are then input into a dynamic feature memory network to update the memory storage in the dynamic feature memory network, resulting in more detailed features. N learnable embeddings and the more detailed features are input into a decoding network, converting the N learnable embeddings into a single feature output. This feature output is then input into three parallel prediction modules, which predict event proposals, descriptions, and event counts, respectively. The three parallel prediction modules are a localization module, a description module, and an event counting module.