A video shadow detection method and device based on a dynamic prompt memory network

Abstract

The application discloses a video shadow detection method and device based on a dynamic prompt memory network, which can process confused shadows with the help of a projected shadow object, can aggregate time information without accumulating errors, and comprises a dynamic prompt module and a prompt-based memory module; the dynamic prompt module converts semantic information provided by a visual base model DINOv2 into a shadow mask when direct encoding of a shadow is difficult, local prompting focuses on semantic conversion and identifies a shadow position based on semantic information, and global prompting identifies a shadow boundary based on color and texture information; the memory module is used for solving deformation and long-term time consistency problems, preventing error accumulation, using local prompting as a time matching agent, reducing memory usage, and minimizing dependence on historical shadow masks; and the method can improve detection performance in a complex scene and maintain time consistency of results during long-term detection.

Description

TECHNICAL FIELD

[0001] The application relates to a video shadow detection method and device based on a dynamic prompt memory network. BACKGROUND

[0002] Shadows are crucial for inferring scene geometry and illumination, and accurate shadow detection enhances downstream vision tasks such as segmentation, tracking, and scene reconstruction. Most existing work has focused on image shadow detection (ISD), and with the development of new deep learning techniques, algorithms for image shadow detection have achieved good results. In recent years, researchers have attempted to extend algorithms to video shadow detection (VSD). However, shadows in videos are often confused with the background, and can quickly deform between video frames, making video shadow detection a more challenging task than image shadow detection.

[0003] Early networks focus on the correspondence between adjacent frames, and they fuse the information of adjacent frames using attention mechanisms or convolutional layers to encode inter-frame temporal information, and these algorithms have achieved good results in simple scenes. With the development of Transformers and diffusion models, VSD methods have begun to adopt the latest architectures to improve performance. By introducing new attention mechanisms and boundary information, researchers have made some contributions to temporal modeling and deformation problems.

[0004] However, there are still two main problems with VSD algorithms that need to be further addressed.

[0005] First, when the camera moves quickly or the light changes rapidly, existing methods often fail to fully capture the shadow area. This is because they only focus on potential dark areas, and they are only trained on shadow datasets, lacking prior knowledge about shadow projection instances. We believe that in cases where shadows are relatively chaotic, it may be difficult for the network to directly extract features. Therefore, it is very important to use information about shadow projection instances to help identify their shadows.

[0006] Second, existing methods do not fully utilize the temporal information in videos. In VSD methods, there are three architectures that can fuse multi-frame features: multi-branch networks, segment-based networks, and memory networks. Multi-branch networks can enhance the information of adjacent frames, but since they focus on short-term features, they are difficult to capture rapid motion and deformation. Segment-based networks input multiple frames as a segment and perform segmentation through intra-segment information, but they lack inter-segment temporal consistency and cannot utilize past features to further eliminate noise. Memory networks can encode long-term temporal features by saving masks, but they largely rely on the results of the previous frame, and are prone to accumulate errors. SUMMARY

[0007] The application aims to solve the problems of inaccurate detection and insufficient time information utilization in complex scenes, and provides a video shadow detection method and device based on a dynamic prompt memory network (DPMNet), which can simultaneously capture the framework of short-term and long-term information; the dynamic prompt memory network in the application introduces time modeling for the visual base model DINOv2 and can propagate its feature representation during long-term processing.

[0008] Technical scheme: To achieve the above-mentioned purpose, the technical scheme adopted by the application is as follows:

[0009] A video shadow detection method based on a dynamic prompt memory network, comprising a dynamic prompt module (DPM) and a prompt-based memory module (PMM); the dynamic prompt module utilizes local prompts and global prompts to convert semantic information provided by the visual base model DINOv2 into a shadow mask when direct encoding of shadows is difficult, the local prompts focus on semantic conversion and identify shadow positions based on semantic information, and the global prompts identify shadow boundaries based on color and texture information; the memory module is used to solve the problems of deformation and long-term time consistency, and simultaneously prevent error accumulation, the memory module uses local prompts as a proxy for time matching, reduces memory usage, and minimizes dependence on historical shadow masks.

[0010] Specifically, the method comprises the following steps:

[0011] (1) Given a video segment I, input the video segment I into each layer encoder of a pre-trained visual base model DINOv2 to obtain an original feature group D={D1, D2, …D i , …, D N}, the i-th layer encoder outputs the i-th layer original feature D i , and the total number of encoders is N;

[0012] (2) Randomly initialize a local prompt vector group L={L1, L2, …L i , …, L N} and a global prompt vector G, L i is the i-th layer local prompt vector; interactively fuse the original feature D i with the local prompt vector L i and the global prompt vector G to obtain two kinds of enhanced features, and perform weighted summation on the two kinds of enhanced features to generate the fusion enhanced feature of the i-th layer encoder;

[0013] (3) Repeat step (2) for each layer of original features to obtain fusion enhanced features of each layer; segmentally fuse the fusion enhanced features of each layer to obtain an encoding feature

[0014] (4) Encoding features Split into T single-frame features according to the time dimension At the same time, the original feature D i Split into T single frames according to the time dimension The local hint vector L i Copy it into T parts according to the time dimension, and make the tth single frame As the original feature D i The current single frame, the tth local hint vector As the corresponding current single frame and single frame features The local prompt sub-vector of Match and interact with key-value pairs stored in the memory bank to make single-frame features Obtain the semantic information of a historical single frame and generate memory enhancement feature Z i ;Select single frame features The corresponding local hint subvector of the last layer encoder Use key-value encoder to local hint sub-vector and single-frame features Encode and generate the current single frame The key-value pair of the current single frame The key-value pairs of the historical single frame are stored in the memory bank, and the key-value pairs of the historical single frame are selectively eliminated according to the first-in-first-out rule;

[0015] (5) Through a lightweight decoder, gradually improve the memory enhancement feature Z i The resolution of the memory enhancement feature Z t Decoded into shadow mask Y t .

[0016] Specifically, in step (1), a video clip is given The video clip I is input into each layer encoder of the pre-trained visual basis model DINOv2, and the i-th layer encoder outputs the i-th layer original feature Represents the original feature D i The current single frame, H and W represent the original features D i In order to retain the original generalization ability of the visual base model DINOv2 to the maximum extent, during the training phase, the encoder of the visual base model DINOv2 is frozen, the gradient information is not recorded, and the parameters of the encoder of the visual base model DINOv2 are not updated.

[0017] Specifically, in the step (2), the global cue vector G identifies the shadow boundary based on color and texture information, and the local cue vector L identifies the shadow position based on semantic information; the dynamic cue module is used to make the original features of each layer interact with the two kinds of cue vectors for fusion, so that the two kinds of cue vectors fully adjust the original features, and the original features are converted into fusion enhanced features for video shadow detection; the dynamic cue module includes a local cue module (LPB) and a global cue module (GPB); the fusion enhanced feature acquisition process includes the following steps:

[0018] (21) identifying the shadow position through the local cue module, specifically including the following steps:

[0019] (211) initializing a set of N l local cues with a length of C for each layer of encoder through the local cue module, and the local cue vector of the i-th layer is denoted as Meanwhile, the local cue vector L i is copied into T parts in the time dimension, and the t-th part of the local cue vector L is used as the local cue sub-vector corresponding to the current single frame and the single frame feature ;

[0020] (212) performing cross-attention operation on the local cue sub-vector and the current single frame , so that the local cue sub-vector selectively aggregates effective information from the current single frame ; in order to introduce time clues to fuse the intra-clip semantic information, all the aggregated effective information in each layer of encoder needs to be spliced together for self-attention operation, and the local cue vectors from different single frames are fused:

[0021]

[0022] wherein CrossAttn and SelfAttn represent cross-attention mechanism and self-attention mechanism respectively, and MLP represents multi-layer perception, and the output of the multi-layer perception MLP is The multi-layer perception MLP includes two fully connected layers, one of which is used to reduce the number of channels of the local cue vector from C to C', so as to reduce the calculation amount, and the other is used to restore the number of channels of the local cue vector from C' to C, so that the local cue extracts more semantic information;

[0023] (213) using a method similar to attention mechanism to make the local cue vector dynamically adjust the features of the original feature D i according to different positions of the image, so that the current single frame The attention map of the current single frame F is calculated, and then the attention map is converted into attention weights using the sofimax function, and finally the current single frame F and F are fused using the attention weights.

[0024]

[0025] F i l =Concat(F i l1 , F i l2 , …, F i lt , …, F i lT )

[0026] wherein: represents the attention map between the current single frame F and F , F i lt represents the feature after fusion of the current single frame F and F , W L is a learnable matrix shared in each layer of the encoder, Concat represents the concatenation operation in the time dimension, and P represents the output of the local prompt module in the i-th layer of the encoder.

[0027] (22) When determining the boundary, the human eye usually pays attention to low-level features such as color and texture, which are essential attributes of the image and are irrelevant to time, so the global prompt vector can be shared in each layer of the encoder, so that all single frames use this set of universal global prompts; the shadow boundary is identified through the global prompt module, which specifically includes the following steps:

[0028] (221) The original feature D i is subjected to spatial average pooling and temporal average pooling operations respectively to obtain spatial global feature P and temporal global feature P

[0029] P i spatial =AvgPool hw (D i )

[0030] P i temporal ​=AvgPool t (D i )

[0031] Where: AvgPool hw Represents the average pooling operation along the spatial dimension, which changes the spatial dimension from Compression is 1; AvgPool t represents the average pooling operation along the time dimension, compressing the time dimension from T to 1;

[0032] (222) Randomly initialize a set of N g , a global hint of length C, forming a global hint vector First, the global prompt vector G and the spatial global feature P are transformed in the time dimension. i spatial Perform cross attention operation, due to the spatial global feature P i spatial The spatial dimension of is converted from The global hint vector G is compressed to 1, so that the global information of each single frame in space can be learned; then the channel is adjusted by the multi-layer perceptron MLP; and then the spatial dimension is continued with the temporal global feature P i temporal Perform cross-attention operations to learn the global information of each position in time; then perform channel adjustment through the multi-layer perceptron MLP, and finally output the enhanced global prompt vector

[0033]

[0034] Among them: TemporalAttn and SpatialAttn represent the cross attention mechanism in the time dimension and the cross attention mechanism in the spatial dimension respectively, the time dimension is T and the spatial dimension is

[0035] (223) Using the same method as (213), using the global hint vector To adjust the original feature D i , and get the output F of the global prompt module in the i-th layer encoder i g :

[0036]

[0037] F i g =Concat(F i g1 , F i g2 ,…,Fi gt ,…F i gT )

[0038] in: Indicates the current single frame and The attention map between i gt Indicates the current single frame and After fusion, represents the output of the global cue module in the i-th layer encoder;

[0039] (23) To F i l and F i g Perform dynamic weighting to refine the original feature D i , get the fusion enhanced feature F of the i-th layer encoder i a :

[0040] F i a =α×F i l +β×F i g +D i

[0041] Among them: α and β are learnable weights used to dynamically adjust the two hint vectors to the original feature D i the impact of; represents the fusion enhanced features of the i-th layer encoder, F i a ={F i a1 , F i a2 ,…,F i at ,…F i aT}.

[0042] Specifically, in step (3), all layers of the encoder are evenly divided into four consecutive stages, and the fusion enhancement features of the last layer of each stage are selected and merged into the encoding features The steps include:

[0043] (31) The encoder The N-layer fusion enhancement features are extracted, and each extracted fusion enhancement feature is used to reduce the number of channels from C to C using a fully connected layer. e , thus obtaining four intermediate features;

[0044] (32)Concatenate the four intermediate features together in the channel dimension, and the number of channels after concatenation is 4C e Then use a 1x1 convolutional layer to reduce the number of channels from 4C e to C, and obtain the encoded feature

[0045] Specifically, in step (4), in order to reduce the background noise of the current single frame by using the historical single frame and establish the connection between different single frames, a prompt-based memory module is designed, which includes a key-value encoder and a memory reading module. The memory module uses the local prompt vector of the historical single frame to establish a memory bank, and uses the memory bank for shadow detection of the current single frame. Specifically, the following steps are included:

[0046] (41) In the key-value encoder, the local prompt sub-vector and the single frame feature are encoded to generate the key-value pair of the current single frame , and then the key-value pair of the current single frame is updated to the memory bank.

[0047] (411) For the single frame feature , first use a 1x1 convolutional layer to reduce the number of channels from C to C m , and then use a convolutional gating unit to extract the value area; the convolutional gating unit includes two processing paths. One processing path selects the value area in the single frame feature by using a fully connected layer and a depth separable convolution; the other processing path aligns the features by using a fully connected layer; then the results of the two processing paths are multiplied element by element, and finally the single frame feature is encoded into a value V t and stored in the memory bank; specifically, the following steps are included:

[0048]

[0049] wherein: ConvGLU represents the convolutional gating unit, Conv 1×1 represents the 1x1 convolutional layer, and V t represents the value of the current single frame obtained after encoding the single frame feature ;

[0050] (412) For the local prompt sub-vector, the local prompt sub-vector corresponding to the last layer of the encoder of the single frame feature is aligned by using a fully connected layer, and the local prompt sub-vector is obtained. Encoding as key K t , the memory bank stores the key-value pairs of the previous T max frame history single frame, as follows:

[0051]

[0052] Where: Linear represents linear layer alignment operation; K t represents the key of the current single frame , V t represents the value of the current single frame , M t refers to the memory bank when processing the next single frame .

[0053] (42) When reading the memory bank, first use a fully connected layer to align the local prompter vector of the next single frame Encode the local prompter vector into a query vector, then match the query vector with the memory bank to generate enhanced memory features H t+1 , expressed as:

[0054]

[0055] H t+1 = softmax(Q t+1 ×K T )×V

[0056] Where: Q t+1 represents the query vector obtained by aligning the local prompter vector through a linear layer; K and V refer to all keys and all values in M t ; H t+1 represents the enhanced memory features when processing the single frame .

[0057] (43) First, pass H t+1 to a fully connected layer to fuse the N l matching results into a single result, then use a fully connected layer to restore the channel number from C m to C, perform element-wise addition on H t+1 and , and finally obtain the memory enhanced features Z t .

[0058] Specifically, in step (5), the shadow mask Y t of the current frame single frame is generated by the lightweight decoder, and the memory enhanced features Z tThe resolution of the memory-enhanced feature Z is too small to express some details of the picture, so a progressive up-sampling method is adopted in the decoder to gradually increase the resolution of the memory-enhanced feature Z t from to Specifically, the number of channels is first reduced from C to C' through a multi-layer perception (MLP) to reduce the amount of calculation, and then the resolution is doubled through a convolution operation and an up-sampling operation. After repeating twice, a shadow mask Y is output through a 1x1 convolution layer t .

[0059] The video shadow detection device based on the dynamic prompt memory network comprises a pre-trained visual base model DINOv2, a dynamic prompt module, a prompt-based memory module and a lightweight decoder.

[0060] The visual base model DINOv2 comprises an N-layer encoder, and an original feature of each layer of the encoder is obtained by performing preliminary encoding on an input image through the encoder. During a training stage, all parameters of the encoder are frozen.

[0061] The dynamic prompt module randomly generates a global prompt vector and a local prompt vector for the original feature output by the visual base model DINOv2, and performs interactive fusion of the original feature and the global prompt vector and the local prompt vector to obtain two enhanced features. The two enhanced features are weighted and summed to generate a fusion enhanced feature of each layer of the encoder.

[0062] The global prompt module is configured to generate the global prompt vector, and the global prompt vector identifies a shadow boundary in an image based on color and texture information to obtain a global prompt enhanced feature.

[0063] The local prompt module is configured to generate the local prompt vector, and the local prompt vector identifies a shadow position in the image based on semantic information to obtain a local prompt enhanced feature.

[0064] The memory module performs segmented fusion on the fusion enhanced feature of each layer of the encoder to obtain an encoding feature, and divides the encoding feature into T single-frame features in a time dimension. The single-frame feature is matched and interacted with a key-value pair stored in a memory bank through a memory reading module, so that the single-frame feature obtains semantic information of a historical single-frame to generate a memory-enhanced feature. A key-value encoder is used to encode the single-frame feature to generate a corresponding key-value pair, and the generated key-value pair is stored in the memory bank. The key-value pairs of the historical single-frames are selectively eliminated according to a first-in-first-out rule.

[0065] The decoder gradually increases the resolution of the memory-enhanced feature, and then decodes the memory-enhanced feature into a shadow mask.

[0066] Beneficial effects: The video shadow detection method and device based on the dynamic prompt memory network (DPMNet) provided by the present invention have the following advantages over the existing technology: 1. The network architecture used in the present invention takes into account the temporal correlation within and between segments at the same time, which can ensure effective interaction within the segment and long-term consistency of features between segments; 2. The dynamic prompt module in the present invention detects blurred shadows by converting projected shadow instances into shadow features, thereby improving the performance of blurred shadow detection in complex scenes; 3. The prompt-based memory module in the present invention can alleviate the error accumulation problem caused by encoding long-term features. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Schematic diagram of the framework structure of the dynamic prompt memory network (DPMNet) in the method of the present invention;

[0068] Figure 2 Flow chart for the implementation of the method of the present invention;

[0069] Figure 3 This is the structural block diagram of the dynamic prompt module and the prompt-based memory module. DETAILED DESCRIPTION

[0070] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] The present invention provides a video shadow detection method based on dynamic prompt memory network (DPMNet), which can aggregate time information without accumulating errors by processing the obfuscated shadows with the help of objects casting shadows. Figure 1 Figure 1 shows the architecture of the Dynamic Prompt Memory Network (DPMNet), which consists of two core modules: the Dynamic Prompt Module (DPM) and the Prompt-Based Memory Module (PMM). Video frames are initially encoded by the DINOv2 encoder and then fed into the DPM. When direct shadow encoding is difficult, the DPM uses local and global cues to convert the semantic information provided by DINOv2 into shadow masks. Local cues focus on semantic transformation, identifying shadow locations based on semantic information, while global cues identify shadow boundaries based on color and texture information. Features processed by the DPM are fed into the PMM for memory matching and updating. The PMM addresses deformation and long-term temporal consistency while preventing error accumulation. The PMM uses local cues as a proxy for temporal matching, reducing memory usage and minimizing reliance on historical shadow masks.

[0072] like Figure 2 FIG. 1 is a flowchart of the implementation of the present invention, and each step is described in detail below.

[0073] Step S01: The video frame is sent into the encoder of DINOv2 for preliminary encoding by segment.

[0074] Given a video segment The video segment I is input into each layer encoder of the pre-trained visual base model DINOv2 to obtain a group of original features D = {D1, D2, …D i , …, D N}, N = 24 represents the total number of layers of the encoder, and the i-th layer encoder outputs the i-th layer original feature The original feature D i is divided into T single frames in the time dimension D i represents the current single frame of the original feature D i , H and W represent the height and width of the original feature D i , and C represents the number of channels.

[0075] In order to maximize the generalization ability of the visual base model DINOv2, in the training stage, the encoder of the visual base model DINOv2 is frozen, the gradient information is not recorded, and the parameters of the encoder of the visual base model DINOv2 are not updated.

[0076] Step S02: The original feature is interactively fused with two groups of prompt vectors randomly initialized in advance to generate two enhanced features, and the two enhanced features are superimposed on the original feature.

[0077] Randomly initialize a local prompt vector group L = {L1, L2, …L i , …, L N} and a global prompt vector G, the local prompt vector L identifies the shadow position based on semantic information, and the global prompt vector G identifies the shadow boundary based on color and texture information, the original feature D i is respectively interactively fused with the local prompt vector L i and the global prompt vector G to obtain two enhanced features, and the two enhanced features are weighted and summed to generate the fusion enhanced feature of the i-th layer encoder.

[0078] A dynamic prompt module (DPM) is designed to interactively fuse each layer of original features with two kinds of prompt vectors, so that the two kinds of prompt vectors can fully adjust the original features, and the original features can be converted into fusion enhanced features for video shadow detection; the DPM includes a local prompt module (LPB) and a global prompt module (GPB), and their specific framework is shown in Figure 3 Based on the framework shown in Figure 3 , the acquisition process of the fusion enhanced feature is described below.

[0079] Step 21, identify the shadow position through the LPB, which specifically includes the following steps.

[0080] Step 211, randomly initialize a set of N local cues for each layer encoder by LPB l , the local cue vector of the i-th layer is denoted as At the same time, the local cue vector L i is copied into T parts in the time dimension, and the t-th part of the local cue vector is denoted as as the local cue sub-vector corresponding to the current single frame and the single frame feature .

[0081] Step 212, perform cross-attention operation on the local cue sub-vector and the current single frame , so that the local cue sub-vector selectively aggregates effective information from the current single frame ; in order to introduce time clues to fuse intra-segment semantic information, all the aggregated effective information in each layer encoder also needs to be spliced together for self-attention operation to fuse local cue vectors from different single frames:

[0082]

[0083] Wherein: CrossAttn and SelfAttn represent cross-attention mechanism and self-attention mechanism respectively, and MLP represents multi-layer perception, and the output of the multi-layer perception MLP is The multi-layer perception MLP includes two fully connected layers, one of which is used to reduce the number of channels of the local cue vector from C to C', so as to reduce the amount of calculation, and the other is used to restore the number of channels of the local cue vector from C' to C, so that the local cue extracts more semantic information.

[0084] Step 213, in a manner similar to the attention mechanism, let the local cue vector dynamically adjust the features of the original feature D i according to different positions of the image, so that the focus of the current single frame moves from the instance object to the shadow area; specifically, first calculate the attention map of the current single frame , then use the softmax function to convert the attention map into attention weights, and finally use the attention weights to fuse the current single frame and

[0085]

[0086] F i l =Concat(F il1 , F i l2 , …, F i lt , …, F i lT )

[0087] wherein: denotes the attention map between the current single frame and , F i lt denotes the feature fused from the current single frame and , is a learnable matrix, W L is shared among layers of the encoder, and Concat denotes the concatenation operation in the time dimension, denotes the output of the local cue module in the i-th layer of the encoder.

[0088] Step 22, the human eye usually focuses on low-level features such as color and texture when determining the boundary, and these low-level features are essential attributes of the image and are irrelevant to time, so the global cue vector can be shared among layers of the encoder, so that all single frames use this set of universal global cues; the shadow boundary is identified through the global cue module, which includes the following steps.

[0089] Step 221, respectively perform spatial average pooling and temporal average pooling on the original feature D i to obtain spatial global feature and temporal global feature

[0090] P i spatial = AvgPool hw (D i )

[0091] P i temporal = AvgPool t (D i )

[0092] wherein: AvgPool hw denotes the average pooling operation along the spatial dimension, which compresses the spatial dimension from to 1; AvgPool t denotes the average pooling operation along the time dimension, which compresses the time dimension from T to 1.

[0093] Step 222, randomly initialize a set of global cues with a number of N g and a length of C to form a global cue vector First, the global prompt vector G and the spatial global feature P are transformed in the time dimension. i spatial Perform cross attention operation, due to the spatial global feature P i spatial The spatial dimension of is converted from The global hint vector G is compressed to 1, so that the global information of each single frame in space can be learned; then the channel is adjusted by the multi-layer perceptron MLP; and then the spatial dimension is continued with the temporal global feature P i temporal Perform cross-attention operations to learn the global information of each position in time; then perform channel adjustment through the multi-layer perceptron MLP, and finally output the enhanced global prompt vector

[0094]

[0095] Among them: TemporalAttn and SpatialAttn represent the cross attention mechanism in the time dimension and the cross attention mechanism in the spatial dimension respectively, the time dimension is T and the spatial dimension is

[0096] Step 223: Using the same method as in step 213, using the global hint vector To adjust the original feature D i , and get the output F of the global prompt module in the i-th layer encoder i g :

[0097]

[0098] F i g =Concat(F i g1 , F i g2 ,…,F i gt ,…F i gT )

[0099] in: Indicates the current single frame and The attention map between i gt Indicates the current single frame and After fusion, represents the output of the global hint module in the i-th layer encoder.

[0100] Step 23, F i l and F i g Perform dynamic weighting to refine the original feature D i , get the fusion enhanced feature F of the i-th layer encoder i a :

[0101] F i a =α×F i l +β×F i g +D i

[0102] Among them: α and β are learnable weights used to dynamically adjust the two hint vectors to the original feature D i the impact of; represents the fusion enhanced features of the i-th layer encoder, F i a ={F i a1 , F i a2 ,…,F i at ,…F i aT}.

[0103] Step S03: Calculate coding features.

[0104] Repeat step 02 for the original features of each layer to obtain the fused enhanced features of each layer. Perform segmented fusion on the fused enhanced features of each layer to obtain the encoded features. Specifically, all layers of the encoder are evenly divided into four consecutive stages, and then the fusion enhancement features of the last layer of each stage are selected and finally merged into the encoding features

[0105] (31) The encoder The N-layer fusion enhancement features are extracted, and each extracted fusion enhancement feature is used to reduce the number of channels from C to C using a fully connected layer. e , thus obtaining four intermediate features;

[0106] (32) The four intermediate features are spliced ​​together in the channel dimension, and the number of channels after splicing is 4C e , and then use a 1×1 convolution layer to reduce the number of channels from 4C e Restore to C and get the encoding feature

[0107] Step S04: The encoded features are divided into single-frame features in the time dimension, and the memory reading module and the memory library are used for matching and interaction to enhance themselves and update and eliminate information in the memory library.

[0108] To reduce the background noise of the current single-frame by using the historical single-frame, and to establish the connection between different single-frames, a prompt-based PMM is designed, which includes a key-value encoder and a memory reading module; the PMM uses the local prompt vector of the historical single-frame to establish a memory library, and uses the memory library for shadow detection of the current single-frame; the structural block diagram of the PMM is as shown in Figure 3 , which specifically includes the following steps.

[0109] Step 41, in the key-value encoder, the local prompt sub-vector and the single-frame feature are encoded to generate the key-value pair of the current single-frame , and then the key-value pair of the current single-frame is updated to the memory library.

[0110] Step 411, the encoded feature is divided into T single-frame features in the time dimension.

[0111] Step 412, for the single-frame feature , first use a 1x1 convolution layer to reduce the number of channels from C to C m , and then extract the value area through a convolution gate unit; the convolution gate unit includes two processing, one of which selects the value area in the single-frame feature through a fully connected layer and a depth separable convolution; the other uses a fully connected layer to align the features; then the results of the two processing are element-wise multiplied, and finally the single-frame feature is encoded into a value V t and stored in the memory library; specifically including the following steps:

[0112]

[0113] Wherein: ConvGLU represents the convolution gate unit, Cony 1×1 represents the 1x1 convolution layer, and V t represents the value of the current single-frame D i t obtained after encoding the single-frame feature .

[0114] Step 413, for the local prompt sub-vector, the local prompt sub-vector of the last layer encoder corresponding to the single-frame feature is selected. Align local cue subvectors using a fully connected layer The local hint subvector Encoded as key K t , when processing the current single frame When the memory bank stores the previous T max The key-value pairs of a single frame in the frame history are as follows:

[0115]

[0116] Among them: Linear represents the linear layer alignment operation; K t Indicates the current single frame Key, V t Indicates the current single frame The value of M t Refers to processing the next single frame Memory bank of time.

[0117] Step 42: When reading the memory library, first use a fully connected layer to align the next single frame The local hint subvector of The local hint subvector Encoded into a query vector, the query vector is then matched with the memory library to generate enhanced memory features H t+1 , expressed as:

[0118]

[0119] H t+1 =softmax(Q t+1 ×K T )×V

[0120] Where: Q t+1 Represents the local hint subvector The query vector obtained after alignment through the linear layer; K and V refer to M t All the keys and all the values ​​in ; Indicates processing of a single frame Enhanced memory characteristics.

[0121] Step 43, first H t+1 Pass it to a fully connected layer to convert this N l The matching results are fused into a single result, and then a fully connected layer is used to reduce the number of channels from C m Restore to C, H t+1 and Add element by element and finally get the memory enhancement feature Z t .

[0122] Step 44: Set the current single frame The key-value pairs of the memory bank are stored in the memory bank, and the key-value pairs of the historical single frame are selectively eliminated according to the first-in first-out rule.

[0123] Step S05: The resolution of the memory-enhanced feature is gradually improved and the memory-enhanced feature is decoded into a shadow mask through a lightweight decoder.

[0124] The shadow mask Y of the current frame single frame is generated through the lightweight decoder . t Since the resolution of the memory-enhanced feature Z t is too small to express some details of the picture, the progressive up-sampling method is used in the decoder to gradually adjust the resolution of the memory-enhanced feature Z t from to . Specifically, the number of channels is reduced from C to C' through a multi-layer perception (MLP) to reduce the calculation amount, and then the resolution is doubled through convolution operation and up-sampling operation. After repeating twice, the shadow mask Y t is output through a 1x1 convolution layer.

[0125] A video shadow detection device based on a dynamic prompt memory network is used to implement the above-mentioned video shadow detection method, mainly including a pre-trained visual base model DINOv2, a dynamic prompt module (DPM), a prompt-based memory module (PMM), and a lightweight decoder. The DPM mainly includes a global prompt module (GPB) and a local prompt module (LPB).

[0126] The visual base model DINOv2 includes an N-layer encoder. The input image is preliminarily encoded through the encoder to obtain a set of original features D = {D1, D2, …D i , …, D N} of each layer encoder; during the training stage, the parameters of the encoder are frozen.

[0127] The DPM randomly generates a global prompt vector G and a local prompt vector L = {L1, L2, …L i , …, L N} for the original features D output by the visual base model DINOv2, and interacts and fuses the original features D i with the global prompt vector G and the local prompt vector L i to obtain two enhanced features F i and F g . i l The two enhanced features are weighted and summed to generate the fusion enhanced features F i of each layer encoder. a

[0128] GPB, for generating a global prompt vector, the global prompt vector identifying shadow boundaries in the image based on color and texture information, obtaining a global prompt enhanced feature F i g ;

[0129] LPB, for generating a local prompt vector, the local prompt vector identifying shadow positions in the image based on semantic information, obtaining a local prompt enhanced feature F i l ;

[0130] Memory module, segmenting and fusing the fused enhanced features of each layer of encoder to obtain an encoded feature Segmenting the encoded feature into T single-frame features in the time dimension Through the memory reading module, the single-frame feature is matched and interacted with the key-value pairs stored in the memory bank, so that the single-frame feature obtains the semantic information of the historical single-frame, and generates a memory enhanced feature Z t ; using a key-value encoder to encode the single-frame feature , generating a corresponding key-value pair, storing the generated key-value pair in the memory bank, and selectively eliminating the key-value pairs of the historical single-frame according to the first-in first-out rule;

[0131] Lightweight decoder, first gradually improving the resolution of the memory enhanced feature Z t , and then decoding the memory enhanced feature Z t into a shadow mask Y t .

[0132] The basic principles, main features and advantages of the present application are shown and described above. Those skilled in the art should understand that the above examples do not limit the present application in any form, and any technical solutions obtained by equivalent replacement or equivalent transformation fall within the protection scope of the present application.

Claims

1. A video shadow detection method based on a dynamic cue memory network, characterized by: The proposed method comprises a dynamic hint module and a hint-based memory module. The dynamic hint module uses local and global hints to convert the semantic information provided by the visual base model DINOv2 into a shadow mask. The local hint focuses on semantic transformation and identifies the shadow position based on semantic information. The global hint identifies the shadow boundary based on color and texture information. The memory module is used to solve the problems of deformation and long-term temporal consistency while preventing error accumulation. The memory module uses local hints as a proxy for temporal matching, reducing memory usage and minimizing dependence on historical shadow masks. The method comprises the following steps: (1) Given a video clip I, the video clip I is input into each layer encoder of the pre-trained visual basic model DINOv2 to obtain the original feature group D = {D1, D2, ... D i ,…,D N }, the i-th layer encoder outputs the i-th layer original feature D i , the total number of layers of the encoder is N; (2) Randomly initialize the local hint vector group L = {L1, L2, ... L i ,…,L N } and the global hint vector G,L i is the local prompt vector of the i-th layer; the original feature D i Respectively with the local hint vector L i Interactively fuse with the global hint vector G to obtain two enhanced features, perform weighted summation on the two enhanced features to generate the fused enhanced features of the i-th layer encoder; (3) Repeat step (2) for the original features of each layer to obtain the fused enhanced features of each layer; The fusion enhancement features of each layer are segmented and fused to obtain the encoding features (4) Encoding features Split into T single-frame features according to the time dimension At the same time, the original feature D i Split into T single frames according to the time dimension The local hint vector L i Copy it into T parts according to the time dimension, and make the tth single frame As the original feature D i The current single frame, the tth local hint vector As the corresponding current single frame and single frame features The local hint subvector of ; Through the memory reading module, the single frame features Match and interact with key-value pairs stored in the memory bank to make single-frame features Obtain the semantic information of a historical single frame and generate memory enhancement feature Z t ;Select single frame features The corresponding local hint subvector of the last layer encoder Use key-value encoder to local hint sub-vector and single-frame features Encode and generate the current single frame The key-value pair of the current single frame The key-value pairs of the historical frames are stored in the memory bank, and the key-value pairs of the historical frames are selectively eliminated according to the first-in-first-out rule. In order to use the historical frames to reduce the background noise of the current frame and establish the connection between different frames, a prompt-based memory module is designed. The memory module includes two parts: a key-value encoder and a memory reading module. The memory module uses the local prompt vectors of the historical frames to build a memory bank, and the memory bank is used for shadow detection in the current frame. The specific steps include the following: (41) In the key-value encoder, the local hint subvector and single-frame features Encode and generate the current single frame The key-value pair, then the current single frame The key-value pairs are updated into the memory; (411) For single frame features First, a 1×1 convolutional layer is used to reduce the number of channels from C to C m , and then extract the value region through the convolutional gating unit; the convolutional gating unit includes two-way processing, one of which is to select single-frame features through a fully connected layer and a depth-separable convolution The other processing uses a fully connected layer to align the features; then the results of the two processing are multiplied element by element, and finally the single frame feature F t Encoded as value V t Store in the memory; specifically includes the following steps: Among them: ConvGLU represents convolutional gate unit, Conv 1×1 represents a 1×1 convolutional layer, V t Represents the single frame feature The current single frame obtained after encoding The value of (412) For the local hint sub-vector, select the single frame feature The corresponding local hint subvector of the last layer encoder Align local cue subvectors using a fully connected layer The local hint subvector Encoded as key K t , when processing the current single frame When the memory bank stores the previous T max The key-value pairs of a single frame in the frame history are as follows: Among them: Linear represents the linear layer alignment operation; K t Indicates the current single frame Key, V t Indicates the current single frame The value of M t Refers to processing the next single frame Memory bank of time; (42) When reading the memory library, first use a fully connected layer to align the next single frame The local hint subvector of The local hint subvector Encoded into a query vector, the query vector is then matched with the memory library to generate enhanced memory features H t+1 , expressed as: H t+1 =softmax(Q t+1 ×K T )×V Where: Q t+1 Represents the local hint subvector The query vector obtained after alignment through the linear layer; K and V refer to M t All the keys and all the values ​​in ; Indicates processing of a single frame Enhanced memory characteristics when (43) First, H t+1 Pass it to a fully connected layer to convert this N l The matching results are fused into a single result, and then a fully connected layer is used to reduce the number of channels from C m Restore to C, H t+1 and Add element by element and finally get the memory enhancement feature Z t ; (5) Through a decoder, first gradually improve the memory enhancement feature Z t The resolution of the memory enhancement feature Z t Decoded into shadow mask Y t .

2. The video shadow detection method based on dynamic cue memory network according to claim 1, characterized in that: In step (1), a video clip is given The video clip I is input into each layer encoder of the pre-trained visual basis model DINOv2, and the i-th layer encoder outputs the i-th layer original feature Represents the original feature D i The current single frame, H and W represent the original features D i The height and width of , C represents the number of channels; during the training phase, the encoder of the visual basis model DINOv2 is frozen, the gradient information is not recorded, and the parameters of the visual basis model DINOv2 encoder are not updated.

3. The video shadow detection method based on dynamic cue memory network according to claim 1, characterized in that: In step (2), the global hint vector G identifies the shadow boundary based on color and texture information, and the local hint vector L identifies the shadow position based on semantic information; the dynamic hint module allows the original features of each layer to be interactively fused with the two hint vectors respectively, so that the two hint vectors fully adjust the original features and convert the original features into fused enhanced features for video shadow detection; The dynamic prompt module includes a local prompt module and a global prompt module; the acquisition process of the fusion enhancement feature includes the following steps: (21) Identifying the shadow position through the local prompt module, specifically including the following steps: (211) A set of N numbers is randomly initialized for each layer of encoder through the local hint module l , a local hint of length C, the local hint vector of the i-th layer is recorded as At the same time, the local hint vector L i Copy it into T parts according to the time dimension, and take the tth local prompt vector As the corresponding current single frame and single frame features The local hint subvector of ; (212) for the local hint subvector and the current single frame Perform cross attention operation to make the local hint sub-vector From the current single frame Selectively aggregate valid information in the CNN; combine all the aggregated valid information in each layer of the encoder to perform self-attention operations, and fuse the local prompt vectors from different single frames: Among them: CrossAttn and SelfAttn represent the cross attention mechanism and self-attention mechanism respectively, MLP represents the multi-layer perceptron, and the output of the multi-layer perceptron MLP is The multi-layer perceptron MLP includes two fully connected layers, one fully connected layer is used to reduce the number of channels of the local prompt vector from C to C', and the other fully connected layer is used to restore the number of channels of the local prompt vector from C' to C; (213) First calculate With the current single frame The attention map is then converted into attention weights using the softmax function, and finally the attention weights are used to fuse the current single frame. and F i l =Concat(F i l1 ,F i l2 ,…,F i lt ,…,F i lT ) in: Indicates the current single frame and The attention map between i lt Indicates the current single frame and After fusion, is a learnable matrix, W L Shared in each layer of the encoder, Concat represents the splicing operation in the time dimension, represents the output of the local prompt module in the i-th layer encoder; (22) Identifying shadow boundaries through the global hint module, specifically including the following steps: (221) For the original feature D i Perform spatial average pooling and temporal average pooling operations respectively to obtain spatial global features and temporal global features P i spatial =AvgPool hw (D i ) P i temporal =AvgPool t (D i ) Where: AvgPool hw Represents the average pooling operation along the spatial dimension, which changes the spatial dimension from Compression is 1; AvgPool t represents the average pooling operation along the time dimension, compressing the time dimension from T to 1; (222) Randomly initialize a set of N g , a global hint of length C, forming a global hint vector First, the global prompt vector G and the spatial global feature P are transformed in the time dimension. i spatial Perform cross attention operation; then adjust the channel through the multi-layer perceptron MLP; then continue with the temporal global feature P in the spatial dimension i temporal Perform cross attention operation; then perform channel adjustment through multi-layer perceptron MLP, and finally output enhanced global prompt vector Among them: TemporalAttn and SpatialAttn represent the cross attention mechanism in the time dimension and the cross attention mechanism in the spatial dimension respectively, the time dimension is T and the spatial dimension is (223) Using the same method as (213), using the global hint vector To adjust the original feature D i , and get the output F of the global prompt module in the i-th layer encoder i g : F i g =Concat(F i g1 ,F i g2 ,…,F i gt ,…F i gT ) in: Indicates the current single frame and The attention map between i gt Indicates the current single frame and After fusion, represents the output of the global cue module in the i-th layer encoder; (23) To F i l and F i g Perform dynamic weighting to obtain the fusion enhanced feature F of the i-th layer encoder i a : F i a =α×F i l +β×F i g +D i Among them: α and β are learnable weights used to dynamically adjust the two hint vectors to the original feature D i the impact of; represents the fusion enhanced features of the i-th layer encoder, F i a ={F i a1 ,F i a2 ,…,F i at ,…F i aT }.

4. The video shadow detection method based on dynamic cue memory network according to claim 1, characterized in that: In step (3), all layers of the encoder are evenly divided into four consecutive stages, and the fusion enhancement features of the last layer of each stage are selected and merged into the encoding features The steps include: (31) The encoder The N-layer fusion enhancement features are extracted, and each extracted fusion enhancement feature is used to reduce the number of channels from C to C using a fully connected layer. e , thus obtaining four intermediate features; (32) The four intermediate features are spliced ​​together in the channel dimension, and the number of channels after splicing is 4C e , and then use a 1×1 convolution layer to reduce the number of channels from 4C e Restore to C and get the encoding feature 5. The video shadow detection method based on dynamic cue memory network according to claim 1, characterized in that: In the step (5), the current frame is generated by the decoder Shadow mask Y t , using the progressive upsampling method, gradually enhance the memory feature Z t The resolution from Adjust to Specifically, the number of channels is first reduced from C to C' through a multi-layer perceptron (MLP), and then the resolution is doubled by using convolution and upsampling operations. After repeating twice, the shadow mask Y is output through a 1×1 convolution layer. t .

6. A video shadow detection device based on a dynamic cue memory network, characterized by: It includes a pre-trained visual basic model DINOv2, a dynamic prompt module, a prompt-based memory module and a decoder, wherein the dynamic prompt module includes a global prompt module and a local prompt module; The visual basic model DINOv2 includes N layers of encoders, which preliminarily encode the input image to obtain the original features output by each layer of encoders; During the training phase, all encoder parameters are frozen; The dynamic prompt module randomly generates a global prompt vector and a local prompt vector for the original features output by the visual basic model DINOv2, interactively fuses the original features with the global prompt vector and the local prompt vector respectively to obtain two enhanced features, and performs a weighted summation on the two enhanced features to generate the fused enhanced features of each layer encoder; The global hint module is used to generate a global hint vector, which identifies shadow boundaries based on color and texture information to obtain global hint enhancement features; The local hint module is used to generate a local hint vector, which identifies the shadow position based on semantic information and obtains local hint enhancement features; The memory module performs segmented fusion of the fusion enhancement features of each layer of encoder to obtain coding features, and divides the coding features into T single-frame features according to the time dimension; Through the memory reading module, the single-frame features are matched and interacted with the key-value pairs stored in the memory bank, so that the single-frame features can obtain the semantic information of the historical single frame and generate memory-enhanced features; the single-frame features are encoded using the key-value encoder to generate the corresponding key-value pairs, and the generated key-value pairs are stored in the memory bank. The key-value pairs of the historical single frame are selectively eliminated according to the first-in-first-out rule; The decoder first gradually improves the resolution of the memory enhancement feature and then decodes the memory enhancement feature into a shadow mask.

Images