Video shadow detection method and device based on dynamic prompt memory network

Abstract

The invention discloses a video shadow detection method and device based on a dynamic prompt memory network, and the method comprises the steps: processing confused shadows through an object which projects the shadows, and polymerizing time information under the condition of not accumulating errors; the method comprises a dynamic prompt module and a memory module based on prompt. The dynamic prompt module converts semantic information provided by the visual basic model DINOv2 into a shadow mask when direct shadow coding is difficult by using local prompt and global prompt, the local prompt focuses on semantic conversion and identifies a shadow position based on the semantic information, and the global prompt identifies a shadow boundary based on color and texture information; the memory module is used for solving the problems of deformation and long-term time consistency, meanwhile, error accumulation is prevented, the memory module uses local prompt as an agent of time matching, memory use is reduced, and dependence on historical shadow masks is minimized. The method aims at the problem that the fuzzy shadow in the complex scene is difficult to detect, the detection performance in the complex scene can be improved, and meanwhile the time consistency of the result during long-term detection is kept.

Description

Technical Field

[0001] The invention relates to a video shadow detection method and device based on a dynamic prompt memory network, and the dynamic prompt memory network is used for video shadow detection, belonging to the video shadow detection technology. Background Art

[0002] Shadows are crucial for inferring scene geometry and lighting. Accurate shadow detection enhances downstream visual tasks such as segmentation, tracking, and scene reconstruction. Most existing work focuses on image shadow detection (ISD). With the continuous development of new deep learning techniques, shadow detection algorithms for images have achieved good results. In recent years, researchers have tried to extend the algorithm to video shadow detection (VSD). However, shadows in videos are often confused with the background and can deform rapidly between video frames, which makes video shadow detection a more challenging task than image shadow detection.

[0003] Early networks focused on the correspondence between adjacent frames. They fused the information of adjacent frames using attention mechanisms or convolutional layers to encode the temporal information between frames. These algorithms achieved good results in simple scenes. With the development of Transformer and diffusion models, VSD methods began to adopt the latest architectures to improve performance. By introducing new attention mechanisms and boundary information, researchers have made some contributions to temporal modeling and deformation problems.

[0004] However, there are still two major problems in the VSD algorithm that need to be further solved.

[0005] First, existing methods often have difficulty fully capturing shadow areas when the camera moves quickly or the light changes quickly. This is because they only focus on potential dark areas, they are only trained on shadow datasets, and lack prior knowledge about shadow casting instances. We believe that directly extracting features may be difficult for the network when shadows are cluttered. Therefore, it is important to use the information of shadow casting instances to help identify their shadows.

[0006] Second, existing methods do not fully utilize the temporal information in videos. Among VSD methods, there are three architectures that can fuse multi-frame features: multi-branch networks, fragment-based networks, and memory networks. Multi-branch networks can enhance the information of adjacent frames, but because they focus on short-term features, they have difficulty capturing fast motion and deformation. Fragment-based networks input multiple frames as a fragment and segment them through intra-fragment information, but they lack temporal consistency between fragments and cannot use past features to further eliminate noise. Although memory networks can encode long-term temporal features by saving masks, they rely heavily on the results of the previous frame and can easily accumulate errors. Summary of the invention

[0007] Purpose of the invention: In order to solve the problems of inaccurate detection and insufficient utilization of temporal information in complex scenes, the present invention proposes a video shadow detection method and device based on a dynamic prompt memory network (DPMNet), which can simultaneously capture the framework of intra-fragment and long-term information; the dynamic prompt memory network in the present invention introduces temporal modeling for the visual base model DINOv2, and is able to propagate its feature representation during long-term processing.

[0008] Technical solution: To achieve the above purpose, the technical solution adopted by the present invention is:

[0009] A video shadow detection method based on a dynamic cue memory network includes two parts: a dynamic cue module (DPM) and a cue-based memory module (PMM). The dynamic cue module uses local cues and global cues to convert semantic information provided by a visual basis model DINOv2 into a shadow mask when it is difficult to directly encode shadows. Local cues focus on semantic conversion and identify shadow positions based on semantic information. Global cues identify shadow boundaries based on color and texture information. The memory module is used to solve deformation and long-term temporal consistency problems while preventing error accumulation. The memory module uses local cues as a proxy for temporal matching to reduce memory usage and minimize dependence on historical shadow masks.

[0010] Specifically, the method comprises the following steps:

[0011] (1) Given a video clip I, the video clip I is input into each layer encoder of the pre-trained visual basis model DINOv2 to obtain the original feature set D = {D1, D2, ... D i , …, D N}, the i-th layer encoder outputs the i-th layer original feature D i , the total number of layers of the encoder is N;

[0012] (2) Randomly initialize the local hint vector set L = {L1, L2, ... L i , …, L N} and the global hint vector G,L i is the local prompt vector of the i-th layer; the original feature D i Respectively with the local hint vector L i Interactively fuse with the global hint vector G to obtain two enhanced features, perform weighted summation on the two enhanced features, and generate the fused enhanced features of the i-th layer encoder;

[0013] (3) Repeat step (2) for the original features of each layer to obtain the fused enhanced features of each layer; perform segmented fusion on the fused enhanced features of each layer to obtain the encoded features

[0014] (4) Encoding features Split into T single-frame features according to the time dimension At the same time, the original feature D i Split into T single frames according to the time dimension The local hint vector L i Duplicate it into T parts in the time dimension, and take the tth single frame As the original feature D i The current single frame, the tth local hint vector As the corresponding current single frame And single frame features The local prompt subvector of Match and interact with key-value pairs stored in the memory bank to make single-frame features Obtain the semantic information of the historical single frame and generate the memory enhancement feature Z i ; Select single frame features The corresponding local hint subvector of the last encoder layer Use key-value encoder to local hint sub-vector and single frame features Encode and generate the current single frame The key-value pair of the current single frame The key-value pairs of the historical single frame are stored in the memory bank, and the key-value pairs of the historical single frame are selectively eliminated according to the first-in-first-out rule;

[0015] (5) Through a lightweight decoder, the memory enhancement feature Z is gradually improved i The resolution of the memory enhancement feature Z t Decoded into shadow mask Y t .

[0016] Specifically, in step (1), a video clip is given The video clip I is input into each layer encoder of the pre-trained visual basis model DINOv2, and the i-th layer encoder outputs the i-th layer original feature Represents the original feature D i The current single frame, H and W represent the original features D i The height and width of , C represents the number of channels; in order to retain the original generalization ability of the visual base model DINOv2 to the maximum extent, during the training phase, the encoder of the visual base model DINOv2 is frozen, the gradient information is not recorded, and the parameters of the encoder of the visual base model DINOv2 are not updated.

[0017] Specifically, in step (2), the global hint vector G identifies the shadow boundary based on color and texture information, and the local hint vector L identifies the shadow position based on semantic information; the dynamic hint module allows the original features of each layer to be interactively fused with the two hint vectors respectively, so that the two hint vectors fully adjust the original features and convert the original features into fused enhanced features for video shadow detection; the dynamic hint module includes a local hint module (LPB) and a global hint module (GPB); the acquisition process of the fused enhanced features includes the following steps:

[0018] (21) Identifying the shadow position through the local prompt module, specifically including the following steps:

[0019] (211) Through the local hint module, a set of N numbers is randomly initialized for each layer of encoder l , a local hint of length C, the local hint vector of the i-th layer is recorded as At the same time, the local hint vector L i Copy it into T copies in the time dimension, and take the tth local hint vector As the corresponding current single frame And single frame features The local hint subvector of ;

[0020] (212) for the local hint subvector and the current single frame Perform a cross-attention operation to make the local hint sub-vector From the current single frame In order to introduce temporal clues to fuse the semantic information in the clip, it is also necessary to piece together all the aggregated effective information in each layer of the encoder for self-attention operation, and fuse the local prompt vectors from different single frames:

[0021]

[0022] Among them: CrossAttn and SelfAttn represent the cross attention mechanism and self-attention mechanism respectively, MLP represents the multi-layer perceptron, and the output of the multi-layer perceptron MLP is The multi-layer perceptron MLP includes two fully connected layers, one fully connected layer is used to reduce the number of channels of the local prompt vector from C to C′ to reduce the amount of calculation, and the other fully connected layer is used to restore the number of channels of the local prompt vector from C′ to C, so that the local prompt can extract more semantic information;

[0023] (213) Using a method similar to the attention mechanism, the local hint vector dynamically adjusts the original feature D according to different positions of the image. i The characteristics of the current single frame The focus of the algorithm is moved from the instance object to the shadow area; specifically, first calculate With the current single frame The attention map is then converted into attention weights using the sofimax function, and finally the attention weights are used to fuse the current single frame. and

[0024]

[0025] F i l =Concat(F i l1 , F i l2 , …, F i lt , …, F i lT )

[0026] in: Indicates the current single frame and The attention map between i lt Indicates the current single frame and After fusion, is a learnable matrix, W L Shared in each layer of the encoder, Concat represents the splicing operation in the time dimension, represents the output of the local cue module in the i-th layer encoder;

[0027] (22) When determining boundaries, the human eye usually focuses on low-level features such as color and texture. These low-level features are essential attributes of the image and are independent of time. Therefore, the global hint vector can be shared among the layers of the encoder so that all single frames use this set of common global hints. The shadow boundary is identified through the global hint module, which specifically includes the following steps:

[0028] (221) For the original feature D i Perform spatial average pooling and temporal average pooling operations respectively to obtain spatial global features and temporal global features

[0029] P i spatial =AvgPool hw (D i )

[0030] P i temporal=AvgPool t (D i )

[0031] Where: AvgPool hw Represents the average pooling operation along the spatial dimension, which converts the spatial dimension from Compression is 1; AvgPool t represents the average pooling operation along the time dimension, compressing the time dimension from T to 1;

[0032] (222) Randomly initialize a set of N g , a global hint of length C, forming a global hint vector First, the global prompt vector G and the spatial global feature P are transformed in the time dimension. i spatial Perform cross attention operation, due to the spatial global feature P i spatial The spatial dimension of is transformed from The global hint vector G is compressed to 1, so that the global information of each single frame in space can be learned; then the multi-layer perceptron MLP is used to adjust the channel; and then the temporal global feature P is used in the spatial dimension. i temporal Perform cross-attention operations to learn the global information of each position in time; then perform channel adjustment through the multi-layer perceptron MLP, and finally output the enhanced global prompt vector

[0033]

[0034] Among them: TemporalAttn and SpatialAttn represent the cross attention mechanism in the time dimension and the cross attention mechanism in the spatial dimension respectively, the time dimension is T and the spatial dimension is

[0035] (223) Using the same method as (213), using the global hint vector To adjust the original feature D i , and obtain the output F of the global prompt module in the i-th layer encoder i g :

[0036]

[0037] F i g =Concat(F i g1 , F i g2 , …, Fi gt , …F i gT )

[0038] in: Indicates the current single frame and The attention map between i gt Indicates the current single frame and After fusion, represents the output of the global cue module in the i-th layer encoder;

[0039] (23) To F i l and F i g Dynamic weighting is performed to refine the original feature D i , get the fusion enhanced feature F of the i-th layer encoder i a :

[0040] F i a =α×F i l +β×F i g +D i

[0041] Among them: α and β are learnable weights used to dynamically adjust the two prompt vectors to the original feature D i The impact of represents the fusion enhanced features of the i-th layer encoder, F i a ={F i a1 , F i a2 , …, F i at , …F i aT}.

[0042] Specifically, in step (3), all layers of the encoder are evenly divided into four consecutive stages, and the fusion enhancement features of the last layer of each stage are selected and merged into the encoding features The steps include:

[0043] (31) The encoder The N-layer fusion enhancement features are extracted, and each fusion enhancement feature is used to reduce the number of channels from C to C using a fully connected layer. e , thus obtaining four intermediate features;

[0044] (32) The four intermediate features are concatenated together in the channel dimension, and the number of channels after concatenation is 4C e , and then use a 1×1 convolutional layer to increase the number of channels from 4C e Restore to C and get the encoding features

[0045] Specifically, in step (4), in order to use the historical single frame to reduce the background noise of the current single frame and establish the connection between different single frames, a prompt-based memory module is designed, and the memory module includes two parts: a key-value encoder and a memory reading module; the memory module uses the local prompt vector of the historical single frame to establish a memory library, and uses the memory library for shadow detection of the current single frame; specifically, the following steps are included:

[0046] (41) In the key-value encoder, the local hint subvector and single frame features Encode and generate the current single frame The key-value pair, and then the current single frame The key-value pairs are updated into the memory;

[0047] (411) For single frame features First, a 1×1 convolutional layer is used to reduce the number of channels from C to C m , and then extract the value region through a convolutional gating unit; the convolutional gating unit includes two-way processing, one of which selects single-frame features through a fully connected layer and a depth-separable convolution The other processing uses a fully connected layer to align the features; then the results of the two processing are multiplied element by element, and finally the single frame features are Encoded as value V t Store into the memory; specifically includes the following steps:

[0048]

[0049] Among them: ConvGLU represents convolutional gated unit, Conv 1×1 represents a 1×1 convolutional layer, V t Represents the single frame feature The current single frame obtained after encoding The value of

[0050] (412) For the local hint sub-vector, select the single frame feature The corresponding local hint subvector of the last encoder layer Align local cue subvectors using a fully connected layer The local hint subvector Encoded as key K t , while processing the current single frame When the memory bank stores the previous T max The key-value pairs of a single frame in the frame history are as follows:

[0051]

[0052] Among them: Linear represents the linear layer alignment operation; K t Indicates the current single frame Key, V t Indicates the current single frame The value of M t Refers to processing the next single frame Memory bank of time;

[0053] (42) When reading the memory library, first use a fully connected layer to align the next single frame The local hint subvector of The local hint subvector Encoded into a query vector, the query vector is then matched with the memory library to generate the enhanced memory feature H t+1 , expressed as:

[0054]

[0055] H t+1 =softmax(Q t+1 ×K T )×V

[0056] Where: Q t+1 Represents the local hint subvector The query vector obtained after alignment through the linear layer; K and V refer to M t All the keys and all the values ​​in; Indicates processing of a single frame Enhanced memory characteristics when

[0057] (43) First, H t+1 Pass it to a fully connected layer to convert this N l The matching results are merged into a single result, and then a fully connected layer is used to reduce the number of channels from C m Restore to C, for H t+1 and Add element by element to finally obtain the memory enhancement feature Z t .

[0058] Specifically, in step (5), a single frame of the current frame is generated by a lightweight decoder. Shadow mask Y t , due to the memory enhancement feature Z tThe resolution is too small to express some details of the picture. Therefore, a progressive upsampling method is required in the decoder to gradually increase the memory enhancement feature Z. t The resolution from Adjust to Specifically, the number of channels is first reduced from C to C′ through a multi-layer perceptron MLP to reduce the amount of calculation, and then the resolution is doubled by using convolution and upsampling operations. After repeating twice, the shadow mask Y is output through a 1×1 convolution layer. t .

[0059] A video shadow detection device based on a dynamic cue memory network, comprising a pre-trained visual basic model DINOv2, a dynamic cue module, a cue-based memory module and a lightweight decoder, wherein the dynamic cue module comprises a global cue module and a local cue module;

[0060] The visual basic model DINOv2 includes N layers of encoders, through which the encoders preliminarily encode the input image to obtain the original features output by the encoders of each layer; in the training stage, all parameters of the encoders are frozen;

[0061] The dynamic prompt module randomly generates a global prompt vector and a local prompt vector for the original features output by the visual basic model DINOv2, interactively fuses the original features with the global prompt vector and the local prompt vector respectively to obtain two enhanced features, performs weighted summation on the obtained two enhanced features, and generates fused enhanced features of encoders at each layer;

[0062] The global hint module is used to generate a global hint vector, which identifies the shadow boundary in the image based on color and texture information to obtain the global hint enhancement feature;

[0063] The local hint module is used to generate a local hint vector, which identifies the shadow position in the image based on semantic information to obtain local hint enhancement features;

[0064] The memory module performs segmented fusion of the fusion enhancement features of the encoders of each layer to obtain the encoding features, and divides the encoding features into T single-frame features according to the time dimension; through the memory reading module, the single-frame features are matched and interacted with the key-value pairs stored in the memory bank, so that the single-frame features obtain the semantic information of the historical single frame and generate the memory enhancement features; the key-value encoder is used to encode the single-frame features to generate the corresponding key-value pairs, and the generated key-value pairs are stored in the memory bank, and the key-value pairs of the historical single frame are selectively eliminated according to the first-in-first-out rule;

[0065] The decoder first gradually improves the resolution of the memory enhancement feature, and then decodes the memory enhancement feature into a shadow mask.

[0066] Beneficial effects: The video shadow detection method and device based on the dynamic prompt memory network (DPMNet) provided by the present invention have the following advantages over the prior art: 1. The network architecture used in the present invention simultaneously considers the temporal correlation within and between segments, which can ensure the effective interaction within the segment and the long-term consistency of the features between segments; 2. The dynamic prompt module in the present invention detects blurred shadows by converting projected shadow instances into shadow features, thereby improving the performance of blurred shadow detection in complex scenes; 3. The prompt-based memory module in the present invention can reduce the error accumulation problem caused by encoding long-term features. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 Schematic diagram of the framework structure of the dynamic prompt memory network (DPMNet) in the method of the present invention;

[0068] Figure 2 1 is a flow chart for implementing the method of the present invention;

[0069] Figure 3 It is a structural block diagram of the dynamic prompt module and the prompt-based memory module. DETAILED DESCRIPTION

[0070] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0071] The present invention provides a video shadow detection method based on a dynamic prompt memory network (DPMNet), which can aggregate time information without accumulating errors by processing the obfuscated shadows with the help of objects that project shadows. Figure 1 The figure shows the framework structure of the dynamic prompt memory network (DPMNet), which includes two core modules: the dynamic prompt module (DPM) and the prompt-based memory module (PMM); the video frame is initially encoded by the DINOv2 encoder and then sent to the DPM. When it is difficult to directly encode the shadow, the DPM uses local and global prompts to convert the semantic information provided by DINOv2 into a shadow mask. The local prompt focuses on semantic conversion and identifies the shadow position based on semantic information. The global prompt identifies the shadow boundary based on color and texture information; the features processed by DPM are sent to the PMM for memory matching and updating. The PMM is used to solve the deformation and long-term temporal consistency problems while preventing error accumulation. The PMM uses local prompts as a proxy for temporal matching to reduce memory usage and minimize dependence on historical shadow masks.

[0072] like Figure 2 As shown, it is the implementation flow chart of the present invention, and each step is specifically described below.

[0073] Step S01: Send the video frames into the DINOv2 encoder in segments for preliminary encoding.

[0074] Given a video clip The video clip I is input into each layer encoder of the pre-trained visual basis model DINOv2 to obtain the original feature set D = {D1, D2, ... D i , …, D N}, N = 24 represents the total number of encoder layers, and the i-th layer encoder outputs the i-th layer original features The original feature D i Split into T single frames according to the time dimension Represents the original feature D i The current single frame, H and W represent the original features D i The height and width of , C represents the number of channels.

[0075] In order to retain the original generalization ability of the visual base model DINOv2 to the maximum extent, during the training phase, the encoder of the visual base model DINOv2 is frozen, the gradient information is not recorded, and the parameters of the encoder of the visual base model DINOv2 are not updated.

[0076] Step S02: interactively fuse the original features with two groups of hint vectors randomly initialized in advance to generate two enhanced features, and superimpose the two enhanced features on the original features.

[0077] Randomly initialize the local hint vector set L = {L1, L2, ... L i , …, L N} and the global hint vector G. The local hint vector L identifies the shadow position based on semantic information, and the global hint vector G identifies the shadow boundary based on color and texture information. i Respectively with the local hint vector L i Interactively fuse with the global prompt vector G to obtain two enhanced features, and perform weighted summation on the two enhanced features to generate the fused enhanced features of the i-th layer encoder.

[0078] A dynamic prompt module (DPM) is designed to allow the original features of each layer to be interactively fused with the two prompt vectors, so that the two prompt vectors can fully adjust the original features and transform the original features into fused enhanced features for video shadow detection; DPM includes a local prompt module (LPB) and a global prompt module (GPB). Their specific frameworks are as follows Figure 3 shown; based on Figure 3 The framework is shown, and the process of obtaining fusion enhanced features is explained below.

[0079] Step 21: Identify the shadow position through LPB, which specifically includes the following steps.

[0080] Step 211: Use LPB to randomly initialize a set of N for each layer encoder l , a local hint of length C, the local hint vector of the i-th layer is recorded as At the same time, the local hint vector L i Copy it into T copies in the time dimension, and take the tth local hint vector As the corresponding current single frame And single frame features The local hint subvector of .

[0081] Step 212: local hint subvector and the current single frame Perform a cross-attention operation to make the local hint sub-vector From the current single frame In order to introduce temporal clues to fuse the semantic information in the clip, it is also necessary to piece together all the aggregated effective information in each layer of the encoder for self-attention operation, and fuse the local prompt vectors from different single frames:

[0082]

[0083] Among them: CrossAttn and SelfAttn represent the cross attention mechanism and self-attention mechanism respectively, MLP represents the multi-layer perceptron, and the output of the multi-layer perceptron MLP is The multi-layer perceptron MLP includes two fully connected layers, one fully connected layer is used to reduce the number of channels of the local prompt vector from C to C′ to reduce the amount of calculation, and the other fully connected layer is used to restore the number of channels of the local prompt vector from C′ to C, so that the local prompt can extract more semantic information.

[0084] Step 213: Using a method similar to the attention mechanism, the local hint vector dynamically adjusts the original feature D according to different positions of the image. i The characteristics of the current single frame The focus of the algorithm is moved from the instance object to the shadow area; specifically, first calculate With the current single frame The attention map is then converted into attention weights using the softmax function, and finally the attention weights are used to fuse the current single frame. and

[0085]

[0086] F i l =Concat(F il1 , F i l2 , …, F i lt , …, F i lT )

[0087] in: Indicates the current single frame and The attention map between i lt Indicates the current single frame and After fusion, is a learnable matrix, W L Shared in each layer of the encoder, Concat represents the splicing operation in the time dimension, represents the output of the local cue module in the i-th layer encoder.

[0088] Step 22: When determining boundaries, the human eye usually focuses on low-level features such as color and texture. These low-level features are essential attributes of the image and are independent of time. Therefore, the global hint vector can be shared among the various layers of the encoder so that all single frames use this set of common global hints. The shadow boundaries are identified through the global hint module, which specifically includes the following steps.

[0089] Step 221: original feature D i Perform spatial average pooling and temporal average pooling operations respectively to obtain spatial global features and temporal global features

[0090] P i spatial =AvgPool hw (D i )

[0091] P i temporal =AvgPool t (D i )

[0092] Where: AvgPool hw Represents the average pooling operation along the spatial dimension, which converts the spatial dimension from Compression is 1; AvgPool t Represents an average pooling operation along the time dimension, compressing the time dimension from T to 1.

[0093] Step 222: Randomly initialize a set of N g , a global hint of length C, forming a global hint vector First, the global prompt vector G and the spatial global feature P are transformed in the time dimension. i spatial Perform cross attention operation, due to the spatial global feature P i spatial The spatial dimension of is transformed from The global hint vector G is compressed to 1, so that the global information of each single frame in space can be learned; then the multi-layer perceptron MLP is used to adjust the channel; and then the temporal global feature P is used in the spatial dimension. i temporal Perform cross-attention operations to learn the global information of each position in time; then perform channel adjustment through the multi-layer perceptron MLP, and finally output the enhanced global prompt vector

[0094]

[0095] Among them: TemporalAttn and SpatialAttn represent the cross attention mechanism in the time dimension and the cross attention mechanism in the spatial dimension respectively, the time dimension is T and the spatial dimension is

[0096] Step 223: Using the same method as in step 213, using the global hint vector To adjust the original feature D i , and obtain the output F of the global prompt module in the i-th layer encoder i g :

[0097]

[0098] F i g =Concat(F i g1 , F i g2 , …, F i gt , …F i gT )

[0099] in: Indicates the current single frame and The attention map between i gt Indicates the current single frame and After fusion, represents the output of the global cue module in the i-th layer encoder.

[0100] Step 23, F i l and F i g Dynamic weighting is performed to refine the original feature D i , get the fusion enhanced feature F of the i-th layer encoder i a :

[0101] F i a =α×F i l +β×F i g +D i

[0102] Among them: α and β are learnable weights used to dynamically adjust the two prompt vectors to the original feature D i The impact of represents the fusion enhanced features of the i-th layer encoder, F i a ={F i a1 , F i a2 , …, F i at , …F i aT}.

[0103] Step S03: Calculate coding features.

[0104] Repeat step 02 for the original features of each layer to obtain the fused enhanced features of each layer. Perform segmented fusion on the fused enhanced features of each layer to obtain the encoded features. Specifically, all layers of the encoder are evenly divided into four consecutive stages, and then the fusion enhancement features of the last layer of each stage are selected and finally merged into the encoding features

[0105] (31) The encoder The N-layer fusion enhancement features are extracted, and each fusion enhancement feature is used to reduce the number of channels from C to C using a fully connected layer. e , thus obtaining four intermediate features;

[0106] (32) The four intermediate features are concatenated together in the channel dimension, and the number of channels after concatenation is 4C e , and then use a 1×1 convolutional layer to increase the number of channels from 4C e Restore to C and get the encoding features

[0107] Step S04: Segment the coded features into single-frame features according to the time dimension, enhance themselves through matching and interaction between the memory reading module and the memory library, and update and eliminate the information in the memory library.

[0108] In order to use the historical single frame to reduce the background noise of the current single frame and establish the connection between different single frames, a prompt-based PMM is designed. The PMM includes two parts: a key-value encoder and a memory reading module. The PMM uses the local prompt vectors of the historical single frame to build a memory library, and uses the memory library for shadow detection in the current single frame. The structural block diagram of the PMM is shown in the figure. Figure 3 As shown, the specific steps include the following steps.

[0109] Step 41: In the key-value encoder, the local hint subvector and single frame features Encode and generate the current single frame The key-value pair, and then the current single frame The key-value pairs are updated into the memory.

[0110] Step 411: Encode the feature Split into T single-frame features according to the time dimension

[0111] Step 412: For single frame features First, a 1×1 convolutional layer is used to reduce the number of channels from C to C m , and then extract the value region through a convolutional gating unit; the convolutional gating unit includes two-way processing, one of which selects single-frame features through a fully connected layer and a depth-separable convolution The other processing uses a fully connected layer to align the features; then the results of the two processing are multiplied element by element, and finally the single frame features are Encoded as value V t Store into the memory; specifically includes the following steps:

[0112]

[0113] Among them: ConvGLU represents convolutional gating unit, Cony 1×1 represents a 1×1 convolutional layer, V t Represents the single frame feature The current single frame D obtained after encoding i t The value of .

[0114] Step 413: For the local hint sub-vector, select the single-frame feature The corresponding local hint subvector of the last encoder layer Align local cue subvectors using a fully connected layer The local hint subvector Encoded as key K t , while processing the current single frame When the memory bank stores the previous T max The key-value pairs of a single frame in the frame history are as follows:

[0115]

[0116] Among them: Linear represents the linear layer alignment operation; K t Indicates the current single frame Key, V t Indicates the current single frame The value of M t Refers to processing the next single frame The memory bank of time.

[0117] Step 42: When reading the memory library, first use a fully connected layer to align the next single frame The local hint subvector of The local hint subvector Encoded into a query vector, the query vector is then matched with the memory library to generate the enhanced memory feature H t+1 , expressed as:

[0118]

[0119] H t+1 =softmax(Q t+1 ×K T )×V

[0120] Where: Q t+1 Represents the local hint subvector The query vector obtained after alignment through the linear layer; K and V refer to M t All the keys and all the values ​​in; Indicates processing of a single frame Enhanced memory characteristics.

[0121] Step 43: First, t+1 Pass it to a fully connected layer to convert this N l The matching results are merged into a single result, and then a fully connected layer is used to reduce the number of channels from C m Restore to C, for H t+1 and Add element by element to finally obtain the memory enhancement feature Z t .

[0122] Step 44: Set the current single frame The key-value pairs of each frame are stored in the memory bank, and the key-value pairs of each frame are selectively eliminated according to the first-in-first-out rule.

[0123] Step S05: Through a lightweight decoder, the resolution of the memory enhancement feature is gradually improved and the memory enhancement feature is decoded into a shadow mask.

[0124] Generate a single frame of the current frame through a lightweight decoder Shadow mask Y t , due to the memory enhancement feature Z t The resolution is too small to express some details of the picture. Therefore, a progressive upsampling method is required in the decoder to gradually increase the memory enhancement feature Z. t The resolution from Adjust to Specifically, the number of channels is first reduced from C to C′ through a multi-layer perceptron MLP to reduce the amount of calculation, and then the resolution is doubled by using convolution and upsampling operations. After repeating twice, the shadow mask Y is output through a 1×1 convolution layer. t .

[0125] A video shadow detection device based on a dynamic prompt memory network is used to implement the above-mentioned video shadow detection method, mainly including a pre-trained visual basic model DINOv2, a dynamic prompt module (DPM), a prompt-based memory module (PMM) and a lightweight decoder. The DPM mainly includes a global prompt module (GPB) and a local prompt module (LPB).

[0126] The visual basic model DINOv2 includes N layers of encoders. The encoders perform preliminary encoding on the input image to obtain the original feature set D = {D1, D2, ... D i , …, D N}; During the training phase, freeze all parameters of the encoder.

[0127] DPM randomly generates a global hint vector G and a local hint vector L = {L1, L2, ... L i , …, L N}, the original feature D i Respectively, the global hint vector G and the local hint vector L i Interactive fusion is performed to obtain two enhanced features F i g and F i l , perform weighted summation on the two enhanced features obtained to generate the fusion enhanced feature F of each layer encoder i a .

[0128] GPB is used to generate a global hint vector, which identifies the shadow boundary in the image based on color and texture information, and obtains the global hint enhancement feature F. i g ;

[0129] LPB is used to generate a local hint vector, which identifies the shadow position in the image based on semantic information and obtains the local hint enhancement feature F. i l ;

[0130] The memory module performs segmented fusion of the fused enhanced features of each layer of encoder to obtain the encoding features Divide the encoded features into T single-frame features according to the time dimension Through the memory reading module, the single frame features Match and interact with key-value pairs stored in the memory bank to make single-frame features Obtain the semantic information of the historical single frame and generate the memory enhancement feature Z t ; Use key-value encoder to encode single frame features Encode and generate corresponding key-value pairs, store the generated key-value pairs in the memory bank, and selectively eliminate the key-value pairs of the historical single frame according to the first-in-first-out rule;

[0131] Lightweight decoder, first gradually improves the memory enhancement feature Z t The resolution of the memory enhancement feature Z t Decoded into shadow mask Y t .

[0132] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A video shadow detection method based on dynamic cue memory network, characterized in that: The method comprises two parts: a dynamic hint module and a hint-based memory module. The dynamic hint module converts the semantic information provided by the visual base model DINOv2 into a shadow mask by using local hints and global hints. The local hint focuses on semantic transformation and identifies the shadow position based on semantic information. The global hint identifies the shadow boundary based on color and texture information. The memory module is used to solve deformation and long-term temporal consistency problems while preventing error accumulation. The memory module uses local hints as a proxy for temporal matching to reduce memory usage and minimize dependence on historical shadow masks.

2. The video shadow detection method based on dynamic cue memory network according to claim 1 is characterized in that: The method comprises the following steps: (1) Given a video clip I, the video clip I is input into each layer encoder of the pre-trained visual basis model DINOv2 to obtain the original feature set D = {D1, D2, ... D i , …, D N }, the i-th layer encoder outputs the i-th layer original feature D i , the total number of layers of the encoder is N; (2) Randomly initialize the local hint vector set L = {L1, L2, ... L i , …, L N } and the global hint vector G,L i is the local prompt vector of the i-th layer; the original feature D i Respectively with the local hint vector L i Interactively fuse with the global hint vector G to obtain two enhanced features, perform weighted summation on the two enhanced features, and generate the fused enhanced features of the i-th layer encoder; (3) Repeat step (2) for the original features of each layer to obtain the fused enhanced features of each layer; The fusion enhancement features of each layer are segmented and fused to obtain the encoding features. (4) Encoding features Split into T single-frame features according to the time dimension At the same time, the original feature D i Split into T single frames according to the time dimension The local hint vector L i Duplicate it into T parts in the time dimension, and take the tth single frame As the original feature D i The current single frame, the tth local hint vector As the corresponding current single frame And single frame features The local prompt subvector of Match and interact with key-value pairs stored in the memory bank to make single-frame features Obtain the semantic information of the historical single frame and generate the memory enhancement feature Z t ; Select single frame features The corresponding local hint subvector of the last encoder layer Use key-value encoder to local hint sub-vector and single frame features Encode and generate the current single frame The key-value pair of the current single frame The key-value pairs of the historical single frame are stored in the memory bank, and the key-value pairs of the historical single frame are selectively eliminated according to the first-in-first-out rule; (5) Through a decoder, the memory enhancement feature Z is gradually improved t The resolution of the memory enhancement feature Z t Decoded into shadow mask Y t .

3. The video shadow detection method based on dynamic cue memory network according to claim 2 is characterized in that: In step (1), a video clip is given The video clip I is input into each layer encoder of the pre-trained visual basis model DINOv2, and the i-th layer encoder outputs the i-th layer original feature Represents the original feature D i The current single frame, H and W represent the original features D i The height and width of , C represents the number of channels; during the training phase, the encoder of the visual base model DINOv2 is frozen, the gradient information is not recorded, and the parameters of the encoder of the visual base model DINOv2 are not updated.

4. The video shadow detection method based on dynamic cue memory network according to claim 2 is characterized in that: In the step (2), the global hint vector G identifies the shadow boundary based on color and texture information, and the local hint vector L identifies the shadow position based on semantic information; the dynamic hint module allows the original features of each layer to be interactively fused with the two hint vectors respectively, so that the two hint vectors fully adjust the original features and convert the original features into fused enhanced features for video shadow detection; The dynamic prompt module includes a local prompt module and a global prompt module; the acquisition process of the fusion enhancement feature includes the following steps: (21) Identifying the shadow position through the local prompt module, specifically including the following steps: (211) Through the local hint module, a set of N numbers is randomly initialized for each layer of encoder l , a local hint of length C, the local hint vector of the i-th layer is recorded as At the same time, the local hint vector L i Copy it into T copies in the time dimension, and take the tth local hint vector As the corresponding current single frame And single frame features The local hint subvector of ; (212) for the local hint subvector and the current single frame Perform a cross-attention operation to make the local hint sub-vector From the current single frame Selectively aggregate valid information in the encoder layer; combine all the aggregated valid information in each layer of the encoder to perform self-attention operations and fuse local prompt vectors from different single frames: Among them: CrossAttn and SelfAttn represent the cross attention mechanism and self-attention mechanism respectively, MLP represents the multi-layer perceptron, and the output of the multi-layer perceptron MLP is The multi-layer perceptron MLP includes two fully connected layers, one fully connected layer is used to reduce the number of channels of the local prompt vector from C to C′, and the other fully connected layer is used to restore the number of channels of the local prompt vector from C′ to C; (213) First calculate With the current single frame The attention map is then converted into attention weights using the softmax function, and finally the attention weights are used to fuse the current single frame. and in: Indicates the current single frame and The attention map between i lt Indicates the current single frame and After fusion, is a learnable matrix, W L Shared in each layer of the encoder, Concat represents the splicing operation in the time dimension, represents the output of the local cue module in the i-th layer encoder; (22) Identifying shadow boundaries through a global hint module, specifically including the following steps: (221) For the original feature D i Perform spatial average pooling and temporal average pooling operations respectively to obtain spatial global features and temporal global features P i spatial =AvgPool hw (D i ) P i temporal =AvgPool t (D i ) Where: AvgPool hw Represents the average pooling operation along the spatial dimension, which converts the spatial dimension from Compression is 1; AvgPool t represents the average pooling operation along the time dimension, compressing the time dimension from T to 1; (222) Randomly initialize a set of N g , a global hint of length C, forming a global hint vector First, the global prompt vector G and the spatial global feature P are transformed in the time dimension. t spatial Perform cross attention operation; then adjust the channel through multi-layer perceptron MLP; then continue with the temporal global feature P in the spatial dimension i temporal Perform cross-attention operation; then adjust the channels through the multi-layer perceptron MLP, and finally output the enhanced global prompt vector Among them: TemporalAttn and SpatialAttn represent the cross attention mechanism in the time dimension and the cross attention mechanism in the spatial dimension respectively, the time dimension is T and the spatial dimension is (223) Using the same method as (213), using the global hint vector To adjust the original feature D i , and obtain the output F of the global prompt module in the i-th layer encoder i g : in: Indicates the current single frame and The attention map between i gt Indicates the current single frame and After fusion, represents the output of the global cue module in the i-th layer encoder; (23) To F i l and F i g Perform dynamic weighting to obtain the fusion enhanced feature F of the i-th layer encoder i a : F i a =α×F i l +β×F i g +D i Among them: α and β are learnable weights used to dynamically adjust the two prompt vectors to the original feature D i The impact of represents the fusion enhanced features of the i-th layer encoder, F i a ={F i al , F i a2 , …, F i at , …F i aT }.

5. The video shadow detection method based on dynamic cue memory network according to claim 2 is characterized in that: In step (3), all layers of the encoder are evenly divided into four consecutive stages, and the fusion enhancement features of the last layer of each stage are selected and merged into the encoding features The steps include: (31) The encoder The N-layer fusion enhancement features are extracted, and each fusion enhancement feature is used to reduce the number of channels from C to C using a fully connected layer. e , thus obtaining four intermediate features; (32) The four intermediate features are concatenated together in the channel dimension. The number of channels after concatenation is 4C. A 1×1 convolutional layer is then used to reduce the number of channels from 4C. e Restore to C and get the encoding features 6. The video shadow detection method based on dynamic cue memory network according to claim 2, characterized in that: In the step (4), in order to use the historical single frame to reduce the background noise of the current single frame and establish the connection between different single frames, a prompt-based memory module is designed, and the memory module includes two parts: a key-value encoder and a memory reading module; the memory module uses the local prompt vector of the historical single frame to establish a memory library, and uses the memory library for shadow detection of the current single frame; specifically, the following steps are included: (41) In the key-value encoder, the local hint subvector and single frame features Encode and generate the current single frame The key-value pair, and then the current single frame The key-value pairs are updated into the memory; (411) For single frame features First, a 1×1 convolutional layer is used to reduce the number of channels from C to C m , and then extract the value region through a convolutional gating unit; the convolutional gating unit includes two-way processing, one of which selects single-frame features through a fully connected layer and a depth-separable convolution The other processing uses a fully connected layer to align the features; then the results of the two processing are multiplied element by element, and finally the single frame features are Encoded as value V t Store into the memory; specifically includes the following steps: Among them: ConvGLU represents convolutional gated unit, Conv 1×1 represents a 1×1 convolutional layer, V t Represents the single frame feature The current single frame obtained after encoding The value of (412) For the local hint sub-vector, select the single frame feature The corresponding local hint subvector of the last encoder layer Align local cue subvectors using a fully connected layer The local hint subvector Encoded as key K t , while processing the current single frame When the memory bank stores the previous T max The key-value pairs of a single frame in the frame history are as follows: Among them: Linear represents the linear layer alignment operation; K t Indicates the current single frame Key, V t Indicates the current single frame The value of M t Refers to processing the next single frame Memory bank of time; (42) When reading the memory library, first use a fully connected layer to align the next single frame The local hint subvector of The local hint subvector Encoded into a query vector, the query vector is then matched with the memory library to generate the enhanced memory feature H t+1 , expressed as: H t+1 =softmax(Q t+1 ×K T )×V Where: Q t+1 Represents the local hint subvector The query vector obtained after alignment through the linear layer; K and V refer to M t All the keys and all the values ​​in; Indicates processing of a single frame Enhanced memory characteristics when (43) First, H t+1 Pass it to a fully connected layer to convert this N l The matching results are merged into a single result, and then a fully connected layer is used to reduce the number of channels from C m Restore to C, for H t+1 and Add element by element to finally obtain the memory enhancement feature Z t .

7. The video shadow detection method based on dynamic cue memory network according to claim 2 is characterized in that: In the step (5), the current frame is generated by the decoder Shadow mask Y t , using the progressive upsampling method, gradually transforming the memory enhancement feature Z t The resolution from Adjust to Specifically, the number of channels is first reduced from C to C′ through a multi-layer perceptron MLP, and then the resolution is doubled by using convolution and upsampling operations. After repeating twice, the shadow mask Y is output through a 1×1 convolution layer. t .

8. A video shadow detection device based on a dynamic cue memory network, characterized in that: It includes a pre-trained visual basic model DINOv2, a dynamic prompt module, a prompt-based memory module and a decoder, wherein the dynamic prompt module includes a global prompt module and a local prompt module; The visual basic model DINOv2 includes N layers of encoders, and the encoders are used to preliminarily encode the input image to obtain the original features output by each layer of encoders; During the training phase, all parameters of the encoder are frozen; The dynamic prompt module randomly generates a global prompt vector and a local prompt vector for the original features output by the visual basic model DINOv2, interactively fuses the original features with the global prompt vector and the local prompt vector respectively to obtain two enhanced features, performs weighted summation on the obtained two enhanced features, and generates fused enhanced features of encoders at each layer; The global hint module is used to generate a global hint vector, which identifies shadow boundaries based on color and texture information to obtain enhanced features about the global hint; The local hint module is used to generate a local hint vector, which identifies the shadow position based on semantic information to obtain local hint enhancement features; The memory module performs segmented fusion of the fused enhancement features of each layer of encoder to obtain a coding feature, and divides the coding feature into T single-frame features according to the time dimension; Through the memory reading module, the single-frame features are matched and interacted with the key-value pairs stored in the memory bank, so that the single-frame features can obtain the semantic information of the historical single frames and generate memory-enhanced features; the single-frame features are encoded using the key-value encoder to generate the corresponding key-value pairs, and the generated key-value pairs are stored in the memory bank, and the key-value pairs of the historical single frames are selectively eliminated according to the first-in-first-out rule; The decoder first gradually improves the resolution of the memory enhancement feature, and then decodes the memory enhancement feature into a shadow mask.

Images