A video camouflage target detection method and device based on explicit motion modeling

Abstract

This invention discloses a video camouflage target detection method based on explicit motion modeling. The method includes acquiring RGB images of the video to be detected, processing the RGB images based on a pre-trained video camouflage target detection model, and obtaining optical flow estimation results and camouflage target detection results. The video camouflage target detection model includes an explicit motion modeling branch and a camouflage target detection branch. This invention combines the explicit motion modeling branch and the camouflage target detection branch, allowing for mutual optimization and error correction. This enables better extraction of effective motion cues and effectively overcomes the error accumulation problem introduced by low-quality optical flow maps and the limitation of model capabilities by implicit motion modeling, thereby obtaining more accurate camouflage target detection results.

Description

Technical Field

[0001] This invention relates to the field of camouflaged target recognition technology, and in particular to a video camouflaged target detection method based on explicit motion modeling. Background Technology

[0002] Camouflaged object detection (COD) aims to detect and segment "hidden objects" that have a high intrinsic similarity to the background. COD has a wide range of applications, such as medical segmentation, industrial inspection, military applications, and agricultural quality inspection. With the rise of deep learning technology, the field of object detection has made significant progress. However, detecting camouflaged targets still faces many challenges, such as blurred target boundaries, difficult-to-locate target positions, excessively small camouflaged targets, and overly cluttered backgrounds. Video camouflaged object detection introduces temporal information to obtain motion cues, breaking the camouflage state of the target and thus better segmenting it. How to effectively utilize motion cues is an important aspect of video camouflaged object detection.

[0003] Currently, deep learning-based video camouflage target detection methods typically employ two types of approaches to utilize motion cues: The first type leverages existing optical flow estimation methods, directly inputting offline optical flow estimation results into the network. However, current optical flow estimation methods are not always successful in capturing the motion of camouflage targets, potentially introducing errors and noise. Inputting low-quality optical flow estimation results into the network often leads to error accumulation, thus affecting the performance of target segmentation. The second type combines implicit motion modeling and target detection within the same framework. However, this strategy, lacking explicit evaluation and regularization of the acquired motion cues, limits the model's capabilities, further impacting the accuracy of camouflage target detection results. Summary of the Invention

[0004] The purpose of this invention is to overcome the limitations of existing technologies in utilizing optical flow estimation results effectively during camouflage target detection, as well as the impact of implicit motion modeling on detection results, thereby providing a video camouflage target detection method based on explicit motion modeling.

[0005] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0006] A video camouflage target detection method based on explicit motion modeling includes:

[0007] Obtain the RGB image of the video to be detected; the RGB image includes two adjacent frames.

[0008] The RGB image is processed based on a pre-trained video camouflage target detection model to obtain optical flow estimation results and camouflage target detection results.

[0009] The video camouflage target detection model includes an encoder, an explicit motion modeling branch, a camouflage target detection branch, a bidirectional feature update module, and a decoder.

[0010] Preferably, the pre-trained video camouflage target detection model processes the RGB image by:

[0011] The encoder is used to obtain the hierarchical feature sequence of the RGB image;

[0012] Based on the aforementioned display motion modeling branch, the hierarchical feature sequence is processed and motion features are obtained;

[0013] The hierarchical feature sequence and the motion feature are updated based on the bidirectional feature update module, and the updated hierarchical feature sequence and the updated motion feature are output.

[0014] Based on the decoder, the updated motion features are identified and optical flow estimation results are obtained;

[0015] The updated hierarchical feature sequence is detected based on the camouflage target detection branch, and the camouflage target detection result is obtained.

[0016] Preferably, the step of processing the hierarchical feature sequence and obtaining motion features based on the display motion modeling branch specifically includes:

[0017] The hierarchical feature sequence is divided into feature pairs in layers, and the 4D correlation volume of one layer is constructed.

[0018] The relevant volume is queried using the current optical flow value as an index, and then input into the update module based on the gated cyclic unit (GRU) to obtain motion features.

[0019] Preferably, the 4D correlation volume is constructed according to the following formula:

[0020]

[0021] in, For 4D related quantities, volume This is a hierarchical feature sequence from the previous frame. Let i be the hierarchical feature sequence of the next frame, i be the layer number, (x, y) be the position index of the hierarchical feature sequence of the previous frame, (u, v) be the position index of the hierarchical feature sequence of the next frame, and c be the channel index.

[0022] Preferably, the update module based on the gated recurrent unit (GRU) obtains the motion features according to the following formula:

[0023] z t =σ(Conv([ht-1 ,x t ],W z )),

[0024] r t =σ(Conv([h t-1 ,x t ],W r )),

[0025]

[0026]

[0027] Where, x t The feature is a concatenation of the current optical flow value, the correlation quantity obtained by querying the 4D correlation volume using the current optical flow value, and the hierarchical features of the previous frame; h t-1 This represents the output state of the previous loop; [,] indicates channel concatenation of the variables within the brackets; ⊙ represents pixel-wise multiplication of two variables, W z W r W h For learnable weight parameters; z t t r , This is an intermediate feature in the computation process of the gated recurrent unit (GRU); h t The output state for this loop is the current motion feature f. m σ represents the sigmoid activation function; tanh represents the tanh activation function; Conv represents the convolutional layer.

[0028] Preferably, the bidirectional feature update module updates the hierarchical feature sequence and the motion features, specifically including:

[0029] The bidirectional feature update module updates the motion features according to the following formula:

[0030]

[0031]

[0032] In this context, SA represents a spatial attention module with the same input and output tensor dimensions, CA represents a channel attention module with the same input and output tensor dimensions, Concat represents concatenation along the channel dimension, UP represents an upsampling module used to align feature dimensions at different levels with motion features, and ConvL consists of a convolutional layer with 3 output channels and a LeakyReLU layer. Indicates intermediate features. This represents the updated motion features. These represent features from different layers in the hierarchical feature sequence of the previous frame;

[0033] The bidirectional feature update module updates the hierarchical feature sequence according to the following formula:

[0034]

[0035] Where i represents the layer number, SA represents the spatial attention module with the same input and output tensor dimensions, CA represents the channel attention module with the same input and output tensor dimensions, Concat represents concatenation along the channel dimension, and DB represents the downsampling module used to process motion features. Align with feature dimensions at different levels. This represents the updated features of the previous frame at different levels.

[0036] Preferably, the decoder obtains the optical flow estimation result by acquiring the optical flow update amount and updating the current optical flow value.

[0037] Preferably, the step of detecting the updated hierarchical feature sequence based on the camouflage target detection branch and obtaining the camouflage target detection result specifically includes: first, generating a rough prediction map by the nearest neighbor decoder, then gradually optimizing it by the grouped reverse attention module, and finally obtaining the final camouflage target detection result after Sigmoid activation.

[0038] Preferably, the pre-trained video camouflage target detection model is trained using a joint learning loss function, which is:

[0039]

[0040] in, To model the branch loss function for explicit motion, For the loss function of the camouflage target detection branch, the binomial cross-entropy loss function is adopted; Represented as:

[0041]

[0042] Where N is the number of iterations in the explicit motion modeling branch, D is the pixel-level photometric loss SSIM, and I t+1 For the next frame, For the previous frame I t The reconstructed frame is distorted based on the optical flow estimation results, where γ is the attenuation coefficient.

[0043] In another aspect, the present invention provides a video camouflage target detection device based on display motion modeling, comprising at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform any of the detection methods described above.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] 1. This invention proposes a video camouflage target detection method based on explicit motion modeling. By combining the explicit motion modeling branch with the camouflage target detection branch, they can optimize and correct each other, thus better extracting effective motion cues. This effectively overcomes the error accumulation problem introduced by the low-quality optical flow graph and the limitation of model capabilities by implicit motion modeling, thereby obtaining more accurate camouflage target detection results.

[0046] 2. By adding a bidirectional feature update module to the camouflage target detection branch and the explicit motion modeling branch, the camouflage target detection results and motion modeling results can promote and restrict each other, further realizing the mutual optimization and error correction of appearance feature information and motion feature information, and improving the accuracy of detection results. Attached Figure Description

[0047] Figure 1 This is a flowchart of the detection method of the present invention;

[0048] Figure 2 This is a schematic diagram of the network structure of the detection method of the present invention;

[0049] Figure 3 This is a network structure diagram of the feature bidirectional update module of the present invention;

[0050] Figure 4 This is a comparison chart of the detection results of the present invention; Detailed Implementation

[0051] The present invention will be further described in detail below with reference to experimental examples and specific embodiments. However, this should not be construed as limiting the scope of the above-mentioned subject matter of the present invention to the following embodiments; all technologies implemented based on the content of the present invention fall within the scope of the present invention.

[0052] Example 1

[0053] This invention provides a video camouflage target detection method based on explicit motion modeling, comprising:

[0054] S1. Obtain the RGB image of the video to be detected; the RGB image includes two adjacent frames.

[0055] S2. Processing the RGB image based on a pre-trained video camouflage target detection model, specifically including:

[0056] S21. Obtain the hierarchical feature sequence of the RGB image based on the encoder;

[0057] S22. Process the hierarchical feature sequence based on the displayed motion modeling branch and obtain motion features;

[0058] S23. Update the hierarchical feature sequence and the motion feature based on the bidirectional feature update module, and output the updated hierarchical feature sequence and the updated motion feature.

[0059] S24. Based on the decoder, identify the updated motion features and obtain the optical flow estimation result;

[0060] S25. Detect the updated hierarchical feature sequence based on the camouflage target detection branch and obtain the camouflage target detection result.

[0061] Example 2

[0062] Figure 1 This paper illustrates the steps of a video camouflage target detection method based on explicit motion modeling according to an exemplary embodiment of the present invention. In this embodiment, two adjacent frames of a video are input into a pre-trained video camouflage target detection model. Through forward propagation of the network, the final video camouflage target detection result and optical flow estimation result are obtained from the decoder. The camouflage target detection branch and the explicit motion modeling branch use the same encoder, which is based on the common PVTV2 network structure. The method of this embodiment mainly includes the following steps:

[0063] As described in S1 above, the RGB images of two adjacent frames of the video to be detected are obtained, the previous frame is recorded as the previous frame and the next frame as the next frame, and both frames are scaled to 352×352.

[0064] As described in S21 above, two adjacent frames are input into the encoder, and the channel dimension is uniformly transformed to a uniform value k using the convolution operator. In this embodiment, k is set to 32, thereby obtaining the hierarchical feature sequence of the previous frame. and the hierarchical feature sequence of the next frame Taking the features of the previous frame as an example, their feature resolutions are: 32×88×88, 32×44×44, 32×22×22, and 32×11×11.

[0065] As described in S22 above, the second layer features from the feature sequences of the previous and next frames are taken to form feature pairs. The generated motion features are then input into the explicit motion modeling branch. Specifically, this embodiment introduces the iterative optimization structure from the paper "Raft: Recurrent All-pairs Field Transforms for Optical Flow" to capture the motion features f. m ,like Figure 2 As shown, it specifically includes:

[0066] Furthermore, take the feature pair Construct the 4D correlation volume using the following formula:

[0067]

[0068] in, For 4D related quantities, volume This is a hierarchical feature sequence from the previous frame. Let be the hierarchical feature sequence of the next frame, where i is the layer number, (x, y) is the position index of the hierarchical feature sequence of the previous frame, (u, v) is the position index of the hierarchical feature sequence of the next frame, and c represents the channel index. Further, the relevant volume is queried using the current optical flow g as the index and input into the update module based on a gated recurrent unit (GRU) to obtain the updated motion feature f. m .

[0069] Specifically, the update module based on the gated recurrent unit (GRU) uses the following formula to update the motion features:

[0070] z t =σ(Conv([h t-1 ,x t ],W z )),

[0071] r t =σ(Conv([h t-1 ,x t ],W r )),

[0072]

[0073]

[0074] Where, x t The feature is a concatenation of the current optical flow, the correlation quantity obtained by querying the 4D correlation volume with the current optical flow value, and the hierarchical features of the previous frame; h t-1 This represents the output state of the previous loop; [,] indicates concatenating the channels of the variables within the square brackets; ⊙ represents the pixel-wise multiplication of two variables; W z W r Wh For learnable weight parameters; z t r t , This is an intermediate feature in the computation process of the gated recurrent unit (GRU); h t The output state for this loop is the current motion feature f. m σ represents the sigmoid activation function; tanh represents the tanh activation function; Conv represents the convolutional layer.

[0075] As described in S23 above, the hierarchical feature sequence of the previous frame is... and the motion feature f described in S3 m All inputs are fed into the bidirectional feature update module to obtain the updated hierarchical feature sequence. and motion characteristics like Figure 3 As shown. Specifically includes:

[0076] Furthermore, the motion features are updated according to the following formula:

[0077]

[0078]

[0079] Here, SA represents a spatial attention module with the same input and output tensor dimensions, and CA represents a channel attention module with the same input and output tensor dimensions. Concat represents concatenation along the channel dimension, and UP represents an upsampling module used to align feature dimensions at different levels with motion features. ConvL consists of a convolutional layer with 3 output channels and a LeakyReLU layer. Indicates intermediate features, This represents the updated motion features. These represent features from different layers in the hierarchical feature sequence of the previous frame.

[0080] Specifically, in this example, when implementing motion feature updates, SA and CA are the spatial attention module and channel attention module proposed in the paper "Cbam: Convolutionalblock attention module", respectively.

[0081] Furthermore, the updated motion features and feature vector sequences at each level The features are concatenated separately, then channel attention is applied, and the feature sequence is updated accordingly to obtain a further updated hierarchical feature sequence. Specifically, the features of the previous frame at different levels are updated according to the following formula:

[0082]

[0083] Where i∈{2,3,4} represents the layer number in the hierarchical feature sequence of the previous frame. SA represents a spatial attention module with the same input and output tensor dimensions, and CA represents a channel attention module with the same input and output tensor dimensions. Concat represents concatenation along the channel dimension, and DB represents a downsampling module used to process motion features. Align with feature dimensions at different levels. This represents the updated features of the previous frame at different levels.

[0084] As described in S24 above, the updated motion features described in S23 will be... The input is fed into the decoder of the explicit motion modeling branch to obtain the current optical flow update amount Δg, and the optical flow is updated: g = g + Δg. Specifically, the decoder consists of two convolutional layers and a ReLU layer. The optical flow is initialized to 0; steps S22 to S24 are repeated (n-1) times to obtain the final optical flow estimation result. Here, n = 6, which is the set number of iterations.

[0085] As described in S25 above, the hierarchical feature sequence of the previous frame, updated n times, is input into the camouflage target detection branch to obtain the final target detection result in the [0,1] interval. Specifically, this embodiment uses the decoder from the literature "Concealed object detection". First, the nearest neighbor decoder (NCD) generates a coarse prediction map, which is then progressively optimized by the grouped inverse attention module (GRA). After Sigmoid activation, the final detection result is obtained, as shown below. Figure 2 As shown.

[0086] Specifically, before using the network, the video camouflage target detection model is trained, the model training loss function is calculated and error backpropagation is performed, and gradient descent optimization is carried out to optimize the network parameters of the target detection branch and the explicit motion modeling branch.

[0087] Specifically, the overall training loss function of the model consists of two parts: the target detection branch loss function and the explicit motion modeling branch loss function, which are calculated using the following formula:

[0088]

[0089] in, To model the branch loss function for explicit motion, The loss function is used for the camouflage target detection branch.

[0090] Specifically, the object detection branch obtains the loss by calculating the binomial cross-entropy loss function of the predicted and ground truth results. The explicit motion modeling branch uses a self-supervised strategy to calculate the loss, adjusting the previous frame I based on the optical flow result of the k-th iteration.t Twist it to get I t+1 The reconstructed frame is represented as We calculate I t+1 and The distance between them supervises the optical flow. The loss function of the explicit motion modeling branch. Represented as:

[0091]

[0092] Where N is the number of iterations in the motion modeling branch, D is the pixel-level photometric loss SSIM, and I t+1 Indicates the next frame. Indicates the previous frame i t The reconstructed frame after distortion is based on optical flow estimation. γ represents the attenuation coefficient. In this example, γ = 0.8 and N = 6.

[0093] In a further embodiment of the present invention, we conduct comparative experiments with three existing video camouflage target detection methods, such as... Figure 4 As shown, the video camouflage target detection model provided by this invention yields more accurate detection results than the three existing methods.

[0094] Example 3

[0095] In another aspect, the present invention provides a video camouflage target detection device based on display motion modeling, comprising at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform any of the detection methods described above.

[0096] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video camouflage target detection method based on explicit motion modeling, characterized in that, include: Obtain the RGB image of the video to be tested; The RGB image includes two adjacent frames; The RGB image is processed based on a pre-trained video camouflage target detection model to obtain optical flow estimation results and camouflage target detection results. The video camouflage target detection model includes an encoder, an explicit motion modeling branch, a camouflage target detection branch, a bidirectional feature update module, and a decoder. The pre-trained video camouflage target detection model processes the RGB image including: The encoder is used to obtain the hierarchical feature sequence of the RGB image; The hierarchical feature sequence is processed based on the explicit motion modeling branch to obtain motion features; The hierarchical feature sequence and the motion feature are updated based on the bidirectional feature update module, and the updated hierarchical feature sequence and the updated motion feature are output. Based on the decoder, the updated motion features are identified and optical flow estimation results are obtained; Based on the camouflage target detection branch, the updated hierarchical feature sequence is detected and the camouflage target detection result is obtained; The specific steps of processing the hierarchical feature sequence and obtaining motion features based on the explicit motion modeling branch include: The hierarchical feature sequence is divided into feature pairs in layers, and the 4D correlation volume of one layer is constructed. The relevant volume is queried using the current optical flow value as an index, and then input into the update module based on the gated cyclic unit (GRU) to obtain motion features.

2. The video camouflage target detection method based on explicit motion modeling as described in claim 1, characterized in that, The 4D correlation volume is constructed according to the following formula: ， in, For 4D related quantities, volume This is a hierarchical feature sequence from the previous frame. For the hierarchical feature sequence of the next frame, For the number of floors, ( ) is the position index of the hierarchical feature sequence of the previous frame, ( () is the position index of the hierarchical feature sequence of the next frame. For channel indexing.

3. The video camouflage target detection method based on explicit motion modeling as described in claim 1, characterized in that, The update module based on the gated recurrent unit (GRU) obtains the motion features according to the following formula: ， ， ， ， in, The feature is a concatenation of the current optical flow value, the correlation quantity obtained by querying the 4D correlation volume with the current optical flow value, and the hierarchical features of the previous frame. This represents the output state of the previous loop; [,] indicates that the variables within the brackets are concatenated into channels. This represents the pixel-wise multiplication of two variables. , , These are learnable weight parameters; , , These are intermediate features in the Gated Recurrent Unit (GRU) computation process; The output state for this loop, i.e., the current motion characteristics. ; represents the sigmoid activation function; tanh represents the tanh activation function; Conv represents a convolutional layer.

4. The video camouflage target detection method based on explicit motion modeling as described in claim 1, characterized in that, The bidirectional feature update module updates the hierarchical feature sequence and the motion features, specifically... include: The bidirectional feature update module updates the motion features according to the following formula: ， ， In this context, SA represents a spatial attention module with the same input and output tensor dimensions, CA represents a channel attention module with the same input and output tensor dimensions, Concat represents concatenation along the channel dimension, UP represents an upsampling module used to align feature dimensions at different levels with motion features, and ConvL consists of a convolutional layer with 3 output channels and a LeakyReLU layer. Indicates intermediate features, This represents the updated motion features. , , These represent features from different layers in the hierarchical feature sequence of the previous frame; The bidirectional feature update module updates the hierarchical feature sequence according to the following formula: , in, The number of layers is represented by SA, which indicates a spatial attention module with the same input and output tensor dimensions; CA, which indicates a channel attention module with the same input and output tensor dimensions; Concat, which indicates concatenation along the channel dimension; and DB, which indicates a downsampling module used to process motion features. Align with feature dimensions at different levels. This represents the updated features of the previous frame at different levels.

5. The video camouflage target detection method based on explicit motion modeling as described in claim 1, characterized in that, The decoder obtains the optical flow estimation result by acquiring the optical flow update amount and updating the current optical flow value.

6. The video camouflage target detection method based on explicit motion modeling as described in claim 1, characterized in that, The process of detecting the updated hierarchical feature sequence based on the camouflage target detection branch and obtaining the camouflage target detection result specifically includes: first, generating a rough prediction map by the nearest neighbor decoder, then gradually optimizing it by the grouped reverse attention module, and finally obtaining the final camouflage target detection result after Sigmoid activation.

7. The video camouflage target detection method based on explicit motion modeling as described in claim 1, characterized in that, The pre-trained video camouflage target detection model is trained using a joint learning loss function, which is: ， in, To model the branch loss function for explicit motion, For the loss function of the camouflage target detection branch, the binomial cross-entropy loss function is adopted; Represented as: ， in, The number of iterations for the explicit motion modeling branch. SSIM for pixel-level photometric loss, For the next frame, The previous frame The reconstructed frame is distorted based on the optical flow estimation results. This is the attenuation coefficient.

8. A video camouflage target detection device based on explicit motion modeling, characterized in that, It includes at least one processor and a memory communicatively connected to the at least one processor; the memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to perform the detection method as described in any one of claims 1 to 7.

Images