Video camouflage object detection method, device and equipment

Abstract

The invention provides a video camouflage object detection method, device and equipment. The method comprises the following steps: obtaining a video clip containing a camouflage object; performing feature extraction on each frame of image in the video clip; and sequentially carrying out intra-frame space positioning processing, inter-frame time consistency processing and context space-time aggregation processing on the extracted features so as to identify and determine the camouflage object existing in the frame image. According to the video camouflage object detection method, camouflage object identification can be accurately carried out on video data.

Description

Technical Field

[0001] The present invention belongs to the technical field of video picture detection, and in particular relates to a video disguised object detection method, device and equipment. Background Art

[0002] Camouflage is a common biological phenomenon in which organisms avoid predators or approach prey by blending their texture, color, shape, and pattern with their surroundings. This phenomenon has motivated the development of camouflaged object detection (COD), an emerging and challenging field that focuses on identifying and segmenting camouflaged objects in static images. COD has wide applications in multiple fields, including polyp segmentation in medical imaging and COVID-19 lung infection segmentation, product defect detection in industrial inspection, security monitoring, autonomous driving, and even fashion design. Most existing COD methods are mainly designed for static images and perform poorly when detecting dynamic camouflaged objects in videos. This limitation has attracted the attention of the computer vision community and has driven the development of motion-based COD methods to achieve accurate video camouflaged object detection (VCOD).

[0003] Current VCOD methods can be divided into two main categories: one is to model motion using optical flow estimation, and the other is to generate pseudo labels using a strategy based on bidirectional optical flow. The first type of methods provides pixel-level motion information, but often performs poorly due to noise and inaccuracies in the optical flow output. The second type of methods filters noise through bidirectional consistency checks, achieving more accurate motion modeling, but still faces challenges in complex scenes. Methods such as SLT-Net suffer from inaccurate positioning, temporal inconsistency, and blurred boundaries in low-contrast videos. The main challenges of VCOD include: (i) Intra-frame spatial enhancement. Disguised objects often blend seamlessly with their environment, complicating spatial positioning. (ii) Inter-frame temporal consistency. Ensuring consistent detection across frames is difficult due to changes in motion dynamics and occlusion. (iii) Contextual spatiotemporal feature fusion. Effective integration of spatial and temporal features is critical to maintaining motion and appearance consistency. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a video disguised object detection method, device and electronic equipment that can accurately identify disguised objects in video data.

[0005] The present invention provides a method for detecting a video disguised object, comprising:

[0006] obtaining a video clip containing a camouflaged object;

[0007] Performing feature extraction on each frame of the video clip;

[0008] The extracted features are sequentially processed with intra-frame spatial positioning, inter-frame temporal consistency, and contextual spatiotemporal aggregation to identify and determine the camouflaged objects present in the frame image.

[0009] In some embodiments, the performing intra-frame spatial positioning processing on the extracted features includes:

[0010] Perform feature tensor adjustment on the extracted features to align the time dimension of each feature;

[0011] The adjusted features are divided into two groups, and both are processed by convolution, normalization and the first activation function to obtain two groups of intra-group features;

[0012] Calculate the spatial attention weights of the features in the first group using the second activation function;

[0013] Calculate the product of the spatial attention weight and each feature in the second group of features to obtain enhanced spatial features corresponding to each frame of the image;

[0014] The enhanced spatial features are rearranged to restore to the original arrangement state, and a spatial enhancement feature map corresponding to each frame of the image is obtained.

[0015] In some embodiments, the step of sequentially performing inter-frame temporal consistency processing on the extracted features includes:

[0016] Performing feature tensor adjustment on the spatially enhanced feature map to align the time dimensions of each feature;

[0017] The adjusted features are subjected to feature conversion processing by the first model, so that the converted features are spliced ​​along the channel dimension in the processing process to form a feature map that retains the time information between frames;

[0018] Performing time shift processing on the features spliced ​​along the width dimension in the feature graph;

[0019] Calculate the difference between the time information of the offset feature and the splicing time information of the feature map;

[0020] Using the second model to perform channel reduction processing on the difference to obtain time difference information, wherein the time difference information describes the motion state of the camouflaged target moving over time;

[0021] A time consistency feature corresponding to the frame image is calculated based on the time difference information.

[0022] In some embodiments, the calculating based on the time difference information to obtain the time consistency feature corresponding to the frame image includes:

[0023] Processing the time difference information using the third model to obtain a channel reduction feature map;

[0024] Adaptively average pooling, convolution, and mapping the channel-reduced feature map to obtain a temporal difference attention weight;

[0025] Multiplying the temporal difference attention weight by each feature in two adjacent frames of images;

[0026] The multiplication results are subjected to feature concatenation, and the fourth model is used to perform channel reduction processing to obtain the time consistency feature.

[0027] In some embodiments, the sequentially performing contextual spatiotemporal aggregation processing on the extracted features includes:

[0028] The features are divided to obtain low-level features and high-level features with different scales;

[0029] Upsampling the high-level features so that the scales of the low-level features and the high-level features are the same;

[0030] Using the fifth model to perform feature transformation, splicing and channel compression processing on the underlying features and high-level features, to obtain contextual spatiotemporal attention features corresponding to each frame of the image;

[0031] The contextual spatiotemporal aggregation feature is calculated based on the contextual spatiotemporal attention feature.

[0032] In some embodiments, the calculating of the contextual spatiotemporal aggregation feature based on the contextual spatiotemporal attention feature includes:

[0033] Multiply the contextual spatiotemporal attention features by the corresponding underlying features and high-level features element by element;

[0034] The multiplication results are concatenated and input into the sixth model for channel compression processing to obtain the contextual spatiotemporal aggregation features.

[0035] In some embodiments, the method further comprises:

[0036] Based on the intra-frame spatial positioning processing results, inter-frame temporal consistency processing results and contextual spatiotemporal aggregation processing results of each frame image, the frame image is analyzed and identified, and then the camouflaged object in each frame image is determined.

[0037] Another embodiment of the present invention also provides a video disguised object detection device, including:

[0038] An acquisition module, used for acquiring a video clip containing a camouflaged object;

[0039] A first processing module, configured to process each frame of the video clip to generate a corresponding binary pixel mask;

[0040] A feature extraction module, used for extracting features from each of the binary pixel masks;

[0041] The second processing module is used to perform intra-frame spatial positioning processing, inter-frame temporal consistency processing and context spatiotemporal aggregation processing on the extracted features in sequence, so as to identify and determine the camouflaged objects existing in the frame image.

[0042] In some embodiments, the performing intra-frame spatial positioning processing on the extracted features includes:

[0043] Perform feature tensor adjustment on the extracted features to align the time dimension of each feature;

[0044] The adjusted features are divided into two groups, and both are processed by convolution, normalization and the first activation function to obtain two groups of intra-group features;

[0045] Calculate the spatial attention weights of the features in the first group using the second activation function;

[0046] Calculate the product of the spatial attention weight and each feature in the second group of features to obtain enhanced spatial features corresponding to each frame of the image;

[0047] The enhanced spatial features are rearranged to restore to the original arrangement state, and a spatial enhancement feature map corresponding to each frame of the image is obtained.

[0048] Another embodiment of the present invention further provides an electronic device, comprising:

[0049] one or more processors;

[0050] a memory configured to store one or more programs;

[0051] When the one or more programs are executed by the one or more processors, the one or more processors implement the video disguised object detection method as described in any one of the above embodiments.

[0052] The beneficial effects of the present invention include constructing a new model for realizing video disguised object detection, which simultaneously models inter-frame, intra-frame and context features, and realizes comprehensive spatiotemporal perception. The model can accurately identify the disguised objects in each frame image by corresponding to the inter-frame, intra-frame and context features of the frame image, effectively realizing accurate detection of disguised objects in the video with high robustness.

[0053] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0054] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0056] Figure 1 FIG. 4 is a flow chart of a method for detecting a disguised object in a video according to an embodiment of the present invention.

[0057] Figure 2 FIG. 4 is a flow chart of a method for detecting disguised objects in a video according to another embodiment of the present invention.

[0058] Figure 3 FIG. 4 is a flow chart of a method for detecting disguised objects in a video according to another embodiment of the present invention.

[0059] Figure 4 FIG. 4 is a flow chart of a method for detecting disguised objects in a video according to another embodiment of the present invention.

[0060] Figure 5 FIG. 4 is a structural block diagram of a video disguised object detection device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings, but are not intended to limit the present invention.

[0062] It should be understood that various modifications may be made to the embodiments disclosed herein. Therefore, the following description should not be considered as limiting, but merely as an example of an embodiment. Other modifications within the scope of the invention will occur to those skilled in the art.

[0063] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the general description of the invention given above and the detailed description of the embodiments given below, serve to explain the principles of the invention.

[0064] These and other characteristics of the invention will become apparent from the following description of a preferred form of embodiment given as a non-limiting example, with reference to the accompanying drawings.

[0065] It should also be understood that, although the invention has been described with reference to some specific examples, those skilled in the art will be able to realize many other equivalent forms of the invention that have the characteristics recited in the claims and are therefore within the scope of protection defined thereby.

[0066] The above and other aspects, features and advantages of the present invention will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings.

[0067] Specific embodiments of the present invention are described hereinafter with reference to the accompanying drawings; however, it should be understood that the disclosed embodiments are merely examples of the present invention, which may be implemented in a variety of ways. Well-known and / or repeated functions and structures are not described in detail to avoid unnecessary or redundant details that obscure the present invention. Therefore, the specific structural and functional details disclosed herein are not intended to be limiting, but merely serve as a basis and representative basis for the claims to teach those skilled in the art to use the present invention in a variety of ways with substantially any suitable detailed structure.

[0068] This specification may use the phrases "in one embodiment," "in another embodiment," "in a further embodiment," or "in other embodiments," which may all refer to one or more of the same or different embodiments according to the present invention.

[0069] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0070] Camouflaged object detection (COD) is a challenging computer vision task that focuses on identifying objects that visually blend with their background. The problem has been widely studied in fields such as agriculture, industry, and art. Early COD methods used hand-crafted features such as texture, motion, and 3D convexity to distinguish objects from backgrounds. Recent advances in deep learning provide more robust solutions for COD. For example, SINet combines search and recognition modules for identifying camouflaged objects. TINet proposes an interactive guidance framework that uses texture labels and boundary weighted losses to enhance COD. BSA-Net simulates human vision by using dual attention streams to enhance the distinction between foreground and background boundaries. ZoomNet uses a mixed-scale triplet network to capture scale-specific features. UEDG combines uncertainty and edge data in the UEDG architecture to improve detection accuracy. FEDER emphasizes feature decomposition and edge reconstruction, and Camodiffusion uses a diffusion model to generate multiple fine predictions for camouflaged object detection, thereby reducing overconfidence errors.

[0071] Video camouflaged object detection (VCOD) extends COD to dynamic environments and emphasizes the importance of using motion cues to identify camouflaged objects. Early VCOD methods rely on homography or optical flow to detect motion patterns. For example, using optical flow vectors to enhance motion detection, a motion-based framework with a differentiable registration module was proposed. The MoCA dataset was also created for detecting camouflaged animals in videos. A self-supervised system that uses a Transformer variant to segment optical flow frames. Motion information is extracted using pseudo-labels. VCOD is performed using a scaling strategy and uncertainty-aware loss. Implicit and explicit motion learning are combined for better boundary supervision. However, accurately estimating optical flow in dynamic scenes with fast-moving objects remains a challenge.

[0072] Current state-of-the-art VCOD methods perform well in simple scenes, but usually rely on heuristic initial motion models and pseudo labels obtained from optical flow estimation. The main challenge of VCOD lies in the effective modeling of spatiotemporal features. Existing models often integrate these features in a generalized way, lacking targeted and detailed modeling, and have difficulty in accurately processing spatial and temporal information. This often leads to problems such as inaccurate positioning of camouflaged objects and temporal inconsistency.

[0073] To solve this problem, this application proposes a method that achieves more accurate camouflaged object detection in complex videos by globally modeling spatiotemporal information without relying on pseudo optical flow labels. Figure 1 As shown, the method of the present invention comprises:

[0074] S1: Obtain a video clip containing a camouflaged object;

[0075] S2: extracting features from each frame of the video clip;

[0076] S3: performing intra-frame spatial positioning processing, inter-frame temporal consistency processing and context spatiotemporal aggregation processing on the extracted features in sequence, so as to identify and determine the camouflaged objects existing in the frame image.

[0077] The above methods in this embodiment are all uniformly executed by the newly created model, which mainly goes through three stages when processing video clips, namely, intra-frame spatial positioning stage, inter-frame temporal consistency stage, and contextual spatiotemporal interaction stage. This structure effectively models inter-frame, intra-frame, and contextual features. Given a video clip containing a camouflaged object, the model of this embodiment outputs a binary pixel mask for each frame to identify the camouflaged object. Assume represents a video clip with T frames, where H and W are the height and width of each frame respectively. For an image sequence, the model generates a sequence of predicted binary masks Used to determine the camouflaged object recognition of each frame, that is, the camouflaged object recognized in each frame of the image is output in the form of a binary pixel mask. The model in this embodiment has three modules, including an intra-frame spatial positioning module (ISPM): This module enhances the intra-frame spatial positioning through a group self-attention mechanism to improve the accuracy of object detection. 2) Inter-frame temporal consistency module (ITCM): This module captures time offsets by performing pairwise consistency operations to ensure inter-frame consistency, thereby obtaining a smoother feature representation. 3) Contextual spatiotemporal aggregation module (CSAM): Unlike traditional fusion methods, CSAM uses an adaptive attention mechanism to aggregate multi-scale contextual spatiotemporal features to enhance prediction accuracy, generalization ability and stability, and can accurately locate and segment objects that are visually integrated with the background in video sequences, that is, camouflaged objects.

[0078] Specifically, this embodiment uses the same visual Transformer backbone as SLT-Net, namely PVTv2-B5, to perform multi-level feature extraction (f 1 ,f 2 ,f 3 ,f 4 ).

[0079] like Figure 2 As shown, the intra-frame spatial positioning processing of the extracted features includes:

[0080] S4: perform feature tensor adjustment on the extracted features to align the time dimensions of each feature;

[0081] S5: Divide the adjusted features into two groups, and process both of them through convolution, normalization and the first activation function to obtain two groups of intra-group features;

[0082] S6: Calculate the spatial attention weights of the features in the first group using the second activation function;

[0083] S7: Calculate the product of the spatial attention weight and each feature in the second group of features to obtain enhanced spatial features corresponding to each frame of the image;

[0084] S8: Rearrange the enhanced spatial features to restore them to their original arrangement state, and obtain a spatial enhancement feature map corresponding to each frame of the image.

[0085] For example, in video disguised object detection (VCOD), it is very challenging to accurately identify and locate disguised objects because they are usually small in scale and highly similar in texture and color to the surrounding environment. To solve these problems, this embodiment constructs an intra-frame spatial positioning module (ISPM). This module aims to solve the problem of unclear and inaccurate positioning of disguised objects in the frame by independently processing each video frame using a group self-attention mechanism.

[0086] Specifically, given a series of video frames with disguised features Where B is the batch size, T is the frame index, C is the number of channels, H and W are the height and width of the feature map, respectively. First, align the time dimension by adjusting the feature tensor: F(T, B, C, H, W) = Rea(F ((B·T)，C，H，W) ), where Rea represents a rearrangement operation.

[0087] For each frame, the feature map is divided into two branches. Each branch undergoes a series of transformation operations, including 3×3 convolution, batch normalization, and ReLU activation function processing. Then, these features are rearranged into N groups to generate feature maps within the group. and F′ t =Rea(CBR(F t )), F″ t =Rea(CBR(F t )). CBR represents a series of operations: 3x3 convolution, batch normalization, and ReLU activation function. In this embodiment, CBR is regarded as the first model, and the subsequent second model, third model, fourth model, and fifth model have the same meaning, but the number of convolution layers and activation functions are not necessarily the same, and can be flexibly changed according to specific needs.

[0088] Next, this embodiment proposes a group space self-attention enhancement mechanism applied to these two groups of features. The softmax activation function is used to calculate F t ′’s spatial attention weights. These attention weights are then combined with F″ t Element-wise multiplication is performed to enhance the spatial features through the self-attention mechanism. The enhanced features are rearranged back to their original shape to obtain the spatially enhanced features Where Soft represents the softmax activation function. This group space self-attention enhancement mechanism is applied to each frame F t , ensuring that the features of each frame are spatially enhanced to achieve better localization. After all frames are processed, all enhanced frame features are superimposed together and then rearranged back to their original shape to finally obtain the enhanced feature map Stack represents the superposition operation of all enhanced frames.

[0089] Furthermore, if Figure 3 As shown, the extracted features are sequentially subjected to inter-frame temporal consistency processing, including:

[0090] S9: performing feature tensor adjustment on the spatial enhancement feature map to align the time dimensions of each feature;

[0091] S10: performing feature conversion processing on the adjusted features through the first model, so that the converted features are spliced ​​along the channel dimension in the processing process to form a feature map that retains the time information between frames;

[0092] S11: performing time shift processing on the features spliced ​​along the width dimension in the feature graph;

[0093] S12: Calculate the difference between the time information of the offset feature and the splicing time information of the feature map;

[0094] S13: performing channel reduction processing on the difference using the second model to obtain time difference information, where the time difference information describes the motion state of the camouflaged target moving over time;

[0095] S14: Calculate the time consistency feature corresponding to the frame image based on the time difference information.

[0096] The step of calculating the time consistency feature corresponding to the frame image based on the time difference information includes:

[0097] S15: Processing the time difference information using the third model to obtain a channel reduction feature map;

[0098] S16: performing adaptive average pooling, convolution, and mapping processing on the channel reduction feature map to obtain a time difference attention weight;

[0099] S17: multiplying the time difference attention weight by each feature in two adjacent frames of images;

[0100] S18: Perform feature concatenation on the multiplication results, and perform channel reduction processing using the fourth model to obtain the time consistency feature.

[0101] Specifically, in video camouflaged object detection (VCOD), the target object is often occluded or blurred due to the influence of motion, resulting in unnatural phenomena such as flickering and jitter between frames. Different from the existing spatiotemporal information modeling methods without segmentation, we propose a novel inter-frame temporal consistency module (ITCM) to effectively learn temporal consistency. This module captures the strong temporal correlation between adjacent frames, resulting in smoother and more consistent detection results. For each spatially enhanced feature map obtained previously First align the time dimension by rearranging the feature tensor: For two adjacent frames and Where 1, C, H and W represent the number of frames, the number of channels, the height and width of the feature map, respectively. Each frame undergoes feature conversion through CBR. The converted features are concatenated along the channel dimension to preserve the temporal information between frames, thus obtaining the feature map. Next, the time shift operation is applied to the concatenated features along the width dimension. The shifted features are then combined with the concatenated temporal information. Subtract them, and then use another CBR to reduce the channel to get the time difference information This information captures the temporal motion cues of the camouflaged target: Where C represents the concatenation operation and TS represents the time shift operation.

[0102] Next, CBR is applied to the time difference information Thus, the feature map after channel reduction is obtained Then, After adaptive average pooling, another 3×3 convolution and sigmoid activation function, the time difference attention weight is obtained. It can be expressed as: Where Sig represents the sigmoid activation function and AAvgP represents the adaptive average pooling operation. Then respectively with the frame and These features are concatenated to obtain adaptive inter-frame temporal consistency information, and then channel reduction is performed through another CBR. Finally, the Rea operation restores the feature map to its original shape to generate the final temporal consistency feature. The process is as follows:

[0103]

[0104] like Figure 4 As shown, the extracted features are sequentially subjected to contextual spatiotemporal aggregation processing, including:

[0105] S19: dividing the features to obtain bottom-level features and high-level features with different scales;

[0106] S20: upsampling the high-level features so that the scales of the low-level features and the high-level features are the same;

[0107] S21: using the fifth model to perform feature transformation, splicing and channel compression processing on the underlying features and high-level features to obtain contextual spatiotemporal attention features corresponding to each frame of the image;

[0108] S22: Calculate the contextual spatiotemporal aggregation features based on the contextual spatiotemporal attention features.

[0109] The step of calculating the contextual spatiotemporal aggregation feature based on the contextual spatiotemporal attention feature includes:

[0110] S23: multiplying the contextual spatiotemporal attention features by the corresponding underlying features and high-level features element by element;

[0111] S24: The multiplication results are concatenated and input into the sixth model for channel compression processing to obtain the contextual spatiotemporal aggregation features.

[0112] For example, camouflaged objects present different shapes, sizes, and proportions due to different shooting angles and distances. In order to improve the robustness and generalization ability of the model in complex environments, it is crucial to integrate multi-scale contextual spatiotemporal features. Unlike other methods that rely on element-wise multiplication or simple concatenation, the contextual spatiotemporal aggregation module (CSAM) proposed in this embodiment adopts an adaptive attention method to fuse spatial and temporal features of different scales. Specifically, for underlying features with different scales, and high-level features Where B, C, H and W represent the batch size, number of channels, height and width of the feature map respectively. Next, firstly, high Upsample to F low Next, F low F high CBR is applied to both. These transformed features are then concatenated together and subjected to another CBR channel compression to obtain the concatenated features. Then, for F cat One branch applies adaptive average pooling to reduce the spatial dimension, followed by CBR transformation, and finally obtains contextual spatiotemporal attention features through sigmoid.

[0113] F′ st =Sig(CBR(AdAvg(F cat))) Where Up represents the upsampling operation. At this time, after obtaining the required data, F′ st and 1-F′ st F′ low Respectively with F′ low and F′ high Then, these context-adaptive features are connected and channel compressed through CBR to finally obtain the aggregated multi-scale contextual spatiotemporal features.

[0114] F cst =CBR(C(F st ×F low ,(1-F st )×F high )).

[0115] Furthermore, the method further comprises:

[0116] S25: Analyze and identify the frame images based on the intra-frame spatial positioning processing result, the inter-frame temporal consistency processing result and the context spatiotemporal aggregation processing result of each frame image, and then determine the camouflaged object in each frame image.

[0117] Among them, the model in this embodiment usually uses weighted binary cross entropy loss in the binary image segmentation task and weighted intersection-over-union loss As a supervision function. Consistent with previous work, this combination and UAL are used as the main loss function L in the model of this embodiment. The loss function is defined as: Among them, p i ∈{0,1} represents the predicted value, g i ∈{0,1} represents the real value, i∈N represents the pixel index, and the parameter λ is the balance coefficient applied to the component. The overall cosine weighting scheme is adopted.

[0118] like Figure 5 As shown, another embodiment of the present invention also provides a video disguised object detection device 100, including:

[0119] An acquisition module, used for acquiring a video clip containing a camouflaged object;

[0120] A feature extraction module, used for extracting features from each frame of the video clip;

[0121] The processing module is used to sequentially perform intra-frame spatial positioning processing, inter-frame temporal consistency processing and contextual spatiotemporal aggregation processing on the extracted features to identify and determine the camouflaged objects existing in the frame image.

[0122] In some embodiments, the performing intra-frame spatial positioning processing on the extracted features includes:

[0123] Perform feature tensor adjustment on the extracted features to align the time dimension of each feature;

[0124] The adjusted features are divided into two groups, and both are processed by convolution, normalization and the first activation function to obtain two groups of intra-group features;

[0125] Calculate the spatial attention weights of the features in the first group using the second activation function;

[0126] Calculate the product of the spatial attention weight and each feature in the second group of features to obtain enhanced spatial features corresponding to each frame of the image;

[0127] The enhanced spatial features are rearranged to restore to the original arrangement state, and a spatial enhancement feature map corresponding to each frame of the image is obtained.

[0128] In some embodiments, the step of sequentially performing inter-frame temporal consistency processing on the extracted features includes:

[0129] Performing feature tensor adjustment on the spatially enhanced feature map to align the time dimensions of each feature;

[0130] The adjusted features are subjected to feature conversion processing by the first model, so that the converted features are spliced ​​along the channel dimension in the processing process to form a feature map that retains the time information between frames;

[0131] Performing time shift processing on the features spliced ​​along the width dimension in the feature graph;

[0132] Calculate the difference between the time information of the offset feature and the splicing time information of the feature map;

[0133] Using the second model to perform channel reduction processing on the difference to obtain time difference information, wherein the time difference information describes the motion state of the camouflaged target moving over time;

[0134] A time consistency feature corresponding to the frame image is calculated based on the time difference information.

[0135] In some embodiments, the calculating based on the time difference information to obtain the time consistency feature corresponding to the frame image includes:

[0136] Processing the time difference information using the third model to obtain a channel reduction feature map;

[0137] Adaptively average pooling, convolution, and mapping the channel-reduced feature map to obtain a temporal difference attention weight;

[0138] Multiplying the temporal difference attention weight by each feature in two adjacent frames of images;

[0139] The multiplication results are subjected to feature concatenation, and the fourth model is used to perform channel reduction processing to obtain the time consistency feature.

[0140] In some embodiments, the sequentially performing contextual spatiotemporal aggregation processing on the extracted features includes:

[0141] The features are divided to obtain low-level features and high-level features with different scales;

[0142] Upsampling the high-level features so that the scales of the low-level features and the high-level features are the same;

[0143] Using the fifth model to perform feature transformation, splicing and channel compression processing on the underlying features and high-level features, to obtain contextual spatiotemporal attention features corresponding to each frame of the image;

[0144] The contextual spatiotemporal aggregation feature is calculated based on the contextual spatiotemporal attention feature.

[0145] In some embodiments, the calculating of the contextual spatiotemporal aggregation feature based on the contextual spatiotemporal attention feature includes:

[0146] Multiply the contextual spatiotemporal attention features by the corresponding underlying features and high-level features element by element;

[0147] The multiplication results are concatenated and input into the sixth model for channel compression processing to obtain the contextual spatiotemporal aggregation features.

[0148] In some embodiments, the apparatus further comprises:

[0149] The recognition module is used to analyze and recognize the frame image according to the intra-frame spatial positioning processing result, the inter-frame temporal consistency processing result and the context spatiotemporal aggregation processing result of each frame image, and then determine the camouflaged object in each frame image.

[0150] Another embodiment of the present invention further provides an electronic device, comprising:

[0151] one or more processors;

[0152] a memory configured to store one or more programs;

[0153] When the one or more programs are executed by the one or more processors, the one or more processors implement the video disguised object detection method as described in any one of the above embodiments.

[0154] Furthermore, an embodiment of the present invention also provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the video disguised object detection method described above is implemented. It should be understood that each solution in this embodiment has the corresponding technical effect in the above method embodiment, which will not be repeated here.

[0155] Furthermore, an embodiment of the present invention also provides a computer program product, which is tangibly stored on a computer-readable medium and includes computer-readable instructions. When the computer-executable instructions are executed, at least one processor executes the video disguised object detection method in the embodiment described above.

[0156] It should be noted that the computer storage medium of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access storage medium (RAM), a read-only storage medium (ROM), an erasable programmable read-only storage medium (EPROM or flash memory), an optical fiber, a portable compact disk read-only storage medium (CD-ROM), an optical storage medium, a magnetic storage medium, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can send, propagate or transmit a program configured to be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, antenna, optical cable, RF, etc., or any suitable combination of the above.

[0157] In addition, it will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0158] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A system that specifies the functions of a box or multiple boxes.

[0159] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction system, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0160] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0161] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

[0162] One or more embodiments of the present application are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the present application. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application should be included in the protection scope of the present application.

Claims

1. A video disguised object detection method, characterized in that: include: obtaining a video clip containing a camouflaged object; Performing feature extraction on each frame of the video clip; The extracted features are sequentially processed with intra-frame spatial positioning, inter-frame temporal consistency, and contextual spatiotemporal aggregation to identify and determine the camouflaged objects present in the frame image.

2. The video disguised object detection method according to claim 1, characterized in that: The performing intra-frame spatial positioning processing on the extracted features includes: Perform feature tensor adjustment on the extracted features to align the time dimension of each feature; The adjusted features are divided into two groups, and both are processed by convolution, normalization and the first activation function to obtain two groups of intra-group features; Calculate the spatial attention weights of the features in the first group using the second activation function; Calculate the product of the spatial attention weight and each feature in the second group of features to obtain enhanced spatial features corresponding to each frame of the image; The enhanced spatial features are rearranged to restore to the original arrangement state, and a spatial enhancement feature map corresponding to each frame of the image is obtained.

3. The video disguised object detection method according to claim 1, characterized in that: The step of sequentially performing inter-frame temporal consistency processing on the extracted features includes: Performing feature tensor adjustment on the spatially enhanced feature map to align the time dimensions of each feature; The adjusted features are subjected to feature conversion processing by the first model, so that the converted features are spliced ​​along the channel dimension in the processing process to form a feature map that retains the time information between frames; Performing time shift processing on the features spliced ​​along the width dimension in the feature graph; Calculate the difference between the time information of the offset feature and the splicing time information of the feature map; Using the second model to perform channel reduction processing on the difference to obtain time difference information, wherein the time difference information describes the motion state of the camouflaged target moving over time; A time consistency feature corresponding to the frame image is calculated based on the time difference information.

4. The video disguised object detection method according to claim 3, characterized in that: The calculating based on the time difference information to obtain the time consistency feature corresponding to the frame image includes: Processing the time difference information using the third model to obtain a channel reduction feature map; Adaptively average pooling, convolution, and mapping the channel-reduced feature map to obtain a temporal difference attention weight; Multiplying the temporal difference attention weight by each feature in two adjacent frames of images; The multiplication results are subjected to feature concatenation, and the fourth model is used to perform channel reduction processing to obtain the time consistency feature.

5. The video disguised object detection method according to claim 1, characterized in that: The extracting features are sequentially subjected to contextual spatiotemporal aggregation processing, including: The features are divided to obtain low-level features and high-level features with different scales; Upsampling the high-level features so that the scales of the low-level features and the high-level features are the same; Using the fifth model to perform feature transformation, splicing and channel compression processing on the underlying features and high-level features, to obtain contextual spatiotemporal attention features corresponding to each frame of the image; The contextual spatiotemporal aggregation feature is calculated based on the contextual spatiotemporal attention feature.

6. The video disguised object detection method according to claim 5, characterized in that: The calculating of the context spatiotemporal aggregation feature based on the context spatiotemporal attention feature includes: Multiply the contextual spatiotemporal attention features by the corresponding underlying features and high-level features element by element; The multiplication results are concatenated and input into the sixth model for channel compression processing to obtain the contextual spatiotemporal aggregation features.

7. The video disguised object detection method according to claim 1, characterized in that: The method further comprises: Based on the intra-frame spatial positioning processing results, inter-frame temporal consistency processing results and contextual spatiotemporal aggregation processing results of each frame image, the frame image is analyzed and identified, and then the camouflaged object in each frame image is determined.

8. A video disguised object detection device, characterized in that: include: An acquisition module, used for acquiring a video clip containing a camouflaged object; A first processing module, configured to process each frame of the video clip to generate a corresponding binary pixel mask; A feature extraction module, used for extracting features from each of the binary pixel masks; The second processing module is used to perform intra-frame spatial positioning processing, inter-frame temporal consistency processing and context spatiotemporal aggregation processing on the extracted features in sequence, so as to identify and determine the camouflaged objects existing in the frame image.

9. The video disguised object detection device according to claim 8, characterized in that: The performing intra-frame spatial positioning processing on the extracted features includes: Perform feature tensor adjustment on the extracted features to align the time dimension of each feature; The adjusted features are divided into two groups, and both are processed by convolution, normalization and the first activation function to obtain two groups of intra-group features; Calculate the spatial attention weights of the features in the first group using the second activation function; Calculate the product of the spatial attention weight and each feature in the second group of features to obtain enhanced spatial features corresponding to each frame of the image; The enhanced spatial features are rearranged to restore to the original arrangement state, and a spatial enhancement feature map corresponding to each frame of the image is obtained.

10. An electronic device, characterized in that: include: one or more processors; a memory configured to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the video disguised object detection method according to any one of claims 1 to 7.

Images