A target detection method, device, equipment and readable storage medium

Abstract

The application discloses the technical field of computer vision, and particularly discloses a target detection method, device and equipment and a readable storage medium, the method comprising: acquiring a depth image corresponding to each original image in a target video to be detected; extracting appearance features from the original image and extracting depth features from the depth image; using target depth information in the depth features to enhance target position information in the corresponding appearance features to obtain depth information guided features; and combining the appearance features, the depth features and the depth information guided features to determine a target detection result. In the application, the depth information is used to compensate for the problem that a disguised target appearance is difficult to distinguish from the background, so that the disguised target in the video can be accurately detected from the background, and the reliability of disguised target detection can be improved.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a target detection method, apparatus, device, and readable storage medium. Background Technology

[0002] Video camouflage target detection aims to detect hidden or disguised targets in video scenes. These targets can camouflage themselves by mimicking the same body color, patterns, and other morphological appearance as the background. Because the targets are camouflaged, the boundary contrast between them and their surroundings is very low, making detection extremely difficult. Furthermore, camouflaged targets are mostly wild animals, and their appearances (e.g., size and shape) often vary, further increasing the difficulty of accurate detection. Video camouflage target detection technology has numerous potential applications, including agriculture (such as disaster detection and locust detection), computer-aided search and rescue, and medical imaging (such as lung infection diagnosis, retinal image segmentation, and polyp detection).

[0003] To address these challenges, deep learning techniques have been applied to video camouflage target detection in recent years, demonstrating significant potential. These methods typically learn the appearance features of RGB images in a video to guide the model in detecting camouflage targets. For example, they improve detection accuracy by detecting the edges between the camouflage target and the background, or by distinguishing the different textures and semantic features of the camouflage target and the background.

[0004] However, in camouflage target detection, the target to be detected is often difficult to distinguish from the background objects in appearance. For example, in complex situations where the camouflaged target is highly similar in appearance to the background, such as a dead leaf butterfly resting on a tree trunk or an Arctic fox standing in the snow, deep learning technology still suffers from insufficient reliability.

[0005] In conclusion, how to effectively solve the reliability problem of camouflaged target detection is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0006] The purpose of this application is to provide a target detection method, apparatus, device, and readable storage medium. This application simultaneously extracts the appearance features and depth features of video images, and uses the depth information of the target as a guide to compensate for the problem that the appearance of a disguised target is difficult to distinguish from the background, thereby accurately detecting the disguised target in the video from the background.

[0007] To solve the above-mentioned technical problems, this application provides the following technical solution:

[0008] A target detection method, comprising:

[0009] Obtain the depth images corresponding to each original image in the target video to be detected;

[0010] Extract appearance features from the original image and depth features from the depth image;

[0011] By utilizing the target depth information in the depth features, the target position information in the corresponding appearance features is enhanced to obtain depth information guiding features;

[0012] The target detection result is determined by combining the appearance features, the depth features, and the depth information guidance features.

[0013] Preferably, after extracting appearance features from the original image and depth features from the depth image, the method further includes:

[0014] The appearance features and depth features are subjected to channel information enhancement processing and / or contextual semantic enhancement processing.

[0015] Preferably, extracting appearance features from the original image and extracting depth features from the depth image includes:

[0016] The original image and its depth image are used as a set of inputs. A dual-branch feature extraction network is used to extract the appearance features from the original image and the depth features from the depth image.

[0017] Preferably, the target depth information in the depth features is used to enhance the target position information in the corresponding appearance features to obtain depth information guiding features, including:

[0018] Spatial attention is used to manipulate the spatial importance weight matrix of the appearance features and the depth features;

[0019] By utilizing the target depth information in the depth features, the target position information of the appearance features in the spatial importance weight matrix is ​​enhanced to obtain the depth information guiding features.

[0020] Preferably, the target detection result is determined by combining the appearance features, the depth features, and the depth information guidance features, including:

[0021] The low-resolution feature and the high-resolution feature are aggregated from the appearance feature, the depth feature and the depth information guiding feature to obtain the aggregated feature;

[0022] The target detection result is determined using the aggregated features.

[0023] Preferably, determining the target detection result using the aggregated features includes:

[0024] The aggregated features are upsampled step by step and then concatenated with the aggregated features from the previous stage in pairs to gradually restore the feature size to the original image size, thereby obtaining the target detection result.

[0025] Preferably, it further includes:

[0026] Based on the target detection results, a segmented video target mask image is output.

[0027] A target detection device, comprising:

[0028] The input processing module is used to acquire the depth images corresponding to each original image in the target video to be detected;

[0029] The feature extraction module is used to extract appearance features from the original image and depth features from the depth image;

[0030] A depth information guidance module is used to enhance the target position information in the corresponding appearance feature by utilizing the target depth information in the depth feature to obtain a depth information guidance feature.

[0031] The feature aggregation module and the step-by-step upsampling module are used to combine the appearance features, the depth features, and the depth information guiding features to determine the target detection result.

[0032] An electronic device, comprising:

[0033] Memory, used to store computer programs;

[0034] A processor is used to implement the steps of the target detection method described above when executing the computer program.

[0035] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the target detection method described above.

[0036] By applying the method provided in the embodiments of this application, depth images corresponding to each original image in the target video to be detected are obtained; appearance features are extracted from the original images, and depth features are extracted from the depth images; target depth information in the depth features is used to enhance the target position information in the corresponding appearance features to obtain depth information guiding features; and the target detection result is determined by combining appearance features, depth features, and depth information guiding features.

[0037] In this application, after acquiring the video to be detected, the depth images corresponding to each original image in the target video are first acquired. Then, appearance features are extracted from the original images, and depth features are extracted from the depth images. Although a camouflaged target can resemble the background in appearance, it cannot conceal its spatial position; that is, the depth information of a camouflaged target must be distinct from the regular background. Therefore, by utilizing the target depth information in the depth features, the target position information in the corresponding appearance features is enhanced to obtain depth information-guided features. Then, by combining appearance features, depth features, and depth information-guided features, the target detection result is determined. In other words, this application uses depth information to compensate for the problem that the appearance of a camouflaged target is difficult to distinguish from the background, thereby accurately detecting camouflaged targets in the video from the background and improving the reliability of camouflaged target detection.

[0038] Accordingly, embodiments of this application also provide target detection apparatus, devices, and readable storage media corresponding to the above-described target detection method, which have the aforementioned technical effects, and will not be elaborated further here. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a flowchart illustrating the implementation of a target detection method in an embodiment of this application.

[0041] Figure 2 This is a schematic diagram of a video-based camouflaged target detection model in an embodiment of this application;

[0042] Figure 3 This is a schematic diagram of a feature enhancement module in an embodiment of this application;

[0043] Figure 4 This is a schematic diagram of a depth information guidance module in an embodiment of this application;

[0044] Figure 5 This is a schematic diagram of a feature aggregation module in an embodiment of this application;

[0045] Figure 6 This is a schematic diagram of a step-by-step upsampling module in an embodiment of this application;

[0046] Figure 7 This is a schematic diagram of the structure of a target detection device according to an embodiment of this application;

[0047] Figure 8 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0048] Figure 9 This is a schematic diagram of the specific structure of an electronic device in an embodiment of this application. Detailed Implementation

[0049] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0050] Please refer to Figure 1 , Figure 1 This is a flowchart of a target detection method according to an embodiment of this application. The method includes the following steps:

[0051] S101. Obtain the depth images corresponding to each original image in the target video to be detected.

[0052] The target video can be any video that needs to be detected. The video can be captured by a monitoring device, obtained through network means such as downloading / receiving, or read directly from a storage device. This application does not limit the acquisition method or the content of the target video.

[0053] In this application, each frame of the RGB image in the target video is referred to as the original image, and the corresponding image generated from the original image using a depth estimation method is referred to as the depth image.

[0054] Specifically, in this application, depth images corresponding to the original images can be generated through methods such as depth estimation. For example, each original image in the target video can be input into a depth estimation model for depth estimation processing to obtain the corresponding depth image. This application does not limit the specific depth estimation method / model used, as long as it can obtain the depth image of the original image.

[0055] S102. Extract appearance features from the original image and extract depth features from the depth image.

[0056] When extracting features, not only are appearance features extracted from the original image, but depth features are also extracted from the depth image.

[0057] Among them, appearance features can be specifically the RGB features of the original image, while depth features are features that indicate the depth information in the original image.

[0058] In this application, the extraction of appearance features and depth features can be performed separately or in parallel. For example, appearance features can be extracted from the original image first, and then depth features can be extracted from the depth image; or depth features can be extracted from the depth image first, and then appearance features can be extracted from the original image; or appearance features can be extracted from the original image and depth features can be extracted from the depth image simultaneously.

[0059] The feature extraction network corresponding to the extracted features can refer to conventional feature extraction networks. For example, convolutional neural networks can be used to extract features, which will not be elaborated on here.

[0060] In one specific embodiment of this application, extracting appearance features from the original image and depth features from the depth image includes: taking the original image and its depth image as a set of inputs, and using a dual-branch feature extraction network to extract appearance features from the original image and depth features from the depth image. That is, for ease of processing, the original image and its depth image can be taken as a set of inputs, and then the appearance features and depth features can be extracted separately based on the dual-branch extraction network.

[0061] For example, the RGB image of the video (i.e., the original image) and its corresponding depth image can be used as a set of inputs, respectively input into two feature extraction branches. Each branch undergoes four stages of feature extraction, outputting appearance features in descending resolution: F1. rgb F2 rgb F3 rgb F4 rgb , and depth features: F1 depth F2 depth F3 depth F4 depth .

[0062] In one specific embodiment of this application, after extracting appearance features from the original image and depth features from the depth image, channel information enhancement processing and / or contextual semantic enhancement processing can be performed on the appearance features and depth features. Here, "and / or" means that channel information enhancement processing can be performed only on appearance features and depth information, or contextual semantic enhancement processing can be performed only on appearance features and depth information, or both channel information enhancement processing and contextual semantic enhancement processing can be performed on appearance features and depth information.

[0063] The above steps will be explained in detail below using channel information enhancement processing and contextual semantic enhancement processing.

[0064] Can be constructed as Figure 3 The Feature Enhancement Module (PEM) shown includes channel information enhancement and contextual semantic enhancement. The extracted four sets of RGB features F... i rgbWith depth features F i depth Four sets of feature enhancement modules are input, where i∈[1,4]. In the feature enhancement modules, firstly, the channel information enhancement module enhances the channel information of the features according to the channel importance weights of each feature; then, the context semantic enhancement module performs multi-scale convolution operations using parallel 1×1, 3×3 (with a dilation rate of 6), and 3×3 (with a dilation rate of 12) convolutional kernels to enhance the context semantics of the features, resulting in four enhanced sets of RGB features F. i re With depth features F i de .

[0065] Furthermore, the feature enhancement module's function is to enhance the four sets of appearance features F1 at different resolutions obtained from the dual-stream feature backbone network (dual-branch feature extraction network). rgb F2 rgb F3 rgb F4 rgb F1 with depth features depth F2 depth F3 depth F4 depth Feature enhancement is performed separately, and the specific process is as follows:

[0066] Step B1: Analyze the appearance features F obtained from the two-stream network at stage i. i rgb Each channel is used as input, and enhancement is performed at the channel level. After global pooling (GAP) and fully connected (FC) operations, the channel importance weights are obtained through an activation function, and then combined with F... i rgb Dot product yields the enhanced features. When the input is a deep feature F i depth Then, perform the same operation as above. The specific formula is shown below:

[0067]

[0068]

[0069] Here, GAP represents global pooling operation, and FC represents fully connected operation. This represents the activation function. This is an element-wise multiplication operation, where i∈[1,4].

[0070] Step B2: Convolution operations are performed using parallel 1×1, 3×3 (with a dilation rate of 6), and 3×3 (with a dilation rate of 12) kernels to extract features at multiple scales and enhance the contextual semantics of the features, resulting in the enhanced RGB features F. i re With depth features F i de The specific formula is as follows:

[0071]

[0072]

[0073] in, is element-wise multiplication, concat is concatenation, conv1, conv2 and conv3 are convolutions of size 1×1, 3×3 with a dilation of 6 and 3×3 with a dilation of 12 respectively.

[0074] S103. Utilize the target depth information in the depth features to enhance the target position information in the corresponding appearance features, thereby obtaining depth information-guided features.

[0075] Because camouflaged targets cannot completely conceal their actual bodies, a depth difference is created between them and the real background in the image. For example, an Arctic fox standing in the snow is a real animal, and although its color is similar to the snow, light reflects off the fox, creating a depth difference between it and the snow. This depth difference can be used to help determine the fox's location.

[0076] Therefore, in this embodiment of the application, the target depth information in the depth features will be used to enhance the target position information in the corresponding appearance features, thereby obtaining depth information guiding features.

[0077] In other words, based on the target depth information corresponding to the target in the depth features, the target position information corresponding to the appearance features is enhanced, so that the depth information guides the features to point to the target based on both appearance and depth.

[0078] In one specific embodiment of this application, target depth information in the corresponding appearance features is enhanced using target position information in the depth features to obtain depth information guiding features, including:

[0079] Step 1: Manipulate the spatial importance weight matrix of appearance and depth features using spatial attention;

[0080] Step 2: Utilize the target depth information in the depth features to enhance the target location information of the appearance features in the spatial importance weight matrix, thereby obtaining depth information-guided features.

[0081] For ease of description, the two steps above will be explained together below.

[0082] In computer vision, methods that focus attention on important regions of an image while discarding irrelevant ones are called attention mechanisms. Attention mechanisms can be viewed as a dynamic selection process for important information from the image input.

[0083] First, the spatial importance weight matrix of appearance features and depth features is manipulated based on spatial attention. Then, the target depth information in the depth features is used to enhance the target position information of appearance features in the spatial importance weight matrix, thereby obtaining depth information-guided features.

[0084] In other words, in this embodiment, the depth information contained in the depth features can be used to guide the detection of camouflaged targets in a video RGB image. Specifically, the extracted appearance features and depth features are used as inputs to each depth information guidance module; spatial attention operations are used to obtain the spatial importance weight matrix of the appearance features and depth features; and the target position information in the appearance features is continuously enhanced using the target depth information contained in the depth features to obtain the depth information-guided feature F. i g (i.e., depth information guided features).

[0085] Specifically, it can be constructed as follows Figure 4 The depth information guidance module shown above is used to guide the detection of the location of a camouflaged target based on depth information. The depth features contain depth information about the camouflaged target. When the location of a camouflaged target is difficult to distinguish by appearance, this module utilizes the depth information contained in the depth features to guide the detection of the camouflaged target's location. The specific process is as follows:

[0086] Step C1: Process the four sets of RGB features F i re With depth features F i de The components undergo parallel average pooling and max pooling operations, are then concatenated along the channels, and finally subjected to 1×1 convolution and activation functions to obtain the spatial importance weight matrix M. i rgb With M i depth The specific formula is as follows:

[0087]

[0088]

[0089] in, The 'concat' option represents the activation function, 'conv1' represents a 1×1 convolution kernel, 'concat' represents concatenation, 'AvgPool' represents average pooling, and 'MaxPool' represents max pooling.

[0090] Step C2: Process the four sets of features F obtained i re Feature F is obtained from multi-scale features. i depth The features are obtained by concatenating along the channel dimension and performing element-wise multiplication with the spatial importance weight matrix of the depth features. This is then combined with the spatial importance weight matrix of the RGB features to obtain the final features. Finally, the features are summed. The specific formula is as follows:

[0091]

[0092] Here, concat means concatenation. Indicates element-wise multiplication. This indicates element addition.

[0093] S104. Combine appearance features, depth features, and depth information-guided features to determine the target detection result.

[0094] After obtaining the depth-guided features, the target detection result can be determined by combining appearance features, depth features, and depth-guided features. Because this target detection result is determined based on the combination of appearance features, depth features, and depth-guided features, it is more accurate and reliable.

[0095] In one specific embodiment of this application, a segmented video target mask image is also output based on the target detection results.

[0096] In one specific embodiment of this application, the target detection result is determined by combining appearance features, depth features, and depth information-guided features, including:

[0097] Step 1: In the appearance features, depth features, and depth information guiding features, low-resolution features and high-resolution features are aggregated to obtain aggregated features;

[0098] Specifically, the feature F with resolution from high to low i g , i∈[1,4], with F1 g With F2 g F2 g With F3 g F3 g With F4 g Grouping serves as input to three Feature Aggregation Modules (FAM); low-resolution features are aggregated with high-resolution features to obtain the aggregated feature F. ia , i∈[1,3].

[0099] Step 2: Use aggregated features to determine the target detection results.

[0100] Specifically, the aggregated features can be upsampled step by step and then concatenated with the aggregated features from the previous stage in pairs to gradually restore the feature size to the original image size, thereby obtaining the target detection result.

[0101] The resulting aggregated features F are processed step by step. i a Upsampling, and aggregating features F with those from the previous stage i-1 a By concatenating the features in pairs, the feature size is gradually restored to the original image size, and finally the output with the same resolution as the original is obtained, i∈[1,3].

[0102] Furthermore, it is possible to create, such as Figure 5 The feature aggregation module shown is responsible for aggregating low-resolution features containing rich semantic information with high-resolution features containing rich appearance and texture information. First, the feature aggregation module aggregates the low-resolution features F... i g Upsampled after 1×1 convolution, and compared with the high-resolution feature F after 1×1 convolution. i-1 g Perform element-wise multiplication to obtain the characteristics. The specific formula is as follows:

[0103]

[0104] Where conv1 represents a 1×1 convolution kernel, For element-wise multiplication, Upsample is for upsampling.

[0105] Then feature F i-1 g , With F i g After convolution, the data is concatenated. The specific formula is as follows:

[0106]

[0107] Here, concat means concatenation, and conv1 means 1×1 convolution.

[0108] Furthermore, it is possible to construct, such as Figure 6 The step-by-step upsampling module (or cascaded upsampling module) shown above obtains the aggregated feature F. i a Upsampling, and aggregating features F with those from the previous stage i-1a Feature O is obtained by splicing two pairs together. i-1 The feature size is gradually restored to the original image size, ultimately yielding an output with the same resolution as the original; the final output is a segmented video target mask image, specifically calculated using the following formula:

[0109] O i-1 =conv1(concat(upsample(F i a ),F i-1 a )),

[0110] Here, upsample(·) represents upsampling, and concat(·) represents concatenation.

[0111] By applying the method provided in the embodiments of this application, depth images corresponding to each original image in the target video to be detected are obtained; appearance features are extracted from the original images, and depth features are extracted from the depth images; target depth information in the depth features is used to enhance the target position information in the corresponding appearance features to obtain depth information guiding features; and the target detection result is determined by combining appearance features, depth features, and depth information guiding features.

[0112] In this application, after acquiring the video to be detected, the depth images corresponding to each original image in the target video are first acquired. Then, appearance features are extracted from the original images, and depth features are extracted from the depth images. Although a camouflaged target can resemble the background in appearance, it cannot conceal its spatial position; that is, the depth information of a camouflaged target must be distinct from the regular background. Therefore, by utilizing the target depth information in the depth features, the target position information in the corresponding appearance features is enhanced to obtain depth information-guided features. Then, by combining appearance features, depth features, and depth information-guided features, the target detection result is determined. In other words, this application uses depth information to compensate for the problem that the appearance of a camouflaged target is difficult to distinguish from the background, thereby accurately detecting camouflaged targets in the video from the background and improving the reliability of camouflaged target detection.

[0113] To facilitate those skilled in the art to better understand the target detection method provided in the embodiments of this application, the target detection method will be described in detail below with specific application scenarios as examples.

[0114] In practical applications, based on the above object detection method, a video camouflage object detection model is constructed and trained, which enables the model to perform object detection in video by incorporating depth information, thereby improving detection reliability.

[0115] Specifically, the overall architecture of this model can be found by referring to... Figure 2 , Figure 2This is a schematic diagram of a video-based camouflage target detection model in an embodiment of this application.

[0116] First, obtain a video camouflage target dataset containing the target's binary mask (same mask), and generate corresponding depth images from the RGB images of each frame of the video using a depth estimation method. Second, perform steps A to F to obtain the constructed video camouflage target detection model. Finally, apply this video camouflage target detection model to obtain the detection results of the camouflage targets in the video.

[0117] Step A: Construct a two-branch feature extraction network comprising four stages. The video RGB image and its corresponding depth image are used as a set of inputs, respectively fed into two feature extraction branches. Each branch undergoes four stages of feature extraction, outputting the appearance features F1 from high to low resolution. rgb F2 rgb F3 rgb F4 rgb F1 with depth features depth F2 depth F3 depth F4 depth .

[0118] Step B: Construct a feature enhancement module (e.g.) Figure 3 As shown), this includes channel information enhancement and contextual semantic enhancement. The four sets of RGB features F output from step A are... i rgb With depth features F i depth Four sets of feature enhancement modules are input, where i∈[1,4]. In the feature enhancement modules, firstly, the channel information enhancement module enhances the channel information of the features according to the channel importance weights of each feature; then, the context semantic enhancement module performs multi-scale extraction of the features through parallel convolution operations with 1×1, 3×3 (with a dilation rate of 6), and 3×3 (with a dilation rate of 12) convolution kernels, enhancing the context semantics of the features, resulting in four enhanced sets of RGB features F. i re With depth features F i de .

[0119] That is, this feature enhancement module uses a residual structure to extract features by concatenating convolutions of multiple sizes in parallel. Specifically, for the detailed steps of step B, please refer to steps B1 and B2 above, which will not be repeated here.

[0120] Step C: Construct the Deep Information Guidance Module (DGM) (e.g.) Figure 4As shown in the diagram, the depth information contained in the depth features guides the model to detect camouflaged targets in the RGB images of the video. In the depth information guidance module, the four sets of RGB features F obtained in step B are used... i re With depth features F i de As input to each depth information guidance module, where i∈[1,4]; RGB features F are obtained using spatial attention operations. i re With depth features F i de Spatial importance weight matrix, utilizing deep feature F i de The target depth information contained in the RGB features continuously enhances the target location information in the RGB features, resulting in a depth-guided feature F. i g .

[0121] That is, this feature enhancement module uses a residual structure to extract features by concatenating convolutions of multiple sizes in parallel.

[0122] Specifically, for the detailed steps of step C, please refer to steps C1 and C2 above, which will not be repeated here.

[0123] Step D: Construct a feature aggregation module (e.g., Figure 5 As shown), the feature aggregation module uses the features F obtained in step C, ranked from high to low resolution. i g , i∈[1,4], with F1 g With F2 g F2 g With F3 g F3 g With F4 g Grouping serves as the input to three feature aggregation modules; low-resolution features are aggregated with high-resolution features to obtain the aggregated feature F. i a , i∈[1,3].

[0124] Specifically, for the detailed steps of step D, please refer to the relevant description of feature aggregation above, which will not be repeated here.

[0125] Step E: Construct the stepwise upsampling module (LUM) (e.g.) Figure 6 As shown), this step-by-step upsampling module progressively upsamples the aggregated features F obtained in step D. i a Upsampling, and aggregating features F with those from the previous stage i-1 aBy concatenating the features in pairs, the feature size is gradually restored to the original image size, and finally the output with the same resolution as the original is obtained, i∈[1,3].

[0126] Specifically, for the detailed steps of step E, please refer to the relevant description of feature upsampling mentioned above, which will not be repeated here.

[0127] Step F: Construct a complete video camouflage target detection model based on steps A to E. Use the video RGB image and the corresponding depth image as a set of inputs, and the binary mask for camouflage target detection in the video image as the output for training, and finally obtain the trained video camouflage target detection model.

[0128] Through experimental verification, the target detection method provided in the embodiments of this application has been used to visualize the detection results of some examples. The results show that camouflaged target objects such as Arctic foxes, moles, dead leaf butterflies, and cheetahs have been segmented from the video image frames.

[0129] Corresponding to the above method embodiments, this application also provides a target detection device, and the target detection device described below can be referred to in correspondence with the target detection method described above.

[0130] See Figure 7 As shown, the device includes the following modules:

[0131] The input processing module 101 is used to acquire the depth images corresponding to each original image in the target video to be detected;

[0132] Feature extraction module 102 is used to extract appearance features from the original image and depth features from the depth image;

[0133] The depth information guidance module 103 is used to enhance the target position information in the corresponding appearance features by utilizing the target depth information in the depth features, so as to obtain the depth information guidance features.

[0134] The feature aggregation module 104 and the stepwise upsampling module 105 are used to combine appearance features, depth features and depth information-guided features to determine the target detection result.

[0135] Using the apparatus provided in the embodiments of this application, depth images corresponding to each original image in the target video to be detected are obtained; appearance features are extracted from the original images, and depth features are extracted from the depth images; target depth information in the depth features is used to enhance the target position information in the corresponding appearance features to obtain depth information guiding features; and the target detection result is determined by combining the appearance features, depth features, and depth information guiding features.

[0136] In this application, after acquiring the video to be detected, the depth images corresponding to each original image in the target video are first acquired. Then, appearance features are extracted from the original images, and depth features are extracted from the depth images. Although a camouflaged target can resemble the background in appearance, it cannot conceal its spatial position; that is, the depth information of a camouflaged target must be distinct from the regular background. Therefore, by utilizing the target depth information in the depth features, the target position information in the corresponding appearance features is enhanced to obtain depth information-guided features. Then, by combining appearance features, depth features, and depth information-guided features, the target detection result is determined. In other words, this application uses depth information to compensate for the problem that the appearance of a camouflaged target is difficult to distinguish from the background, thereby accurately detecting camouflaged targets in the video from the background and improving the reliability of camouflaged target detection.

[0137] In one specific embodiment of this application, the feature enhancement module is used to extract appearance features from the original image and depth features from the depth image, and then perform channel information enhancement processing and / or contextual semantic enhancement processing on the appearance features and depth features.

[0138] In one specific embodiment of this application, the feature extraction module 102 is specifically used to take the original image and the depth image of the original image as a set of inputs, and use a dual-branch feature extraction network to extract appearance features from the original image and depth features from the depth image.

[0139] In one specific embodiment of this application, the depth information guidance module 103 is specifically used to manipulate the spatial importance weight matrix of appearance features and depth features using spatial attention.

[0140] By utilizing the target depth information in the depth features, the target location information of the appearance features in the spatial importance weight matrix is ​​enhanced to obtain depth information-guided features.

[0141] In one specific embodiment of this application, the feature aggregation module 104 is specifically used to aggregate low-resolution features and high-resolution features among appearance features, depth features and depth information guide features to obtain aggregated features;

[0142] The stepwise upsampling module 105 is specifically used to determine the target detection result by utilizing aggregated features.

[0143] In one specific embodiment of this application, the step-by-step upsampling module 105 is specifically used to perform step-by-step upsampling on the aggregated features and splice them with the aggregated features of the previous stage in pairs to gradually restore the feature size to the original image size and obtain the target detection result.

[0144] In one specific embodiment of this application, it further includes:

[0145] The result output module is used to output segmented video target mask images based on the target detection results.

[0146] Corresponding to the above method embodiments, this application also provides an electronic device. The electronic device described below can be referred to in correspondence with the target detection method described above.

[0147] See Figure 8 As shown, the electronic device includes:

[0148] Memory 332 is used to store computer programs;

[0149] The processor 322 is used to implement the steps of the target detection method in the above method embodiments when executing a computer program.

[0150] For details, please refer to Figure 9 , Figure 9 This is a schematic diagram of a specific structure of an electronic device provided in this embodiment. The electronic device can vary significantly due to differences in configuration or performance. It may include one or more central processing units (CPUs) 322 (e.g., one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 can be temporary or permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 and execute the series of instruction operations stored in the memory 332 on the electronic device 301.

[0151] Electronic device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0152] The steps in the target detection method described above can be implemented by the structure of an electronic device.

[0153] Corresponding to the above method embodiments, this application also provides a readable storage medium. The readable storage medium described below can be referred to in conjunction with the target detection method described above.

[0154] A readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the target detection method described in the above method embodiments.

[0155] The readable storage medium can specifically be a USB flash drive, external hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or any other readable storage medium capable of storing program code.

[0156] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0157] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0158] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0159] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "include," "contain," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus.

[0160] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A target detection method, characterized in that, include: Obtain the depth images corresponding to each original image in the target video to be detected; The original image and its depth image are used as a set of inputs. A dual-branch feature extraction network is used to extract appearance features from the original image and depth features from the depth image. The appearance features and the depth features are respectively input into the feature enhancement module; In the feature enhancement module, the channel information enhancement module enhances the channel information of the features according to the channel importance weights of each feature. Specifically, after global pooling and fully connected operations, the channel importance weights are obtained through an activation function, and then multiplied by the feature to obtain the enhanced features. Then, the context semantic enhancement module performs convolution operations with convolution kernels with different dilation rates in parallel to extract the features at multiple scales and enhance the context semantics of the features to obtain the enhanced appearance features and the depth features. The spatial importance weight matrix is ​​obtained by enhancing the appearance features and depth features using spatial attention operations; wherein each feature is subjected to parallel average pooling and max pooling operations, then concatenated along the channels, and then subjected to convolution and activation functions to obtain the spatial importance weight matrix. By utilizing the target depth information in the depth features, the target position information of the appearance features in the spatial importance weight matrix is ​​enhanced to obtain depth information guiding features; The low-resolution features and high-resolution features among the appearance features, the depth features, and the depth information guiding features are aggregated to obtain aggregated features; The target detection result is determined using the aggregated features.

2. The target detection method according to claim 1, characterized in that, Using the aggregated features, the target detection result is determined, including: The aggregated features are upsampled step by step and then concatenated with the aggregated features from the previous stage in pairs to gradually restore the feature size to the original image size, thereby obtaining the target detection result.

3. The target detection method according to claim 1 or 2, characterized in that, Also includes: Based on the target detection results, a segmented video target mask image is output.

4. A target detection device, characterized in that, include: The input processing module is used to acquire the depth images corresponding to each original image in the target video to be detected; The feature extraction module is used to take the original image and the depth image of the original image as a set of inputs, and use a dual-branch feature extraction network to extract appearance features from the original image and depth features from the depth image. The feature enhancement module is used to receive the appearance features and the depth features; The channel information enhancement module enhances the channel information of the features based on the channel importance weights of each feature. Specifically, after global pooling and fully connected operations, the channel importance weights are obtained through an activation function, and then multiplied by the feature to obtain the enhanced features. The context semantic enhancement module then performs convolution operations with convolution kernels with different dilation rates in parallel to extract the features at multiple scales and enhance the context semantics of the features, resulting in the enhanced appearance features and depth features. The depth information guidance module is used to enhance the spatial importance weight matrix of the appearance features and the depth features using spatial attention operations. Each feature is subjected to parallel average pooling and max pooling operations, then concatenated along the channels, and then subjected to convolution and activation functions to obtain the spatial importance weight matrix. The target depth information in the depth features is used to enhance the target position information of the appearance features in the spatial importance weight matrix to obtain the depth information guidance feature. The feature aggregation module and the stepwise upsampling module are used to aggregate low-resolution features and high-resolution features from the appearance features, the depth features, and the depth information guiding features to obtain aggregated features; and to determine the target detection result using the aggregated features.

5. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the target detection method as described in any one of claims 1 to 3 when executing the computer program.

6. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the target detection method as described in any one of claims 1 to 3.

Images