Target detection method, device, electronic device and storage medium

Abstract

The present application discloses a target detection method, device, electronic device and storage medium. The target detection method comprises: extracting local features and global attention features of an image to be detected in a parallel interactive manner through a backbone network to obtain a graph feature pyramid of the image to be detected and a global semantic feature of the image to be detected; fusing the graph feature pyramid according to the global semantic feature to obtain a fusion feature of the image to be detected, wherein the fusion feature includes multi-scale information of the image to be detected; decoding the fusion feature to obtain a decoded feature of the image to be detected; obtaining a detection result of the image to be detected according to the decoded feature, and the detection result is used to determine whether there is a target object in the image to be detected, and when there is a target object in the image to be detected, determine the location information of the target object and the category of the target object. The present application can enhance the small target detection performance and improve the training convergence.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a target detection method, device, electronic device and storage medium. Background Art

[0002] Object detection has always been a basic but very important task in the field of computer vision, which aims to predict the bounding box of the object of interest in an image and its corresponding category.

[0003] At present, detectors in the field of target detection are mainly divided into two mainstream architectures: target detectors based on convolutional neural network architecture (CNN-based) and target detectors based on transformer architecture (Transformer-based).

[0004] CNN-based algorithms still need to use non-maximum suppression (NMS) to filter out a large number of redundant frames in the post-processing process, which will lead to computational bottlenecks, and the hyperparameters of NMS will have a great impact on the accuracy and speed of the detector. Therefore, CNN-based algorithms cannot be called true end-to-end detectors.

[0005] Compared with CNN-based object detectors, the introduction of DETR (Detection Transformer) abandons manual elements such as pre-selection box selection and NMS processing, thus enabling true end-to-end detection of image targets.

[0006] DETR is mainly divided into four parts. The first is to use a convolutional neural network as the backbone network to extract features, and then use a feedforward neural network for prediction after the encoder and decoder of the Transformer. DETR uses the attention mechanism in the Transformer to effectively model the long-distance feature dependencies in the image. DETR can accurately identify objects and their relationships, and can better implement the filtering function of redundant prediction boxes. However, DETR still has some problems, such as the difficulty in converging the attention mechanism, which leads to long training time, and DETR only encodes and decodes the last layer of feature maps of the convolutional neural network. Due to the lack of multi-scale feature information, the model has low accuracy in detecting small targets. Summary of the invention

[0007] In view of the above problems, the purpose of the present application is to provide a target detection method, device, electronic device and computer-readable storage medium to solve or at least alleviate the above problems.

[0008] According to a first aspect of the present application, a target detection method is provided, comprising: extracting local features and global attention features of an image to be detected in a parallel interactive manner through a backbone network to obtain a graph feature pyramid of the image to be detected and a global semantic feature of the image to be detected; performing feature fusion on the graph feature pyramid according to the global semantic feature to obtain a fused feature of the image to be detected, wherein the fused feature includes multi-scale information of the image to be detected; decoding the fused feature to obtain a decoded feature of the image to be detected; obtaining a detection result of the image to be detected according to the decoded feature, the detection result being used to determine whether a target object exists in the image to be detected, and determining the position information of the target object and the category of the target object when the target object exists in the image to be detected.

[0009] Preferably, the backbone network includes a first branch and a second branch, the first branch includes multiple cascaded convolutional neural network modules, which are used to extract local features of the image to be detected; the second branch includes multiple cascaded attention mechanism-based encoding modules, and the encoding modules are used to extract global attention features.

[0010] Preferably, through the backbone network, feature extraction is performed on the local features and global attention features of the image to be detected in a parallel interactive manner to obtain the graph feature pyramid of the image to be detected and the global semantic features of the image to be detected, including: directly inputting the image to be detected into the convolutional neural network module of the first level, and converting the image to be detected into an image slice sequence and inputting the image slice sequence into the encoding module of the first level; in the feature extraction process of multiple cascaded convolutional neural network modules and multiple cascaded encoding modules, the local features output by the convolutional neural network modules of each level are alternately interacted with the global attention features output by the encoding modules of each level; the local features output by the convolutional neural network modules of the last multiple levels are used as the graph feature pyramid of the image to be detected; and the global attention features output by the encoding module of the last level are used as the global semantic features of the image to be detected.

[0011] Preferably, alternating the local features output by the convolutional neural network module of each level with the global attention features output by the encoding module includes: merging the local features output by the convolutional neural network module of the previous level with the global attention features output by the encoding module of the previous level and inputting them into the encoding module of the current level; merging the global attention features output by the encoding module of the current level with the local features output by the convolutional neural network module of the current level and inputting them into the convolutional neural network module of the next level; and so on, alternating the local features output by the convolutional neural network module with the global attention features output by the encoding module.

[0012] Preferably, the local features output by the first-level convolutional neural network module are merged with the global attention features output by the first-level encoding module and then input into the second-level encoding module; or the global attention features output by the first-level encoding module and the local features output by the first-level convolutional neural network module are merged with the local features output by the first-level convolutional neural network module and then input into the second-level convolutional neural network module.

[0013] Preferably, the feature merging of the local features output by the convolutional neural network module and the global attention features output by the encoding module includes: performing convolution operations, average pooling operations, and layer normalization operations on the local features output by the convolutional neural network at the same level to align the local features with the global attention features output by the encoding module at the same level; superimposing the local features after feature alignment with the global attention features and inputting them into the encoding module at the next level; and the feature merging of the global attention features output by the encoding module with the local features output by the convolutional neural network module includes: performing interpolation operations, convolution operations, and batch normalization operations on the global attention features output by the encoding module at the same level to align the global attention features with the local features output by the convolutional neural network module at the same level; superimposing the local features after feature alignment with the global attention features and inputting them into the convolutional neural network module at the next level.

[0014] Preferably, the fusion feature of the image to be detected is obtained by performing feature fusion on the graph feature pyramid according to the global semantic feature, wherein the fusion feature includes multi-scale information of the image to be detected, including: expanding the graph feature pyramid into a one-dimensional vector and splicing it to obtain a splicing vector; fusing the splicing vector with the global semantic feature through a cross-attention layer and a feedforward neural network layer to obtain a fusion feature.

[0015] Preferably, decoding the fused features to obtain decoded features of the image to be detected includes: performing self-attention processing on the target query vector to obtain a first target query vector; performing cross-attention processing or deformable attention processing on the fused features and the first target query vector to map the fused features to the first target query vector to obtain decoded features.

[0016] According to a second aspect of the present application, a target detection device is provided, characterized in that it includes: a backbone network, which is used to extract features of local features and global attention features of an image to be detected in a parallel interactive manner to obtain a graph feature pyramid of the image to be detected and a global semantic feature of the image to be detected; a feature fusion module, which performs feature fusion on the graph feature pyramid according to the global semantic feature to obtain a fusion feature of the image to be detected, wherein the fusion feature includes multi-scale information of the image to be detected; a decoder, which is used to decode the fusion feature to obtain a decoded feature of the image to be detected; and a prediction head, which is used to obtain a detection result of the image to be detected according to the decoded feature, and the detection result is used to determine whether there is a target object in the image to be detected, and to determine the position information of the object and the category of the object when there is a target object in the image to be detected.

[0017] Preferably, the backbone network includes a first branch and a second branch, the first branch includes multiple cascaded convolutional neural network modules, which are used to extract local features of the image to be detected; the second branch includes multiple cascaded attention mechanism-based encoding modules, and the encoding modules are used to extract global attention features.

[0018] Preferably, the backbone network is also used to directly input the image to be detected into the convolutional neural network module of the first level and convert the image to be detected into an image slice sequence, and input the image slice sequence into the encoding module of the first level; in the feature extraction process of multiple cascaded convolutional neural network modules and multiple cascaded encoding modules, the local features output by the convolutional neural network module of each level are alternately interacted with the global attention features output by the encoding module of each level; the local features output by the last multi-level convolutional neural network module are used as the graph feature pyramid of the image to be detected; the global attention features output by the encoding module of the last level are used as the global semantic features of the image to be detected.

[0019] Preferably, the backbone network also includes: a first feature interaction module, connected between the convolutional neural network module at the same level and the encoding module at the same level, for merging the local features output by the convolutional neural network module at the same level with the global attention features output by the encoding module at the same level, and then inputting the merged features into the encoding module at the current level; a second feature interaction module, connected between the encoding module at the same level and the convolutional neural network module at the same level, for merging the global attention features output by the encoding module at the same level with the local features output by the convolutional neural network module at the same level, and then inputting the merged features into the convolutional neural network module at the next level; wherein the first feature interaction module and the second feature interaction module are alternately connected between the encoding module and the convolutional neural network module.

[0020] Preferably, the first feature interaction module is connected between the output end of the first-level convolutional neural network module and the output end of the first-level encoding module; or the second feature interaction module is connected between the output end of the first-level encoding module and the output end of the first-level convolutional neural network module.

[0021] Preferably, the first feature interaction module includes: a first processing unit, used to align the local features output by the convolutional neural network module of the same level with the global attention features output by the encoding module of the previous level through convolution operations, average pooling operations and layer normalization operations; a first superposition unit, used to superimpose the local features after feature alignment with the global attention features and input them into the current encoding module.

[0022] Preferably, the second feature interaction module includes: a second processing unit, used to align the global attention features output by the same encoding module with the local features output by the convolutional neural network module of the current level through interpolation operations, convolution operations and batch normalization operations; a second superposition unit, used to superimpose the local features after feature alignment with the global attention features and input them into the convolutional neural network module of the next level.

[0023] Preferably, the decoder includes a self-attention module and a cross-attention module, wherein the self-attention module is used to perform self-attention processing on the target query vector to obtain an output first target query vector; the cross-attention module is used to perform cross-attention processing on the fused features and the first target query vector to map the fused features to the first target query vector to obtain a decoded feature; or, the decoder includes a self-attention module and a deformable attention module, wherein the self-attention module is used to perform self-attention processing on the target query vector to obtain an output first target query vector; the deformable attention module is used to perform deformable attention processing on the fused features and the first target query vector to map the fused features to the first target query vector to obtain a decoded feature.

[0024] According to the third aspect of the present application, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the target detection method described above.

[0025] According to a fourth aspect of the present application, a computer-readable storage medium is provided, characterized in that the computer-readable storage medium stores a computer program, and the computer program implements the above-described method when executed by a processor.

[0026] The target detection method, device, electronic device and storage medium provided by the present application perform feature extraction on the local features and global attention features of the image to be detected in a parallel interactive manner through a backbone network to obtain a graph feature pyramid of the image to be detected and a global semantic feature of the image to be detected; perform feature fusion on the graph feature pyramid according to the global semantic feature to obtain a fusion feature of the image to be detected, wherein the fusion feature includes multi-scale information of the image to be detected; decode the fusion feature to obtain a decoded feature of the image to be detected; obtain a detection result of the image to be detected according to the decoded feature, and the detection result is used to determine whether there is a target object in the image to be detected, and when there is a target object in the image to be detected, determine the location information of the object and the category of the object. The present application uses a backbone network to perform feature interaction on the local features and the global attention features in a parallel interactive manner to obtain a graph feature pyramid and a global attention feature, and then performs correlation fusion on the graph feature pyramid through a cross attention layer to obtain a fusion feature with multi-scale information, which can enhance the performance of small target detection.

[0027] Furthermore, the backbone network includes a first branch and a second branch, the first branch includes multiple cascaded convolutional neural network modules, which are used to extract local features of the image to be detected; the second branch includes multiple cascaded attention mechanism-based encoding modules, which are used to extract global attention features, so the encoder can be omitted, simplifying the structure of the target detection network and reducing the network complexity.

[0028] Furthermore, the decoder adopts a deformable attention mechanism, which helps to enhance the convergence of algorithm training and improve the performance and efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other objects, features and advantages of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0030] Figure 1 A schematic diagram of the structure of a target detection framework based on DETR in the prior art is shown;

[0031] Figure 2 A flow chart showing a target detection method provided by an embodiment of the present application is shown;

[0032] Figure 3 A flowchart of step S110 provided in an embodiment of the present application is shown;

[0033] Figure 4 A flowchart of step S112 provided in an embodiment of the present application is shown;

[0034] Figure 5A schematic diagram showing the structure of a target detection device provided in an embodiment of the present application is shown;

[0035] Figure 6 A schematic diagram showing the structure of a backbone network provided in an embodiment of the present application is shown;

[0036] Figure 7 A schematic diagram showing the structure of another backbone network provided in an embodiment of the present application is shown;

[0037] Figure 8 A schematic diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] Various embodiments of the present application will be described in more detail below with reference to the accompanying drawings. In each of the accompanying drawings, the same elements are represented by the same or similar reference numerals. For the sake of clarity, the various parts in the accompanying drawings are not drawn to scale.

[0039] In related technologies, DETR is an end-to-end target detection framework based on Transformer (such as Figure 1 As shown in Figure 1, the image is extracted through the backbone network (Backbone), and then the image features with positional encoding are mapped into a one-dimensional sequence of visual tokens. These visual tokens are input into the encoder (Transformer Encoder) based on the attention mechanism. The Transformer Encoder encodes these visual tokens and outputs the encoded features, and then inputs the encoded features and the preset query vector into the decoder based on the attention mechanism. The output of the decoder is processed by the prediction head Head, and the object category and bounding box are predicted using a feedforward neural network. DETR uses a set of queues to predict the object position and category, and uses an encoder-decoder architecture to process the image and output the prediction. This framework is simple and intuitive, and directly predicts the detection box for the image sequence, solving the shortcoming of traditional target detection that requires non-maximum suppression. However, DETR only encodes and decodes the feature map of the last layer of the convolutional neural network. Due to the lack of multi-scale feature information, the model has low detection accuracy for small targets; in addition, the attention mechanism requires a larger number of parameters, which is more difficult to converge than the traditional convolutional target detection network.

[0040] In response to the problems of low detection accuracy of DETR for small targets and poor training convergence in related technologies, the embodiments of the present application provide a target detection method, which at least to a certain extent overcomes the defects of the existing related technologies.

[0041] Specifically, the embodiments of the present application provide a target detection method, device, electronic device, and computer-readable storage medium. The target detection device can be integrated in an electronic device, which can be a server or a terminal.

[0042] The target detection method provided in the embodiment of the present application uses a backbone network to perform feature interaction between local features and global attention features in a parallel interactive manner to obtain a graph feature pyramid and a global attention feature, and then uses a cross-attention layer to perform correlation fusion on the graph feature pyramid to obtain a fused feature with multi-scale information, which can enhance the small target detection performance.

[0043] The execution subject of the target detection method provided in the embodiment of the present application can be the target detection device provided in the embodiment of the present application, or different types of electronic devices such as a server device, a physical host or a user equipment (UE) that integrates the target detection device, wherein the target detection device can be implemented in hardware or software, and the UE can specifically be a terminal device such as a smart phone, a tablet computer, a laptop computer, a PDA, a desktop computer or a personal digital assistant (PDA).

[0044] The electronic device can adopt a working mode of independent operation, or can also adopt a working mode of a device cluster. By applying the target detection method provided in the embodiment of the present application, the accuracy of target detection can be improved.

[0045] The specific implementation of the present application is further described in detail below in conjunction with the drawings and examples.

[0046] Figure 2 FIG. 1 is a flow chart showing a target detection method provided by an embodiment of the present application. Figure 2 As shown, the target detection method includes the following steps.

[0047] In step S110, the local features and global attention features of the image to be detected are extracted in a parallel interactive manner through the backbone network to obtain a graph feature pyramid of the image to be detected and a global semantic feature of the image to be detected.

[0048] In this embodiment, the image to be detected is an image that needs to perform target detection. The image to be detected can be an image collected online, such as the image to be detected that can be collected online through web crawler technology; or, the image to be detected can also be an image collected offline; or, the image to be detected can be an image collected in real time; or, the image to be detected can also be an artificially synthesized image, etc., and the present disclosed embodiment does not impose any restrictions on this.

[0049] It should be understood that the image to be detected can also be a frame of video in the video, and the image to be detected can be extracted from the video, wherein the above video can be a target video, and the method of acquiring the target video is similar to the method of acquiring the image to be detected, which will not be repeated here.

[0050] The embodiment of the present application aims to perform target detection in an image to be detected to determine whether there is a target object in the image to be detected. The target object may be one or more. When a target object exists in the image to be detected, the position information of the target object and the category of the target object are determined.

[0051] The backbone network (Backbone) includes a first branch 211 and a second branch 212, wherein the first branch 211 is a CNN network, which is used to extract local features of the image to be detected, and the first branch 211 includes multiple cascaded convolutional neural network modules, for example, n cascaded convolutional neural network modules (2111-222n), n is a positive integer, and n≥3; the second branch 222 is an encoder based on an attention mechanism, which is used to extract global attention features of the image to be detected, and the second branch 222 includes multiple cascaded attention mechanism-based encoding modules, for example, n cascaded encoding modules (2121-212n).

[0052] Specifically, see Figure 3 , step S110 includes steps S111 to S114.

[0053] In step S111, the image to be detected is directly input into the convolutional neural network module of the first level, and the image to be detected is converted into an image slice sequence, and the image slice sequence is input into the encoding module of the first level.

[0054] In this embodiment, the image to be detected (Image) is subjected to local feature extraction by multiple levels of convolutional neural network modules (CNN) of the first branch to obtain image features of different scales. At the same time, the image to be detected is segmented to obtain image slice sequences (Image patches), and the image slice sequences are subjected to global feature extraction by multiple attention-based encoding modules (Transformer encoder) of the second branch.

[0055] In step S112, during the feature extraction process of multiple cascaded convolutional neural network modules and multiple cascaded encoding modules, the local features output by the convolutional neural network module at each level are alternately interacted with the global attention features output by the encoding module at each level.

[0056] In this embodiment, since the local features output by the convolutional neural network module are obtained through local convolution operations, and the global attention features output by the encoding module are obtained by aggregation through a global self-attention mechanism, there is an asymmetry between the local features and the global attention features, so a parallel interaction method is adopted to eliminate the semantic differences between the two.

[0057] See also Figure 4 , step S112 includes step S1121-step S1123.

[0058] In step S1121, the local features output by the convolutional neural network module of the previous layer and the global attention features output by the encoding module of the previous layer are merged and input into the encoding module of the current layer.

[0059] In this embodiment, the feature merging of the local features output by the convolutional neural network module and the global attention features output by the encoding module includes: performing convolution operations (Conv), average pooling operations (Avgpooling) and layer normalization operations (LayerNorm) on the local features output by the convolutional neural network at the same level to align the local features with the global attention features output by the encoding module of the current level; superimposing the local features after feature alignment with the global attention features and inputting them into the encoding module of the next level.

[0060] In step S1122, the global attention features output by the encoding module of the current level and the local features output by the convolutional neural network module of the current level are merged and input into the convolutional neural network module of the next level.

[0061] In this embodiment, the feature merging of the global attention features output by the encoding module and the local features output by the convolutional neural network module includes: performing interpolation operations (Interploate), convolution operations (Conv) and batch normalization operations (BatchNorm) on the global attention features output by the encoding module of the same level to align the global attention features with the local features output by the convolutional neural network module of the same level; superimposing the local features after feature alignment with the global attention features and inputting them into the convolutional neural network module of the next level.

[0062] The feature dimensions of local features and global attention features are inconsistent. The dimension of local features is C×H×W, where C is the number of channels, H is the height, and W is the width; the dimension of global attention features is K×E, where K is the number of image sequences and E is the dimension of global attention features. Therefore, the two features need to be aligned when merging features. Specifically, when aligning local features with global attention features, 1×1 convolution operation (Conv) is used for dimensionality reduction, downsampling is used for feature resolution alignment, and the specific method of downsampling is average pooling operation (Avgpooling), and then layer normalization operation (LayerNorm) is used for feature value alignment. When aligning global attention features with local features, upsampling is used for feature resolution alignment, and the specific method of upsampling is interpolation operation (Interpolate), 1×1 convolution operation (Conv) convolution operation is used for channel size alignment, and then batch normalization operation (BatchNorm) is used for feature value alignment.

[0063] In step S1123, by analogy, the local features output by the convolutional neural network module are alternately interacted with the global attention features output by the encoding module.

[0064] In this embodiment, from the first level to the nth level, the local features output by the convolutional neural network module and the global attention features output by the encoding module are first merged; then the global attention features output by the encoding module and the local features output by the convolutional neural network module are merged, and the two feature merges are performed alternately. Alternatively, the global attention features output by the encoding module and the local features output by the convolutional neural network module are first merged, and then the local features output by the convolutional neural network module and the global attention features output by the encoding module are merged.

[0065] For example, the local features output by the first-level CNN module 2111 are merged with the global attention features output by the first-level encoding module 2121 and then input to the second-level encoding module 2122; the global attention features output by the second-level encoding module 2122 are merged with the local features output by the second-level CNN module 2112 and then input to the third-level CNN module 2113; and so on, the local features and global attention features of adjacent levels are alternately interacted (such as Figure 6Alternatively, the global attention features output by the encoding module 2121 of the first level are merged with the local features output by the CNN module 2111 of the first level and then input to the CNN module 2112 of the second level; the local features output by the CNN module 2112 of the second level are merged with the global attention features output by the encoding module 2122 of the second level and then input to the encoding module 2123 of the third level; and so on, the local features and global attention features of adjacent levels are alternately interacted (such as Figure 7 shown).

[0066] In step S113, the local features output by the convolutional neural network modules of the last multiple levels are used as the graph feature pyramid of the image to be detected.

[0067] In this embodiment, the last multiple levels include at least 3 levels, and the convolutional neural network modules at different levels output image features of different scales. The graph feature pyramid includes at least 3 scales of image features, which are unrelated to each other.

[0068] In step S114, the global attention feature output by the encoding module of the last level is used as the global semantic feature of the image to be detected.

[0069] In this embodiment, the global attention features output by the encoding module 212n of the last level are used as the global semantic features of the image to be detected.

[0070] In step S120, feature fusion is performed on the graph feature pyramid according to the global semantic feature to obtain a fused feature of the image to be detected, wherein the fused feature includes multi-scale information of the image to be detected.

[0071] In this embodiment, the graph feature pyramid is expanded into a one-dimensional vector and concatenated to obtain a concatenated vector; the concatenated vector and the global semantic feature are fused through a cross attention layer and a feedforward neural network layer to obtain a fused feature.

[0072] Specifically, the concatenated vector and the global semantic feature are fused through a cross-attention layer (Cross-Attention) and a feedforward neural network layer (FNN) to obtain a fused feature, including: determining the query feature (i.e., Q value) according to the concatenated vector, determining the content feature (i.e., V value) and the key-value feature (i.e., K value) according to the global attention feature, performing cross-attention processing based on the content feature, the key-value feature and the query feature, and then outputting the fused feature through the feedforward neural network layer (FNN).

[0073] In step S130, the fused features are decoded to obtain decoded features of the image to be detected.

[0074] In this embodiment, the decoded features are obtained by inputting the object queries and the fused features into the decoder. The fused features are fed into the decoder as visual tokens. The decoder may include a self-attention module and a cross-attention module.

[0075] In some embodiments, the decoder includes M decoding layers, each of which includes a self-attention module and a cross-attention module. The decoder processes the object query vector based on the self-attention module and outputs a first object query vector, wherein the first object query vector includes distinguishing information between the object query vectors. Subsequently, the decoder maps the fused feature to the first object query vector based on the cross-attention module to obtain a decoded feature.

[0076] In some embodiments, the size of the target query vector is pre-set, and the size of the target query vector is determined by the size of a single target query object and the number of target query objects. For example, if the dimension of a single target query object (objectquery) is 1×256, and 100 target query objects are preset, the size of the target query vector is 100×256. The content of the initial target query vector is randomly generated and subsequently changes during the training process.

[0077] In some embodiments, mapping the fused features to a preset first target query vector can be by determining content features (i.e., V value) and key value features (i.e., K value) based on the fused features, determining query features (i.e., Q value) based on the first target query vector, performing cross-attention processing based on content features, key value features, and query features, and completing feature mapping.

[0078] In some embodiments, mapping the fused features to a preset first target query vector can be by determining content features (i.e., V value) and key-value features (i.e., K value) based on the fused features, determining query features (i.e., Q value) based on the first target query vector, performing deformable attention processing based on the content features, key-value features, and query features, and completing feature mapping.

[0079] In step S140, a detection result of the image to be detected is obtained according to the decoding feature, and the detection result is used to determine whether there is a target object in the image to be detected, and when there is a target object in the image to be detected, determine the position information of the target object and the category of the target object.

[0080] The target detection method provided in the embodiment of the present application uses a backbone network to perform feature interaction between local features and global attention features in a parallel interactive manner to obtain a graph feature pyramid and a global attention feature, and then uses a cross-attention layer to perform correlation fusion on the graph feature pyramid to obtain a fused feature with multi-scale information, which can enhance the small target detection performance.

[0081] Furthermore, the backbone network includes a first branch and a second branch, the first branch includes multiple cascaded convolutional neural network modules, which are used to extract local features of the image to be detected; the second branch includes multiple cascaded attention mechanism-based encoding modules, which are used to extract global attention features, so the encoder can be omitted, simplifying the structure of the target detection network and reducing the network complexity.

[0082] Furthermore, the decoding process adopts a deformable attention mechanism, which helps to enhance the convergence of algorithm training and improve the performance and efficiency of the model.

[0083] Figure 5 FIG. 1 is a schematic diagram showing the structure of the target detection device provided in the embodiment of the present application. Figure 5 As shown, the target detection device includes a backbone network 210, a feature fusion module 220, a decoder 230 and a prediction head 240.

[0084] Among them, the backbone network 210 is used to extract local features and global attention features of the image to be detected in a parallel interactive manner to obtain a graph feature pyramid of the image to be detected and a global semantic feature of the image to be detected.

[0085] In this embodiment, the image to be detected is an image that needs to perform target detection. The image to be detected can be an image collected online, such as the image to be detected that can be collected online through web crawler technology; or, the image to be detected can also be an image collected offline; or, the image to be detected can be an image collected in real time; or, the image to be detected can also be an artificially synthesized image, etc., and the present disclosed embodiment does not impose any restrictions on this.

[0086] It should be understood that the image to be detected can also be a frame of video in the video, and the image to be detected can be extracted from the video, wherein the above video can be a target video, and the method of acquiring the target video is similar to the method of acquiring the image to be detected, which will not be repeated here.

[0087] The embodiment of the present application aims to perform target detection in an image to be detected to determine whether there is a target object in the image to be detected. The target object may be one or more. When a target object exists in the image to be detected, the position information of the target object and the category of the target object are determined.

[0088] The backbone network (Backbone) includes a first branch 211 and a second branch 212, wherein the first branch 211 is a CNN network, which is used to extract local features of the image to be detected, and the first branch 211 includes multiple cascaded convolutional neural network modules, for example, n cascaded convolutional neural network modules (2111-222n), n is a positive integer, and n≥3; the second branch 222 is an encoder based on an attention mechanism, which is used to extract global attention features of the image to be detected, and the second branch 222 includes multiple cascaded attention mechanism-based encoding modules, for example, n cascaded encoding modules (2121-212n).

[0089] The backbone network 210 is also used to directly input the image to be detected into the convolutional neural network module of the first level and convert the image to be detected into an image slice sequence, and input the image slice sequence into the encoding module of the first level; in the feature extraction process of multiple cascaded convolutional neural network modules and multiple cascaded encoding modules, the local features output by the convolutional neural network module of each level are alternately interacted with the global attention features output by the encoding module of each level; the local features output by the last multi-level convolutional neural network module are used as the graph feature pyramid of the image to be detected; the global attention features output by the encoding module of the last level are used as the global semantic features of the image to be detected.

[0090] Specifically, the image to be detected (Image) is subjected to local feature extraction by convolutional neural network modules (CNN) of multiple levels of the first branch to obtain image features of different scales. At the same time, the image to be detected is segmented to obtain an image slice sequence (Image patches), and the image slice sequence is subjected to global feature extraction by multiple encoding modules (Transformer encoder) based on the attention mechanism of the second branch. The last multiple levels include at least 3 levels, and convolutional neural network modules of different levels output image features of different scales. The graph feature pyramid includes at least 3 scales of image features, which are unrelated to each other. The global attention features output by the encoding module 212n of the last level are used as the global semantic features of the image to be detected.

[0091] Since the local features output by the convolutional neural network module are obtained through local convolution operations, and the global attention features output by the encoding module are aggregated through the global self-attention mechanism, there is an asymmetry between the local features and the global attention features. Therefore, a parallel interaction method is used to eliminate the semantic differences between the two.

[0092] The backbone network 210 further includes a plurality of first feature merging modules 213 and a plurality of second feature merging modules 214 .

[0093] Among them, the first feature interaction module 213 is connected between the convolutional neural network module at the same level and the encoding module at the same level, and is used to merge the local features output by the convolutional neural network module at the same level with the global attention features output by the encoding module at the same level, and then input the merged features into the encoding module at the current level.

[0094] In this embodiment, the first feature interaction module 213 includes a first processing unit 2131 and a first superposition unit 2132. The first processing unit 2131 is used to align the local features output by the convolutional neural network module of the same level with the global attention features output by the encoding module of the same level through convolution operation (Conv), average pooling operation (Avgpooling) and layer normalization operation (LayerNorm); the first superposition unit 2132 is used to superimpose the local features after feature alignment with the global attention features and input them into the current encoding module.

[0095] The second feature interaction module 214 is connected between the encoding module at the same level and the convolutional neural network module at the same level, and is used to merge the global attention features output by the encoding module at the same level with the local features output by the convolutional neural network module at the same level, and then input the merged features into the convolutional neural network module at the next level. Among them, the first feature interaction module 213 and the second feature interaction module 214 are alternately connected between the encoding module and the convolutional neural network module.

[0096] In this embodiment, the first feature interaction module 214 includes a second processing unit 2141 and a second superposition unit 2142. The second processing unit 2141 is used to align the global attention features output by the encoding module of the same level with the local features output by the convolutional neural network module of the same level through interpolation, convolution and batch normalization; the second superposition unit 2142 is used to superimpose the local features after feature alignment with the global attention features and input them into the convolutional neural network module of the next level.

[0097] In this embodiment, see Figure 6 , from the first level to the nth level, the first feature interaction module 213 is connected between the output end of the convolutional neural network module 2111 of the first level and the output end of the encoding module 2121 of the first level; the second feature interaction module 214 is connected between the output end of the encoding module 2122 of the second level and the output end of the convolutional neural network module 2112 of the second level, and then the first feature interaction module 213 and the second feature interaction module 214 are alternately connected between the convolutional neural network module 211 and the encoding module 212 of the same level.

[0098] In one possible implementation, see Figure 7 , from the first level to the nth level, the second feature interaction module 214 is connected between the output end of the encoding module 2121 of the first level and the output end of the convolutional neural network module 2111 of the first level; the first feature interaction module 213 is connected between the output end of the convolutional neural network module 2112 of the second level and the output end of the encoding module 2122 of the second level, and then the first feature interaction module 213 and the second feature interaction module 214 are alternately connected between the convolutional neural network module 211 and the encoding module 212 of the same level.

[0099] The feature dimensions of local features and global attention features are inconsistent. The dimension of local features is C×H×W, where C is the number of channels, H is the height, and W is the width; the dimension of global attention features is K×E, where K is the number of image sequences and E is the dimension of global attention features. Therefore, the two features need to be aligned when merging features. Specifically, when aligning local features with global attention features, 1×1 convolution operation (Conv) is used for dimensionality reduction, downsampling is used for feature resolution alignment, and the specific method of downsampling is average pooling operation (Avgpooling), and then layer normalization operation (LayerNorm) is used for feature value alignment. When aligning global attention features with local features, upsampling is used for feature resolution alignment, and the specific method of upsampling is interpolation operation (Interpolate), 1×1 convolution operation (Conv) convolution operation is used for channel size alignment, and then batch normalization operation (BatchNorm) is used for feature value alignment.

[0100] The feature fusion module 220 performs feature fusion on the graph feature pyramid according to the global semantic features to obtain fused features of the image to be detected, wherein the fused features include multi-scale information of the image to be detected.

[0101] In this embodiment, the feature fusion module 220 includes a feature processing unit 221 and a feature fusion unit 222, wherein the feature processing unit 221 is used to expand the graph feature pyramid into a one-dimensional vector for splicing to obtain a splicing vector; the feature fusion unit 222 is used to fuse the splicing vector with the global semantic feature through a cross attention layer and a feedforward neural network layer to obtain a fused feature.

[0102] In some possible implementations, the feature fusion unit 222 can determine the query feature (i.e., Q value) based on the splicing vector, determine the content feature (i.e., V value) and the key value feature (i.e., K value) based on the global attention feature, perform cross-attention processing based on the content feature, key value feature and query feature, and then output the fused feature through the feedforward neural network layer.

[0103] The decoder 230 is used to decode the fused features to obtain decoded features of the image to be detected.

[0104] In this embodiment, the decoder 230 obtains the decoded features by inputting the object queries and the fused features into the decoder. The fused features are sent to the decoder as visual tokens. The decoder 230 may include a self-attention module and a cross-attention module.

[0105] In some embodiments, the decoder includes M decoding layers, each of which includes a self-attention module and a cross-attention module. The decoder 230 processes the object query vector based on the self-attention module to obtain a first object query vector, wherein the first object query vector contains distinguishing information between the object query vectors. Subsequently, the decoder 230 maps the fused feature to the first object query vector based on the cross-attention module to obtain a decoded feature.

[0106] In some embodiments, the size of the target query vector is pre-set, for example, the target query vector is a vector of 256. The content of the initial target query vector is randomly generated and subsequently changed during the training process.

[0107] In some embodiments, the cross attention module in the decoder 230 can be replaced by a deformable attention module. For the deformable self-attention module, unlike the traditional self-attention mechanism, the deformable self-attention module only focuses on a set of key sampling points around the image reference point, and does not need to pay attention to the size of the feature map. Compared with the traditional self-attention mechanism, which has a quadratic exponential complexity for the input, the deformable self-attention module reduces the complexity of attention for the input by sparsely querying, and it has a linear complexity relative to the input.

[0108] In some possible implementations, the decoder 230 maps the fused features to a preset query feature vector by determining content features (i.e., V value) and key-value features (i.e., K value) based on the fused features, determining query features (i.e., Q value) based on the query feature vector, performing cross-attention processing based on the content features, key-value features, and query features, and completing feature mapping.

[0109] The prediction head 240 is used to obtain the detection result of the image to be detected according to the decoded features, and the detection result is used to determine the position information of the object and the category of the object.

[0110] The target detection device provided in the present application uses a backbone network to use a parallel interactive method to interact local features and global attention features to obtain a graph feature pyramid and a global attention feature, and then uses a cross-attention layer to perform correlation fusion on the graph feature pyramid to obtain a fused feature with multi-scale information, which can enhance the small target detection performance.

[0111] Furthermore, the backbone network includes a first branch and a second branch, the first branch includes multiple cascaded convolutional neural network modules, which are used to extract local features of the image to be detected; the second branch includes multiple cascaded attention mechanism-based encoding modules, which are used to extract global attention features, so the encoder can be omitted, simplifying the structure of the target detection network and reducing the network complexity.

[0112] Furthermore, the decoder adopts a deformable attention mechanism, which helps to enhance the convergence of algorithm training and improve the performance and efficiency of the model.

[0113] Figure 8 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 8 As shown, the electronic device 800 of this embodiment includes: at least one processor 801 ( Figure 8 Only one is shown in the figure) a processor, a memory 802, and a computer program 803 stored in the memory 802 and executable on the at least one processor 801, and when the processor 801 executes the computer program 803, the steps in any of the above-mentioned storage space allocation method embodiments are implemented.

[0114] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 8 This is merely an example of the electronic device 800 and does not constitute a limitation on the electronic device 800 . The electronic device 800 may include more or fewer components than those shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0115] The processor 801 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0116] In some embodiments, the memory 802 may be an internal storage unit of the electronic device 800, such as a hard disk or memory of the electronic device 800. In other embodiments, the memory 802 may also be an external storage device of the electronic device 800, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 800. Further, the memory 802 may also include both an internal storage unit of the electronic device 800 and an external storage device. The memory 802 is used to store an operating system, an application program, a boot loader (Boot Loader), data and other programs, such as the program code of the computer program, etc. The memory 802 may also be used to temporarily store data that has been output or is to be output.

[0117] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0118] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0119] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / electronic device, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0120] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0121] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0122] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0123] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0124] The embodiments of the present application are described above, and these embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and changes can be made based on the above description. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can make good use of the present application and the modifications based on the present application. The present application is only limited by the claims and their full scope and equivalents.

Claims

1. A target detection method, characterized in that: include: Through the backbone network, local features and global attention features of the image to be detected are extracted in a parallel interactive manner to obtain a graph feature pyramid of the image to be detected and a global semantic feature of the image to be detected; Performing feature fusion on the graph feature pyramid according to the global semantic feature to obtain a fusion feature of the image to be detected, wherein the fusion feature includes multi-scale information of the image to be detected; Decoding the fused features to obtain decoded features of the image to be detected; Acquire a detection result of the image to be detected according to the decoded feature, wherein the detection result is used to determine whether a target object exists in the image to be detected, and when a target object exists in the image to be detected, determine the position information of the target object and the category of the target object; Among them, the backbone network includes a first branch and a second branch, the first branch includes multiple cascaded convolutional neural network modules, which are used to extract local features of the image to be detected; the second branch includes multiple cascaded attention mechanism-based encoding modules, and the encoding modules are used to extract global attention features. The local features output by the convolutional neural network module of each level and the global attention features output by the encoding module of each level are merged and then alternately input into the convolutional neural network module or encoding module of the next level.

2. The target detection method according to claim 1, characterized in that: Through the backbone network, local features and global attention features of the image to be detected are extracted in a parallel interactive manner to obtain a graph feature pyramid of the image to be detected and global semantic features of the image to be detected, including: The image to be detected is directly input into the convolutional neural network module of the first layer, and the image to be detected is converted into an image slice sequence and the image slice sequence is input into the encoding module of the first layer; In the process of feature extraction by multiple cascaded convolutional neural network modules and multiple cascaded encoding modules, the local features output by the convolutional neural network modules at each level are alternately interacted with the global attention features output by the encoding modules at each level; Using the local features output by the convolutional neural network modules of the last multiple levels as the graph feature pyramid of the image to be detected; The global attention features output by the encoding module of the last level are used as the global semantic features of the image to be detected.

3. The target detection method according to claim 2, characterized in that: Alternating the local features output by the convolutional neural network module at each level with the global attention features output by the encoding module includes: The local features output by the convolutional neural network module of the previous layer and the global attention features output by the encoding module of the previous layer are combined and input into the encoding module of the current layer; The global attention features output by the encoding module of the current level are combined with the local features output by the convolutional neural network module of the current level and then input into the convolutional neural network module of the next level; By analogy, the local features output by the convolutional neural network module are alternately interacted with the global attention features output by the encoding module.

4. The target detection method according to claim 3, characterized in that: The local features output by the first-level convolutional neural network module are merged with the global attention features output by the first-level encoding module and then input into the second-level encoding module; or the global attention features output by the first-level encoding module are merged with the local features output by the first-level convolutional neural network module and then input into the second-level convolutional neural network module.

5. The target detection method according to claim 3, characterized in that: The local features output by the convolutional neural network module and the global attention features output by the encoding module are combined to form a feature map including: The local features output by the convolutional neural network at the same level are subjected to convolution operations, average pooling operations, and layer normalization operations to align the local features with the global attention features output by the encoding module at the same level; The local features after feature alignment are superimposed with the global attention features and input into the encoding module of the next level; and The global attention features output by the encoding module and the local features output by the convolutional neural network module are combined to form a feature map, which includes: Perform interpolation, convolution and batch normalization operations on the global attention features output by the encoding module at the same level to align the global attention features with the local features output by the convolutional neural network module at the same level; The local features after feature alignment are superimposed with the global attention features and input into the convolutional neural network module of the next level.

6. The target detection method according to claim 1, characterized in that: The fusion feature of the image to be detected is obtained by performing feature fusion on the graph feature pyramid according to the global semantic feature, wherein the fusion feature includes multi-scale information of the image to be detected, including: Expanding the graph feature pyramid into a one-dimensional vector and concatenating them to obtain a concatenated vector; The concatenated vector and the global semantic feature are fused through a cross attention layer and a feedforward neural network layer to obtain the fused feature.

7. The target detection method according to claim 1, characterized in that: Decoding the fused features to obtain decoded features of the image to be detected includes: Performing self-attention processing on the target query vector to obtain a first target query vector; Cross attention processing or deformable attention processing is performed on the fused features and the first target query vector to map the fused features to the first target query vector to obtain decoded features.

8. A target detection device, characterized in that: include: A backbone network is used to extract local features and global attention features of the image to be detected in a parallel interactive manner to obtain a graph feature pyramid of the image to be detected and a global semantic feature of the image to be detected; A feature fusion module, which performs feature fusion on the graph feature pyramid according to the global semantic feature to obtain a fusion feature of the image to be detected, wherein the fusion feature includes multi-scale information of the image to be detected; A decoder, used for decoding the fused features to obtain decoded features of the image to be detected; A prediction head, used to obtain a detection result of the image to be detected according to the decoded feature, wherein the detection result is used to determine whether a target object exists in the image to be detected, and to determine the position information of the object and the category of the object when the target object exists in the image to be detected; The backbone network includes a first branch and a second branch, the first branch includes a plurality of cascaded convolutional neural network modules for extracting local features of the image to be detected; the second branch includes a plurality of cascaded attention mechanism-based encoding modules, and the encoding module is used to extract global attention features; The backbone network is also used to merge the local features output by the convolutional neural network module of each level with the global attention features output by the encoding module of each level, and then alternately input them into the convolutional neural network module or encoding module of the next level.

9. The target detection device according to claim 8, characterized in that: The backbone network is also used to directly input the image to be detected into the convolutional neural network module of the first level and convert the image to be detected into an image slice sequence, and input the image slice sequence into the encoding module of the first level; in the feature extraction process of multiple cascaded convolutional neural network modules and multiple cascaded encoding modules, the local features output by the convolutional neural network module of each level are alternately interacted with the global attention features output by the encoding module of each level; the local features output by the last multi-level convolutional neural network module are used as the graph feature pyramid of the image to be detected; the global attention features output by the encoding module of the last level are used as the global semantic features of the image to be detected.

10. The target detection device according to claim 9, characterized in that: The backbone network also includes: The first feature interaction module is connected between the convolutional neural network module at the same level and the encoding module at the same level, and is used to merge the local features output by the convolutional neural network module at the same level with the global attention features output by the encoding module at the same level, and then input the merged features into the encoding module at the current level; The second feature interaction module is connected between the encoding module at the same level and the convolutional neural network module at the same level, and is used to merge the global attention features output by the encoding module at the same level with the local features output by the convolutional neural network module at the same level, and then input the merged features into the convolutional neural network module at the next level; Among them, the first feature interaction module and the second feature interaction module are alternately connected between the encoding module and the convolutional neural network module.

11. The target detection device according to claim 10, characterized in that: The first feature interaction module is connected between the output end of the first-level convolutional neural network module and the output end of the first-level encoding module; or the second feature interaction module is connected between the output end of the first-level encoding module and the output end of the first-level convolutional neural network module.

12. The target detection device according to claim 10, characterized in that: The first feature interaction module includes: A first processing unit is used to align the local features output by the convolutional neural network module of the same layer with the global attention features output by the encoding module of the previous layer through convolution operations, average pooling operations and layer normalization operations; The first superposition unit is used to superimpose the local features after feature alignment with the global attention features and input them into the current encoding module.

13. The target detection device according to claim 10, characterized in that: The second feature interaction module includes: The second processing unit is used to align the global attention features output by the same encoding module with the local features output by the convolutional neural network module of the current level through interpolation operation, convolution operation and batch normalization operation; The second superposition unit is used to superimpose the local features after feature alignment with the global attention features and input them into the convolutional neural network module of the next layer.

14. The target detection device according to claim 8, characterized in that: The decoder includes a self-attention module and a cross-attention module, The self-attention module is used to perform self-attention processing on the target query vector to output a first target query vector; The cross attention module is used to perform cross attention processing on the fused feature and the first target query vector to map the fused feature to the first target query vector to obtain a decoded feature; or, The decoder includes a self-attention module and a deformable attention module, The self-attention module is used to perform self-attention processing on the target query vector to output a first target query vector; The deformable attention module is used to perform deformable attention processing on the fused features and the first target query vector to map the fused features to the first target query vector to obtain a decoded feature.

15. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the target detection method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images