Target detection method, device and equipment

Abstract

The invention provides a target detection method, device and equipment. According to the method provided by the invention, feature extraction is carried out on a prompt text of a to-be-detected image and a prompt text of a to-be-detected target, and feature maps and text features of multiple scales are obtained; generating a self-attention feature of the text feature through a self-attention mechanism; for the feature map of each scale, mapping the feature map of the scale to a weight space of a convolution kernel, and generating a weight of a dynamic convolution kernel corresponding to the scale; performing convolution processing on the feature map of the scale by using the dynamic convolution kernel corresponding to the scale to obtain a dynamic convolution feature under the scale; and performing target detection by using the plurality of dynamic convolution features under the plurality of scales and the self-attention features to obtain a detection result. According to the target detection method, device and equipment provided by the invention, the calculation amount of target detection can be reduced, so that the applicability of a target detection algorithm is improved, and the target detection algorithm is applied to edge low-calculation-power equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, and particularly relates to a target detection method, device and equipment. BACKGROUND

[0002] With the development of deep learning and convolutional neural network technology, the accuracy and efficiency of target detection are significantly improved, and the application of target detection in various fields is more and more extensive. In recent years, the target detection model based on Transformer is widely used in various fields. The target detection model based on Transformer processes image features through a self-attention mechanism or a deformable attention mechanism, has high computational complexity and large amount of calculation, and is not suitable for application in edge low-power devices. SUMMARY

[0003] Therefore, the present application provides a target detection method, device and equipment to reduce the amount of calculation of target detection, improve the applicability of the target detection algorithm, and make it applicable to edge low-power devices.

[0004] Specifically, the present application is implemented through the following technical solutions:

[0005] The first aspect of the present application provides a target detection method, which comprises:

[0006] Respectively extracting features of a to-be-detected image and a prompt text of a to-be-detected target to obtain feature maps of multiple scales and text features;

[0007] Generating a self-attention feature of the text feature through a self-attention mechanism;

[0008] For each scale of the feature map, mapping the feature map of the scale to a weight space of a convolution kernel to generate a weight of a dynamic convolution kernel corresponding to the scale;

[0009] Performing convolution processing on the feature map of the scale by using the dynamic convolution kernel corresponding to the scale to obtain a dynamic convolution feature under the scale;

[0010] Performing target detection by using the dynamic convolution features under the multiple scales and the self-attention feature to obtain a detection result.

[0011] The second aspect of the present application provides a target detection device, which comprises an extraction module, a generation module, a convolution module and a detection module; wherein,

[0012] The extraction module is configured to respectively extract features of a to-be-detected image and a prompt text of a to-be-detected target to obtain feature maps of multiple scales and text features;

[0013] The generation module is configured to generate a self-attention feature of the text feature through a self-attention mechanism.

[0014] The generation module is further configured to, for the feature map of each scale, map the feature map of the scale to a weight space of a convolution kernel, and generate a weight of a dynamic convolution kernel corresponding to the scale.

[0015] The generation module is further configured to perform convolution processing on the feature map of the scale by using the dynamic convolution kernel corresponding to the scale, to obtain a dynamic convolution feature under the scale.

[0016] The detection module is configured to perform target detection by using the dynamic convolution features under the multiple scales and the self-attention feature, to obtain a detection result.

[0017] The third aspect of the present application provides a target detection device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the methods provided in the first aspect of the present application when executing the program.

[0018] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executable by a processor to implement the steps of any one of the methods provided in the first aspect of the present application.

[0019] The target detection method, device and equipment provided in the present application can, after obtaining the feature maps of multiple scales, for the feature map of each scale, map the feature map of the scale to a weight space of a convolution kernel, generate a weight of a dynamic convolution kernel corresponding to the scale, and then perform convolution processing on the feature map of the scale by using the dynamic convolution kernel corresponding to the scale, to obtain a dynamic convolution feature under the scale. In this way, the size and weight of the dynamic convolution kernel under the scale can be dynamically adjusted according to the scale of the feature map, the dynamic convolution kernel performs adaptive calculation on the feature maps of different scales, which helps to reduce unnecessary calculation while maintaining efficient feature extraction, significantly reduces the amount of calculation, and makes the target detection method adaptable to edge devices with lower computing power. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1 The flowchart of the target detection method provided in the present application is shown in Embodiment One.

[0021] Figure 2 The schematic diagram of the dynamic convolution kernels corresponding to different scales is shown in an exemplary embodiment of the present application.

[0022] Figure 3 The flowchart of the target detection method provided in the present application is shown in Embodiment Two.

[0023] Figure 4A schematic diagram of a target detection model shown in an example embodiment of the present application;

[0024] Figure 5 A schematic diagram of a first point-wise convolution module shown in an example embodiment of the present application;

[0025] Figure 6 A schematic diagram of a weight generation module shown in an example embodiment of the present application;

[0026] Figure 7 A flow chart of a target detection method embodiment three provided by the present application;

[0027] Figure 8 A schematic diagram of a cross-attention module shown in an example embodiment of the present application;

[0028] Figure 9 A flow chart of a target detection method embodiment four provided by the present application;

[0029] Figure 10 A schematic diagram of a target detection model shown in another example embodiment of the present application;

[0030] Figure 11 A hardware structure diagram of a target detection device in which a target detection apparatus provided by the present application is located;

[0031] Figure 12 A structural schematic diagram of a target detection apparatus embodiment one provided by the present application. DETAILED DESCRIPTION

[0032] The example embodiments will be described in detail herein with reference to the attached drawings. The following description is only exemplary and is not intended to limit the present application. The same reference numbers in different drawings represent the same or similar elements unless otherwise indicated.

[0033] The terms used in the present application are merely for the purpose of describing particular embodiments and are not intended to limit the present application. The singular forms "a," "an," and "the" used in the present application are also intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application means and includes any or all possible combinations of one or more associated listed items.

[0034] It should be understood that, although the terms first, second, third, etc. can be employed in this application to describe various information, the information should not be limited to these terms. These terms are only used to differentiate one piece of information from another piece of information. For example, without departing from the scope of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon determining" or "in response to determining".

[0035] It should be noted that the target detection method and device provided in the present application are applied to a target detection device, which can be an edge low-computing-power device. The edge low-computing-power device refers to a device deployed at the edge of a network, which has low computing power and low energy consumption.

[0036] Further, the target detection method, device and equipment provided in the present application can be applied to security monitoring, autonomous driving, smart home and industrial automation scenarios, for example, in the security monitoring scenario, suspicious persons, left-over articles, dangerous and prohibited articles, etc. can be identified; for another example, in the autonomous driving scenario, road markings, traffic signal lights, pedestrians, vehicles, etc. can be identified based on prompt text and to-be-detected images, so as to improve the safety and navigation ability of vehicles; for another example, in the smart home scenario, the target detection method can be used for security monitoring, object identification or intelligent control, etc., for example, lost articles can be queried based on the target detection method. For another example, in the industrial automation scenario, the target detection technology can be used for quality detection, part identification and classification, etc.

[0037] The following specific embodiments are given to introduce the technical solutions of the present application in detail.

[0038] Figure 1 The flowchart of the target detection method provided in the present application is shown in Figure 1. It should be noted that the method provided in the present application can include the following steps. Figure 1 The method provided in the present embodiment can include the following steps.

[0039] S101, respectively extracting features of the to-be-detected image and the prompt text of the to-be-detected target to obtain feature maps of multiple scales and text features.

[0040] Specifically, the to-be-detected image refers to an image that needs to be subjected to target detection. The to-be-detected target refers to a target that needs to be identified and located. The to-be-detected target can be a pedestrian, a vehicle, an animal, an object, etc. in the image.

[0041] Further, the to-be-detected target is set according to actual needs, and in the embodiment, the to-be-detected target is not limited. Further, the prompt text of the to-be-detected target refers to the text description information of the target expected to be detected. It should be noted that the to-be-detected target is different, and the prompt text of the to-be-detected target is also different. For example, in a possible implementation, the to-be-detected image is picture 1, picture 1 is a picture of many people dancing in the square, the to-be-detected target is a woman dancing in a red dress, and the prompt text of the to-be-detected target can be a woman dancing in a red dress.

[0042] In a specific implementation, feature extraction can be performed on the to-be-detected image to obtain a plurality of scale feature maps. Feature extraction is also performed on the prompt text of the to-be-detected target to obtain text features.

[0043] Further, traditional methods can be used for feature extraction, or neural network-based methods can be used for feature extraction, and in the embodiment, the methods are not limited. For example, in a possible implementation, when feature extraction is performed on the to-be-detected image, SIFT (Scale-Invariant Feature Transform, SIFT for short), HOG (Histogram of Oriented Gradients, HOG for short), convolutional neural network (CNN), and the like can be used for feature extraction. For another example, in another possible implementation, for the prompt text of the to-be-detected target, a bag-of-words model, term frequency-inverse document frequency, word embedding, a neural network model, and the like can be used for feature extraction.

[0044] It should be noted that the number of the plurality of scale feature maps is set according to actual needs, and in the embodiment, the number is not limited. In addition, the specific dimensions (specific scales) of each scale feature map are also set according to actual needs, and in the embodiment, the specific dimensions (specific scales) are not limited. For example, in a possible implementation, the plurality of scale feature maps includes three scale feature maps, and for the convenience of description, the three scale feature maps are denoted as feature Figure 1 (scale 1), feature Figure 2 (scale 2), and feature Figure 3 (scale 3), respectively; wherein the dimension of feature Figure 1 is H*W*C, the dimension of feature Figure 2 is H / 2*W / 2*C / 2, and the dimension of feature Figure 3 is H / 4*W / 4*C / 4. In addition, the dimension of the text features is set according to actual needs, and in the embodiment, the dimension is not limited.

[0045] In S102, a self-attention feature of the text features is generated through a self-attention mechanism.

[0046] Specifically, the self-attention mechanism gives different weights to different positions of elements in the text features by calculating the relevance between different positions of the text features, so as to better capture global dependencies. In the self-attention mechanism, each element in the text features can interact with other elements in the text features, and then generate self-attention features of the text features. The specific implementation principle and implementation process of the self-attention mechanism can be referred to the description in the related art, which will not be described here.

[0047] S103, for each scale of the feature map, mapping the feature map of the scale to the weight space of the convolution kernel to generate the weight of the dynamic convolution kernel corresponding to the scale.

[0048] Specifically, for each scale of the feature map, the weight of the dynamic convolution kernel corresponding to the scale can be generated according to the feature map of the scale. It should be noted that the size and weight of the dynamic convolution kernel corresponding to a certain scale are generated based on the feature map of the scale, which is adapted to the feature map of the scale. The size and weight of the dynamic convolution kernel corresponding to different scales generated based on different scales of feature maps are different. The size and weight of the dynamic convolution kernel corresponding to different scales change with the feature map, and the size and weight of the dynamic convolution kernel can be dynamically adjusted according to the feature map. For example, in one possible implementation, the larger the scale of the feature map, the larger the size of the dynamic convolution kernel under the scale.

[0049] In combination with the above example, for the convenience of description, the dynamic convolution kernels corresponding to the three scales are respectively denoted as dynamic convolution kernel K1 (dynamic convolution kernel corresponding to scale 1), dynamic convolution kernel K2 (dynamic convolution kernel corresponding to scale 2), and dynamic convolution kernel K3 (dynamic convolution kernel corresponding to scale 3).

[0050] Further, in combination with the above example, in this step, the feature Figure 1 is mapped to the weight space of the convolution kernel to generate the weight of the dynamic convolution kernel K1 corresponding to scale 1; the feature Figure 2 is mapped to the weight space of the dynamic convolution kernel to generate the weight of the dynamic convolution kernel K2 corresponding to scale 2; and the feature Figure 3 is mapped to the weight space of the dynamic convolution kernel to generate the weight of the dynamic convolution kernel K3 corresponding to scale 3. For example, Figure 2 is a schematic diagram of the dynamic convolution kernels corresponding to different scales shown by an exemplary embodiment of the present application. Please refer to Figure 2For example, in a possible implementation, the relevant information of the convolution kernel K1 is size=7, padding=3, strike=1; the relevant information of the convolution kernel K2 is size=5, padding=2, strike=1; and the size of the convolution kernel K3 is size=3, padding=1, strike=1.

[0051] S104, convolve the feature map of the scale with the dynamic convolution kernel corresponding to the scale to obtain a dynamic convolution feature at the scale.

[0052] In a specific implementation, for a feature map of a certain scale, the dynamic convolution kernel corresponding to the scale is applied to the feature map of the scale, the feature map of the scale is convolved with the dynamic convolution kernel corresponding to the scale, and then the obtained convolution result is determined as the dynamic convolution feature at the scale.

[0053] In combination with the above example, in this step, the feature map F1 is convolved with the dynamic convolution kernel K1 to obtain the dynamic convolution feature 11 at scale 1; similarly, the feature map F2 is convolved with the dynamic convolution kernel K2 to obtain the dynamic convolution feature 21 at scale 2; and the feature map F3 is convolved with the dynamic convolution kernel K3 to obtain the dynamic convolution feature 31 at scale 3. Figure 1 Figure 2 Figure 3

[0054] It should be noted that for a feature map of each scale, the weights of the dynamic convolution kernel corresponding to the scale are generated by mapping the feature map of the scale to the weight space of the convolution kernel, and then the feature map of the scale is convolved with the dynamic convolution kernel corresponding to the scale to obtain the dynamic convolution feature at the scale. In this way, the dynamic convolution kernel is adaptively calculated for feature maps of different scales, which helps to reduce unnecessary calculations while maintaining efficient feature extraction.

[0055] For example, in a specific implementation, the size of the convolution kernel can be adaptively adjusted. When processing a feature map of a larger scale, the dynamic convolution kernel can select a convolution kernel with a larger receptive field, and when processing a feature map of a smaller scale, the dynamic convolution kernel can select a convolution kernel with a smaller receptive field. In this way, compared with processing all scale feature maps with a fixed convolution kernel, the dynamic convolution kernel can dynamically adjust the calculation amount according to the feature map, avoid unnecessary calculation overhead, and reduce the calculation amount.

[0056] S105, utilize the multiple dynamic convolution features at the multiple scales and the self-attention feature to perform target detection to obtain a detection result.

[0057] ​​​In a specific implementation, the multiple dynamic convolution features at multiple scales can be taken as multi-scale image features, and the self-attention feature can be taken as a text feature. Then, target detection is performed based on the multi-scale image features and the text feature to obtain a detection result. The detection result can include a bounding box and category information of the detected target, i.e., the detection result includes the bounding box and the category information of the target to be detected.

[0058] As introduced above, the image to be detected refers to an image that needs to be subjected to target detection, the target to be detected refers to a target that is expected to be detected, and the prompt text of the target to be detected refers to text description information of the target to be detected. For example, in a possible implementation, the prompt text of the target to be detected can be category description information of the target to be detected (for example, in an embodiment, the prompt text is: detect a gun in the image). For another example, in another possible implementation, the prompt text of the target to be detected can be attribute description information of the target to be detected (for example, in an embodiment, the prompt text is: detect a woman in red dancing).

[0059] Further, after target detection is performed on the image to be detected in combination with the prompt text of the target to be detected, the position information of the target to be detected in the image to be detected can be recognized and located, and the position information can be represented by a bounding box, i.e., the detection result includes the bounding box of the target to be detected.

[0060] Further, in combination with the above introduction, it can be understood that in a security monitoring scenario, the image to be detected can be an image captured in real time by a security monitoring camera, and the prompt text can be: please detect a knife in the image. At this time, through target detection, the knife in the image can be recognized and located. For another example, in an autonomous driving scenario, the image to be detected can be a real-time road scene image captured by a camera on an autonomous driving vehicle, and the real-time road scene image contains real-time visual information of the environment around the vehicle, such as a road, a pedestrian, a vehicle, and a traffic signal. Further, the prompt text can be: please detect a traffic signal, a road marking, a vehicle, and a pedestrian. At this time, through target detection, the traffic signal, the road marking, the vehicle, and the pedestrian in the real-time road scene image can be detected, i.e., at this time, the detection result includes the position information of each detected target and the category information of the target.

[0061] The method provided in this embodiment, after obtaining the feature maps of multiple scales, for each scale of feature map, the weight of the dynamic convolution kernel corresponding to the scale is generated by mapping the feature map of the scale to the weight space of the convolution kernel, and then the feature map of the scale is convoluted by using the dynamic convolution kernel corresponding to the scale, to obtain the dynamic convolution feature under the scale. In this way, the size and weight of the dynamic convolution kernel under the scale can be dynamically adjusted according to the scale of the feature map, and the dynamic convolution kernel is adaptively calculated for feature maps of different scales, which helps to reduce unnecessary calculations while maintaining efficient feature extraction, significantly reducing the amount of calculation, so that the target detection method can be adapted to edge devices with lower computing power.

[0062] Figure 3 The flowchart of the second embodiment of the target detection method provided in this application is provided. Please refer to Figure 3 On the basis of the above-mentioned embodiments, the target detection method provided in this embodiment is realized based on a pre-trained target detection model, the target detection model includes an encoder, the encoder includes a dynamic depth separable convolution module, the dynamic depth separable convolution module includes a first pointwise convolution module and a weight generation module; wherein the mapping of the feature map of the scale to the weight space of the convolution kernel to generate the weight of the dynamic convolution kernel corresponding to the scale includes:

[0063] S301, using the first pointwise convolution module to process the feature map of the scale to obtain a mixed feature map that fuses multi-channel information.

[0064] Specifically, Figure 4 The schematic diagram of the target detection model shown in an exemplary embodiment of this application is shown. Please refer to Figure 4 The target detection model includes a feature extraction module, which can extract features from the to-be-detected image and the prompt text through the feature extraction module. Further, in a possible implementation manner, the feature extraction module can include an image backbone network and a text backbone network, and the image backbone network can be used to extract features from the to-be-detected image to obtain feature maps of multiple scales; similarly, the text backbone network is used to extract features from the prompt text to obtain text features.

[0065] It should be noted that the image backbone network can be a pre-trained image backbone network such as ResNet, Swin Transformer, ViT, etc., which is not limited in this embodiment. In addition, the text backbone network can be Bert, CLIP, etc., which is not limited in this embodiment. The specific implementation principle and implementation process of the image backbone network and the text backbone network can be referred to the description in related technologies, which will not be repeated here.

[0066] Further, please continue to refer to Figure 4The object detection model also includes an encoder, which includes a self-attention module connected to the text backbone network. The self-attention module can be used to process text features and output self-attention features of the text features.

[0067] For further details, please refer to [link / reference]. Figure 4 The encoder also includes a dynamic depthwise separable convolution module, which includes a first pointwise convolution module and a weight generation module. The dynamic depthwise separable convolution module can be used to generate weights for a dynamic convolution kernel corresponding to a certain scale.

[0068] The following explains the generation principle of the weights of the dynamic convolution kernel corresponding to a certain scale:

[0069] Specifically, the dynamic depthwise separable convolution module processes feature maps at various scales in a similar way. The following example, using a feature map at a specific scale, illustrates the principle behind the generation of the weights for the dynamic convolution kernel at that scale. Please refer to... Figure 4 Combining the previous examples, for instance, for features Figure 1 The first pointwise convolutional module performs this feature Figure 1 The data is processed to obtain a hybrid feature map that incorporates information from multiple channels. Specifically, the first pointwise convolutional module can process the features... Figure 1 Perform depthwise separable convolution, i.e., on features Figure 1 First, perform channel convolution (using k*k convolution kernels to perform convolution operations independently on each channel, generating feature maps with the same number of input channels), then perform pointwise convolution (using 1*1 convolution kernels to linearly combine the feature maps generated by channel convolution, generating the number of output channels) to obtain a mixed feature map.

[0070] It should be noted that pointwise convolution processing refers to linearly combining the features at each location of the feature map to fuse information from multiple channels.

[0071] Combining the examples above, features Figure 1 The dimension is H*W*C, meaning the size of the feature map input to the first pointwise convolutional module is H*W*C. For example, in one embodiment, the size of the convolutional kernel in the first pointwise convolutional module is 1*1*C*C. out Convolution kernel for features Figure 1 After performing pointwise convolution, the features are obtained. Figure 1 The corresponding blended feature map has dimensions H*W*C. out .

[0072] Based on the above introduction, it can be understood that the location of the blended feature map (H, W, C) out The value of ) is calculated using the following formula:

[0073]

[0074] Y is a mixed feature map;

[0075] X is a feature map;

[0076] K is a convolution kernel.

[0077] It should be noted that, compared with the standard convolution, the parameter amount of the point-wise convolution is significantly reduced, and the calculation is significantly reduced. For example, a standard 3*3 convolution kernel has 3*3*C in *C out parameters on C in *C out channels, and the point-wise convolution has only 1*1*C in *C out parameters.

[0078] Optionally, Figure 5 is a schematic diagram of a first point-wise convolution module shown in an exemplary embodiment of the present application. Please refer to Figure 5 , in a possible implementation manner, the first point-wise convolution module includes a channel convolution layer, a first batch normalization layer and a first activation function layer; wherein the first point-wise convolution module is used to process the feature map of the scale to obtain a mixed feature map fused with multi-channel information, including:

[0079] (1) using the channel convolution layer to perform point-wise convolution on the feature map of the scale to obtain a first feature map.

[0080] In the specific implementation, combined with the above example, for example, for the feature Figure 1 , a 1*1*C in *C out convolution kernel can be used to perform point-wise convolution operation on the feature Figure 1 to obtain the first feature map. It can be understood that the dimension of the first feature map is H*W*C out .

[0081] It should be noted that the dimension of the first feature map can be the same as the dimension of the feature map of the scale, or can be different, that is, the number of output channels can be the same as the number of input channels, or can be different. In this embodiment, it is not limited. For example, when the dimension of the first feature map is the same as the dimension of the feature map of the scale, the dimension of the first feature map is H*W*C.

[0082] (2) using the first batch normalization processing layer to perform batch normalization processing on the first feature map to obtain a second feature map.

[0083] Specifically, the batch normalization processing refers to a process of performing normalization processing on the first feature map. In a specific implementation, for each batch of input data, the mean and variance thereof are calculated, and then the input data is normalized using the mean and variance of the batch, and then the normalized data is scaled and translated. For specific implementation principles and implementation processes of batch normalization processing, please refer to the description in the related art, which will not be repeated here.

[0084] It should be noted that the dimensions of the input and output of the first batch normalization processing layer can be the same or different, and in the present embodiment, no limitation is made thereto.

[0085] In the following, taking the case where the dimensions of the input and output of the first batch normalization processing layer are the same as an example, in combination with the above example, the dimension of the second feature map is H*W*C.

[0086] (3) performing activation processing on the second feature map using the first activation function layer to obtain the mixed feature map.

[0087] Specifically, the first activation function layer performs activation processing on the second feature map through an activation function. It can be understood that the activation function can be a ReLU function (Rectified Linear Unit), a Sigmoid function, a Tanh function, etc., and in the present embodiment, no limitation is made thereto.

[0088] It should be noted that the dimensions of the input and output of the first activation function layer can be the same or different, and in the present embodiment, no limitation is made thereto.

[0089] For example, in a possible implementation, when the dimensions of the input and output of the first activation function layer are the same, in combination with the above example, the dimension of the mixed feature map is H*W*C.

[0090] The method provided in the present embodiment gives a specific structure diagram of a first point-by-point convolution module. Through the first point-by-point convolution module, not only can each scale of feature map be processed to obtain a mixed feature map that fuses multi-channel information, but also the calculation amount can be further reduced.

[0091] S302, input the mixed feature map into the weight generation module, and generate the weight of the dynamic convolution kernel corresponding to the scale according to the mixed feature map by the weight generation module.

[0092] In a specific implementation, a neural network can be used to generate the weight of the dynamic convolution kernel corresponding to the scale, for example, a fully connected layer is used to map the mixed feature map to the weight space of the dynamic convolution kernel, and reshaped into the shape of the convolution kernel. Or use a set of linear networks to dynamically generate the weight of the dynamic convolution kernel.

[0093] In combination with the above description, the weight of the dynamic convolution kernel corresponding to the scale can be generated according to the following process, for example:

[0094] W = WeightGenerator (F)

[0095] Wherein, W is the weight of the dynamic convolution kernel corresponding to the scale, and F is the feature map of the scale.

[0096] Optionally, Figure 6 For a schematic diagram of the weight generation module shown in an exemplary embodiment of the present application, please refer to Figure 6 In a possible implementation, the weight generation module includes a pooling layer, a channel dimension reduction convolution layer, a second batch normalization processing layer, a second activation function layer, a channel dimension increase convolution layer and a shape remodeling layer; wherein, the inputting the mixed feature map into the weight generation module, and the weight generation module generating the weight of the dynamic convolution kernel corresponding to the scale according to the mixed feature map, includes:

[0097] (1) The pooling layer is used to perform average pooling processing on each channel of the mixed feature map to obtain a pooling processing result.

[0098] Specifically, in combination with the above description, it can be understood that the mixed feature map is a multi-channel feature map. For example, in the above example, when the dimension of the mixed feature map is H*W*C, it is a multi-channel feature map with C channels, and the size of the sub-feature map corresponding to each channel is H*W.

[0099] It should be noted that in this embodiment, the average pooling processing generates the down-sampled sub-feature map by sliding a fixed-size window on the sub-feature map corresponding to a channel and calculating the average value of all values in the window.

[0100] Referring to the above example, in the specific implementation, for the sub-feature map corresponding to a channel, the H*W elements on the sub-feature map can be subjected to average pooling processing to obtain the processing result (which contains 1*1 elements) corresponding to the channel. Further, the processing results corresponding to all channels are spliced together, which is the final pooling processing result. In combination with the above example, it can be understood that in this embodiment, the dimension of the pooling processing result is 1*1*C.

[0101] (2) The channel dimension reduction convolution layer is used to perform dimension reduction processing on the pooling processing result to obtain a channel dimension reduction result.

[0102] Specifically, the channel dimension reduction convolution is used to reduce the number of channels of the feature map while keeping the spatial dimension of the feature map unchanged. In the specific implementation, point-by-point convolution can be used to achieve the purpose of reducing the number of channels of the feature map.

[0103] It should be noted that the degree of reduction of the number of channels is set according to actual needs, and in the embodiment, the degree of reduction of the number of channels is not limited. In the following, the degree of reduction of the number of channels is taken as 1 / 4 as an example. Referring to the above examples, for example, in a possible implementation manner, the dimension of the pooling processing result is 1*1*C, and after the dimension reduction processing is performed on the pooling processing result, the dimension of the channel dimension reduction result can be 1*1*C / 4. In the specific implementation, a convolution kernel with a size of 1*1*C*C / 4 can be used to perform point-by-point convolution on the pooling processing result to achieve the purpose of dimension reduction.

[0104] (3) performing batch normalization processing on the channel dimension reduction result by using the second batch normalization processing layer to obtain a batch normalization processing result.

[0105] The specific implementation principle of the batch normalization processing can be referred to the foregoing description, and details are not described herein.

[0106] It should be noted that the dimension of the input and the output of the second batch normalization processing layer can be the same or different, and in the embodiment, the dimension of the input and the output of the second batch normalization processing layer is not limited. In the following, the dimension of the input and the output of the second batch normalization processing is taken as the same as an example. Referring to the above examples, for example, in a possible implementation manner, the dimension of the channel dimension reduction result input to the second batch normalization processing layer is 1*1*C / 4, and the dimension of the batch normalization processing result output by the second batch normalization processing layer is also 1*1*C / 4.

[0107] (4) performing activation processing on the batch normalization processing result by using the second activation function layer to obtain an activation processing result.

[0108] Specifically, the second activation function layer performs activation processing on the batch normalization processing result by using an activation function. It can be understood that the activation function can be a ReLU function (Rectified Linear Unit), a Sigmoid function, a Tanh function, or the like, and in the embodiment, the activation function is not limited.

[0109] It should be noted that the dimension of the input and the output of the second activation function layer can be the same or different, and in the embodiment, the dimension of the input and the output of the second activation function layer is not limited. In the following, the dimension of the input and the output of the second activation function layer is taken as the same as an example. Referring to the above examples, for example, in a possible implementation manner, the dimension of the batch normalization processing result input to the second activation function layer is 1*1*C / 4, and the dimension of the activation processing result output by the second activation function layer is also 1*1*C / 4.

[0110] (5) performing dimension increasing processing on the activation processing result by using the channel dimension increasing convolution layer to obtain a dimension increasing processing result.

[0111] Specifically, the channel dimension increasing convolution is used to increase the channel number of the feature map while keeping the spatial dimension of the feature map unchanged. In a specific implementation, the point-wise convolution can be used to achieve the purpose of increasing the channel number of the feature map.

[0112] It should be noted that the degree of increasing the channel number is determined according to actual needs, and is not limited in the embodiment.

[0113] Further, referring to the above examples, for example, in a possible implementation, the dimension of the activation processing result output by the second activation function layer is 1*1*C / 4, and after the point-wise convolution processing is performed on the activation processing result by using a 1*1*C / 4*36C convolution kernel, the dimension of the channel dimension increasing result is 1*1*9C, and the channel number of the feature map is increased.

[0114] For another example, in a possible implementation, the dimension of the activation processing result output by the second activation function layer is 1*1*C / 4, and after the point-wise convolution processing is performed on the activation processing result by using a 1*1*C / 4*64C convolution kernel, the dimension of the channel dimension increasing result is 1*1*16C, and the channel number of the feature map is increased.

[0115] (6) using the shape remodeling layer to remodel the dimension increasing processing result into the shape of the convolution kernel, to obtain the weight of the dynamic convolution kernel corresponding to the feature map of the scale.

[0116] Specifically, shape remodeling refers to converting a multi-dimensional array into an array with different dimensions while keeping the total number of elements unchanged. In a specific implementation, the Reshape operation can be used to remodel the dimension increasing processing result into the shape of the convolution kernel.

[0117] Further, referring to the above examples, for example, in the first example, the dimension of the channel dimension increasing result is 1*1*9C, and after the shape remodeling is performed on the channel dimension increasing result, the size of the remodeled convolution kernel is C*1*3*3.

[0118] For another example, in the second example, the dimension of the channel dimension increasing result is 1*1*16C, and after the shape remodeling is performed on the channel dimension increasing result, the size of the remodeled convolution kernel is C*1*4*4.

[0119] In the embodiment, a specific structure of the weight generation module is given, and through the weight generation module, the weight of the dynamic convolution kernel can be dynamically generated, so as to perform the convolution processing on the feature map based on the dynamic convolution kernel, thereby reducing the calculation amount.

[0120] The method provided in the embodiment can utilize the first point-by-point convolution module to process the feature map of the scale, obtain a mixed feature map fused with multi-channel information, and input the mixed feature map into the weight generation module, so as to generate the weight of the dynamic convolution kernel corresponding to the scale by the weight generation module according to the mixed feature map. In this way, the convolution operation can be adaptively adjusted for the feature maps of different scales, redundant calculation caused by a fixed convolution kernel can be effectively avoided, the calculation amount can be reduced, and the calculation efficiency can be improved. In addition, the standard convolution is decomposed into a channel-by-channel channel convolution and a point-by-point convolution by the dynamic depth separable convolution module, so that the parameter amount and the calculation amount can be reduced.

[0121] Optionally, please continue to refer to Figure 4 In Figure 4 In the example shown in the figure, the dynamic depth separable convolution module further includes a convolution processing layer; wherein the feature map of the scale is convolved by using the dynamic convolution kernel corresponding to the scale, to obtain the dynamic convolution feature under the scale, including:

[0122] The convolution processing layer convolves the mixed feature map output by the first point-by-point convolution module by using the dynamic convolution kernel corresponding to the scale, to obtain the dynamic convolution feature under the scale.

[0123] Specifically, for example, for the feature Figure 1 The feature Figure 1 The mixed feature map corresponding to the feature is denoted as F1, and the dynamic convolution kernel corresponding to the scale determined based on the feature map is denoted as B1. As introduced above, the dimension of F1 is H*W*C, and the dimension of B1 is C*1*3*3. In this step, F1 is convolved by using B1, to obtain the dynamic convolution feature under the scale.

[0124] It can be understood that the first point-by-point convolution module can fuse the multi-channel original feature into a mixed feature map, can capture more rich and abstract feature information, and improves the expression ability of the feature. Therefore, when convolving, the mixed feature map output by the first point-by-point convolution module is convolved by using the dynamic convolution kernel. Since the mixed feature map after the point-by-point convolution has fused the multi-channel information of the original feature, the dynamic convolution kernel can perform feature extraction on a higher dimension when performing convolution on the mixed feature map, further enhances the expression ability and accuracy of the feature.

[0125] Figure 7 The flowchart of the target detection method embodiment three provided in the present application. Please continue to refer to Figure 4On the basis of the above-mentioned embodiments, the encoder further comprises a cross-attention module, and the target detection model further comprises a decoder; wherein the target detection is performed by using the multiple dynamic convolution features at the multiple scales and the self-attention feature, comprising:

[0126] S701, processing the multiple dynamic convolution features and the self-attention feature by the cross-attention module to obtain text feature representation and image feature representation.

[0127] Specifically, Figure 8 The schematic diagram of the cross-attention module shown in an exemplary embodiment of the present application. Please refer to Figure 8 , the cross-attention module comprises a first cross-attention layer, a second cross-attention layer, a first feedforward network layer and a second feedforward network layer; wherein the processing of the multiple channel convolution features and the self-attention feature by the cross-attention module to obtain text feature representation and image feature representation comprises the following steps:

[0128] (1) The first cross-attention layer performs attention calculation by taking the image feature as the query and the self-attention feature as the key and the value to obtain the first fusion feature; wherein the image feature is the multiple dynamic convolution features.

[0129] Specifically, in the first cross-attention layer, the image feature is used for query, and the self-attention feature extracted from the text feature is used as the key and the value. The first cross-attention layer performs attention calculation by the similarity between the multiple dynamic convolution features and the self-attention feature to obtain the attention weight, and then performs weighted summation on the self-attention feature according to the obtained attention weight to output the first fusion feature.

[0130] (2) The second cross-attention layer performs attention calculation by taking the self-attention feature as the query and the first fusion feature as the key and the value to obtain the second fusion feature.

[0131] It should be noted that the second cross-attention layer has the same structure as the first cross-attention layer.

[0132] Specifically, the second cross-attention layer takes the self-attention feature as the query and the first fusion feature as the key and the value, and performs attention calculation by the similarity between the self-attention feature and the first fusion feature to obtain the attention weight, and then performs weighted summation on the first fusion feature according to the obtained attention weight to output the second fusion feature.

[0133] (3) The first feedforward network layer generates the text feature representation according to the first fusion feature.

[0134] Specifically, the first feedforward network layer is configured to further process the first fused feature to generate a higher-level text feature representation through a series of operations.

[0135] (4) The second feedforward network layer is configured to generate the image feature representation according to the second fused feature.

[0136] It should be noted that the second feedforward network layer has the same structure as the first feedforward network layer.

[0137] Specifically, the second feedforward network layer is configured to further process the second fused feature to generate a higher-level image feature representation through a series of operations.

[0138] S702, decoding the text feature representation and the image feature representation through the decoder to obtain a detection result.

[0139] In a specific implementation, the decoder calculates the similarity between the text feature representation and the image feature representation, then selects Topk query vectors from the image feature representation based on the similarity, and finally outputs the final detection result based on the query vectors and the text feature representation.

[0140] The method provided in this embodiment gives a specific method of target detection using multiple dynamic convolution features and self-attention features at multiple scales. Through this method, the target to be detected in the image to be detected can be accurately identified.

[0141] Figure 9 The flowchart of the target detection method embodiment four provided in this application is shown in the following figure. Figure 10 The schematic diagram of the target detection model shown in another exemplary embodiment of this application is shown in the following figure. Figure 10 In the example shown in the following figure, the encoder further includes a second pointwise convolution module and a cross-attention module, and the target detection model further includes a decoder. Figure 10 The target detection model shown in the example includes a first pointwise convolution module and a cross-attention module.

[0142] S901, for the dynamic convolution feature at each scale, processing the dynamic convolution feature at each scale using the second pointwise convolution module to obtain a channel convolution feature at each scale.

[0143] It should be noted that the structure and function of the second pointwise convolution module are the same as those of the first pointwise convolution module. In this embodiment, it will not be described again.

[0144] As can be understood from the foregoing, the second point-by-point convolution module can fuse the dynamic convolution features at each scale into the channel convolution features at the scale, can capture richer and more abstract feature information, and improve the expression ability of the features. Through the second point-by-point convolution module, higher-level channel convolution features can be generated.

[0145] S902, processing the plurality of channel convolution features and the self-attention feature through the cross-attention module to obtain a text feature representation and an image feature representation.

[0146] Specifically, please continue to refer to Figure 8 The cross-attention module in this step has the same structure and processing process as the cross-attention module in step S701 described above, and will not be described here.

[0147] S903, decoding the text feature representation and the image feature representation through the decoder to obtain a detection result.

[0148] Specifically, the specific implementation process and implementation principle of this step can be referred to the description in the foregoing embodiments, and will not be described here.

[0149] The method provided in this embodiment gives a specific method of target detection using a plurality of dynamic convolution features at multiple scales and a self-attention feature. Through this method, the target to be detected in the image to be detected can be accurately identified. In addition, in this embodiment, for the dynamic convolution features at each scale, the second point-by-point convolution module is used to process the dynamic convolution features to obtain the channel convolution features at each scale, and then the channel convolution features are sent to the cross-attention module for processing. In this way, since the second point-by-point convolution module can capture richer and more abstract feature information and improve the expression ability of the features, the subsequent processing can be more accurate, and the accuracy of target detection can be further improved.

[0150] Corresponding to the foregoing embodiment of the target detection method, the present application also provides an embodiment of a target detection device.

[0151] The embodiment of the target detection device of the present application can be applied to a target detection device. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by the processor of the target detection device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for execution. From the hardware level, as shown in Figure 11 the hardware structure diagram of the target detection device in which the target detection device provided by the present application is located, in addition to Figure 11In addition to the processor, the memory, the network interface, and the non-volatile memory shown, the target detection device in which the apparatus in the embodiment is located can also include other hardware according to the actual functions of the target detection apparatus, which will not be described again.

[0152] Figure 12 A structural schematic diagram of an embodiment of the target detection apparatus provided in the present application is shown. Please refer to Figure 12 The apparatus provided in the embodiment includes an extraction module 121, a generation module 122, and a detection module 123, wherein

[0153] The extraction module 121 is configured to perform feature extraction on a to-be-detected image and a prompt text of a to-be-detected target respectively, to obtain a plurality of scale feature maps and text features.

[0154] The generation module 122 is configured to generate a self-attention feature of the text features through a self-attention mechanism.

[0155] The generation module 122 is further configured to, for each scale feature map, map the scale feature map to a weight space of a convolution kernel, to generate a weight of a dynamic convolution kernel corresponding to the scale.

[0156] The generation module 122 is further configured to perform convolution processing on the scale feature map by using the dynamic convolution kernel corresponding to the scale, to obtain a dynamic convolution feature under the scale.

[0157] The detection module 123 is configured to perform target detection by using a plurality of dynamic convolution features under a plurality of scales and the self-attention feature, to obtain a detection result.

[0158] The apparatus in the embodiment can be used to execute Figure 1 The steps of the method embodiment shown, the specific implementation principle and the implementation process are similar, which will not be described again here.

[0159] Please continue to refer to Figure 11 The present application also provides a target detection device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the program to implement the steps of any one of the methods provided in the first aspect of the present application.

[0160] It should be noted that in a possible implementation, the target detection device can be an edge low-computing-power device, wherein the edge low-computing-power device refers to a device deployed at the network edge, which has relatively low computing power and energy consumption.

[0161] The present application also provides a computer-readable storage medium having a computer program stored thereon, and the program is executed by a processor to implement the steps of any one of the methods provided in the present application.

[0162] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0163] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can be seen from the part of the method embodiment. The device embodiment described above is only schematic, and the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or also distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the scheme of the present application according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0164] The above description is only the preferred embodiment of the present application, and is not used to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of protection of the present application.

Claims

1. A target detection method, characterized in that, The method includes: Feature extraction is performed on the image to be detected and the prompt text of the target to be detected separately to obtain feature maps and text features at multiple scales; The text features are generated using a self-attention mechanism; For each scale of feature map, the feature map of that scale is mapped to the weight space of the convolution kernel to generate the weights of the dynamic convolution kernel corresponding to that scale. By using the dynamic convolution kernel corresponding to this scale to perform convolution processing on the feature map of this scale, dynamic convolution features at this scale are obtained. Target detection is performed using multiple dynamic convolutional features at multiple scales and the self-attention features to obtain detection results.

2. The method according to claim 1, characterized in that, The object detection method is implemented based on a pre-trained object detection model. The object detection model includes an encoder, which includes a dynamically depthwise separable convolutional module. This dynamically depthwise separable convolutional module includes a first pointwise convolutional module and a weight generation module. The step of mapping the feature map at that scale to the weight space of the convolutional kernel to generate the weights of the dynamically convolutional kernel corresponding to that scale includes: The feature map at this scale is processed using the first pointwise convolution module to obtain a hybrid feature map that incorporates multi-channel information; The hybrid feature map is input into the weight generation module, which then generates the weights of the dynamic convolution kernel corresponding to that scale based on the hybrid feature map.

3. The method according to claim 2, characterized in that, The first pointwise convolutional module includes a channel convolutional layer, a first batch normalization layer, and a first activation function layer; wherein, the process of using the first pointwise convolutional module to process the feature map at this scale to obtain a hybrid feature map that integrates multi-channel information includes: The first feature map is obtained by performing point-by-point convolution on the feature map at this scale using the channel convolutional layer. The first feature map is batch-normalized using the first batch of normalization processing layers to obtain the second feature map; The second feature map is activated using the first activation function layer to obtain the hybrid feature map.

4. The method according to claim 2 or 3, characterized in that, The weight generation module includes a pooling layer, a channel-reduced convolutional layer, a second batch normalization layer, a second activation function layer, a channel-incremented convolutional layer, and a shape reshaping layer; wherein, the step of inputting the mixed feature map into the weight generation module, and the weight generation module generating weights for the dynamic convolutional kernel corresponding to that scale based on the mixed feature map, includes: The pooling layer is used to perform average pooling on each channel of the hybrid feature map to obtain the pooling result. The pooling result is reduced in dimensionality using the channel-reducing convolutional layer to obtain the channel-reduced result. The second batch standardization layer is used to perform batch standardization on the channel dimensionality reduction result to obtain the batch standardization result. The batch normalization result is activated using the second activation function layer to obtain the activation result. The activation processing result is then subjected to dimensionality upscaling using the aforementioned channel-based dimensionality upscaling convolutional layer to obtain the dimensionality upscaling result. The shape reshaping layer is used to reshape the dimensionality increase result into the shape of the convolution kernel, thereby obtaining the weights of the dynamic convolution kernel corresponding to that scale.

5. The method according to claim 2, characterized in that, The dynamically depth-separable convolutional module further includes a convolutional processing layer; wherein, the feature map at this scale is convolved using a dynamic convolutional kernel corresponding to this scale to obtain the dynamic convolutional features at this scale, including: The convolutional processing layer uses the dynamic convolutional kernel corresponding to this scale to perform convolution processing on the hybrid feature map output by the first pointwise convolutional module to obtain the dynamic convolutional features at this scale.

6. The method according to claim 2 or 5, characterized in that, The encoder further includes a cross-attention module, and the object detection model further includes a decoder; wherein, the object detection using multiple dynamic convolutional features at multiple scales and the self-attention features includes: The cross-attention module processes the multiple dynamic convolutional features and the self-attention features to obtain text feature representations and image feature representations. The detection result is obtained by decoding the text feature representation and the image feature representation using the decoder.

7. The method according to claim 2 or 5, characterized in that, The encoder further includes a second pointwise convolutional module and a cross-attention module, and the object detection model further includes a decoder; wherein, the object detection using multiple dynamic convolutional features at multiple scales and the self-attention features includes: For the dynamic convolutional features at each scale, the second pointwise convolution module is used to process the dynamic convolutional features to obtain the channel convolutional features at each scale. The cross-attention module processes the convolutional features of the multiple channels and the self-attention features to obtain text feature representations and image feature representations. The detection result is obtained by decoding the text feature representation and the image feature representation using the decoder.

8. The method according to claim 6 or 7, characterized in that, The cross-attention module includes a first cross-attention layer, a second cross-attention layer, a first feedforward network layer, and a second feedforward network layer; wherein, the process of processing the multiple channel convolutional features and the self-attention features through the cross-attention module to obtain text feature representations and image feature representations includes: The first cross-attention layer uses image features as queries and the self-attention features as keys and values ​​to perform attention calculations to obtain the first fusion feature; wherein, the image features are the multiple dynamic convolution features or the multiple channel convolution features; The second cross-attention layer uses the self-attention feature as the query and the first fusion feature as the key and value to perform attention calculation, thereby obtaining the second fusion feature; The first feedforward network layer generates the text feature representation based on the first fusion feature; The second feedforward network layer generates the image feature representation based on the second fusion feature.

9. A target detection device, characterized in that, The device includes an extraction module, a generation module, and a detection module; wherein, The extraction module is used to extract features from the image to be detected and the prompt text of the target to be detected, respectively, to obtain feature maps and text features at multiple scales; The generation module is used to generate self-attention features of the text features through a self-attention mechanism; The generation module is also used to map the feature map at each scale to the weight space of the convolution kernel for each scale, and generate the weights of the dynamic convolution kernel corresponding to that scale. The generation module is also used to perform convolution processing on the feature map of the scale using the dynamic convolution kernel corresponding to the scale, so as to obtain the dynamic convolution feature at the scale. The detection module is used to perform target detection using multiple dynamic convolutional features at multiple scales and the self-attention features, and obtain detection results.

10. A target detection device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the program, implements the steps of the method according to any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1-8.

Images