A target detection method, apparatus and device

Abstract

The application provides a target detection method, device and equipment. The method provided by the application performs feature extraction on a to-be-detected image and prompt text of a to-be-detected target respectively to obtain a plurality of scale feature maps and text features; a self-attention feature of the text features is generated through a self-attention mechanism; for each scale feature map, the feature map of the scale is mapped to a weight space of a convolution kernel to generate a weight of a dynamic convolution kernel corresponding to the scale; the feature map of the scale is processed through convolution by using the dynamic convolution kernel corresponding to the scale to obtain a dynamic convolution feature under the scale; and target detection is performed by using a plurality of dynamic convolution features under the plurality of scales and the self-attention feature to obtain a detection result. The target detection method, device and equipment provided by the application can reduce the calculation amount of target detection, improve the applicability of the target detection algorithm, and make the target detection algorithm applicable to edge low-computing-power devices.

Description

Technical Field

[0001] This application relates to the field of target detection technology, and in particular to a target detection method, apparatus and equipment. Background Technology

[0002] With the development of deep learning and convolutional neural network technologies, the accuracy and efficiency of object detection have been significantly improved, and its application in various fields is becoming increasingly widespread. In recent years, Transformer-based object detection models have been widely used in various fields. However, Transformer-based object detection models process image features through self-attention or deformable attention mechanisms, resulting in high computational complexity and a large computational load, making them unsuitable for use in low-computing-power edge devices. Summary of the Invention

[0003] In view of this, this application provides a target detection method, apparatus, and device to reduce the computational load of target detection, thereby improving the applicability of target detection algorithms and enabling their application in edge devices with low computing power.

[0004] Specifically, this application is implemented through the following technical solution:

[0005] A first aspect of this application provides a target detection method, the method comprising:

[0006] Feature extraction is performed on the image to be detected and the prompt text of the target object to be detected separately to obtain feature maps and text features at multiple scales;

[0007] The text features are generated using a self-attention mechanism;

[0008] For each scale of feature map, the feature map of that scale is mapped to the weight space of the convolution kernel to generate the weights of the dynamic convolution kernel corresponding to that scale.

[0009] By using the dynamic convolution kernel corresponding to this scale to perform convolution processing on the feature map of this scale, dynamic convolution features at this scale are obtained.

[0010] Target detection is performed using multiple dynamic convolutional features at multiple scales and the self-attention features to obtain detection results.

[0011] A second aspect of this application provides a target detection device, the device comprising an extraction module, a generation module, a convolution module, and a detection module; wherein...

[0012] The extraction module is used to extract features from the image to be detected and the prompt text of the target to be detected, respectively, to obtain feature maps and text features at multiple scales;

[0013] The generation module is used to generate self-attention features of the text features through a self-attention mechanism;

[0014] The generation module is also used to map the feature map at each scale to the weight space of the convolution kernel for each scale, and generate the weights of the dynamic convolution kernel corresponding to that scale.

[0015] The generation module is also used to perform convolution processing on the feature map of the scale using the dynamic convolution kernel corresponding to the scale, so as to obtain the dynamic convolution feature at the scale.

[0016] The detection module is used to perform target detection using multiple dynamic convolutional features at multiple scales and the self-attention features, and obtain detection results.

[0017] A third aspect of this application provides a target detection device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods provided in the first aspect of this application.

[0018] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods provided in the first aspect of this application.

[0019] The object detection method, apparatus, and device provided in this application, after obtaining feature maps at multiple scales, generate dynamic convolution kernel weights corresponding to each scale by mapping the feature map at that scale to the weight space of the convolution kernel. Then, the dynamic convolution kernel corresponding to that scale is used to perform convolution processing on the feature map at that scale to obtain dynamic convolution features at that scale. In this way, the size and weights of the dynamic convolution kernel at that scale can be dynamically adjusted according to the scale of the feature map. The dynamic convolution kernel performs adaptive calculations for feature maps at different scales, which helps to reduce unnecessary calculations while maintaining efficient feature extraction, significantly reducing the amount of computation, and making the object detection method adaptable to edge devices with low computing power. Attached Figure Description

[0020] Figure 1 A flowchart of an embodiment of the target detection method provided in this application;

[0021] Figure 2 This is a schematic diagram illustrating dynamic convolution kernels at different scales, as shown in an exemplary embodiment of this application.

[0022] Figure 3 A flowchart of Embodiment 2 of the target detection method provided in this application;

[0023] Figure 4This is a schematic diagram illustrating a target detection model as an exemplary embodiment of this application;

[0024] Figure 5 This is a schematic diagram of a first pointwise convolution module shown in an exemplary embodiment of this application;

[0025] Figure 6 This is a schematic diagram illustrating a weight generation module as shown in an exemplary embodiment of this application;

[0026] Figure 7 A flowchart of Embodiment 3 of the target detection method provided in this application;

[0027] Figure 8 This is a schematic diagram illustrating a cross-attention module as shown in an exemplary embodiment of this application;

[0028] Figure 9 A flowchart of Embodiment 4 of the target detection method provided in this application;

[0029] Figure 10 A schematic diagram illustrating a target detection model for another exemplary embodiment of this application;

[0030] Figure 11 A hardware structure diagram of the target detection equipment containing the target detection device provided in this application;

[0031] Figure 12 This is a schematic diagram of the structure of the target detection device according to Embodiment 1 of this application. Detailed Implementation

[0032] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0033] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0034] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0035] It should be noted that the target detection method and apparatus provided in this application are applied in a target detection device, which can be an edge low-computing-power device. An edge low-computing-power device refers to a device deployed at the network edge with low computing power and low energy consumption.

[0036] Furthermore, the target detection method, apparatus, and equipment provided in this application can be applied to scenarios such as security monitoring, autonomous driving, smart homes, and industrial automation. For example, in security monitoring scenarios, it can identify suspicious persons, abandoned items, and dangerous or prohibited items. In autonomous driving scenarios, it can identify road markings, traffic lights, pedestrians, and vehicles based on prompt text and the image to be detected, thereby improving vehicle safety and navigation capabilities. In smart home scenarios, this target detection method can be used for security monitoring, object recognition, or intelligent control; for example, it can be used to locate lost items. In industrial automation scenarios, this target detection technology can be used for quality inspection, parts identification, and classification.

[0037] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0038] Figure 1 This is a flowchart of an embodiment of the target detection method provided in this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0039] S101. Perform feature extraction on the image to be detected and the prompt text of the target to be detected, respectively, to obtain feature maps and text features at multiple scales.

[0040] Specifically, the image to be detected refers to the image for which object detection needs to be performed. The target to be detected refers to the target that needs to be identified and located. The target to be detected can be pedestrians, vehicles, animals, objects, etc. in the image.

[0041] Furthermore, the target to be detected is set according to actual needs, and is not limited in this embodiment. Furthermore, the prompt text for the target to be detected refers to the textual description information of the target to be detected. It should be noted that the prompt text for the target to be detected will be different depending on the target. For example, in one possible implementation, the image to be detected is Image 1, which is a picture of many people dancing in a square, and the target to be detected is a woman dancing in red. The prompt text for the target to be detected could be "woman dancing in red."

[0042] In practice, feature extraction can be performed on the image to be detected to obtain feature maps at multiple scales. Additionally, feature extraction can be performed on the prompt text of the target to be detected to obtain text features.

[0043] Furthermore, feature extraction can be performed using traditional methods or neural network-based methods; this embodiment does not limit the specific methods used. For example, in one possible implementation, when extracting features from the image to be detected, methods such as SIFT (Scale-Invariant Feature Transform), HOG (Histogram of Oriented Gradients), and Convolutional Neural Networks (CNN) can be used. As another possible implementation, for the prompt text of the target to be detected, features can be extracted from the prompt text using a bag-of-words model, term frequency-inverse document frequency, word embeddings, and neural network models.

[0044] It should be noted that the number of feature maps at multiple scales is set according to actual needs, and is not limited in this embodiment. Furthermore, the specific dimensions (specific scales) of each feature map are also set according to actual needs, and are not limited in this embodiment. For example, in one possible implementation, the feature maps at multiple scales include feature maps at three scales. For ease of explanation, these three scale feature maps are respectively denoted as features. Figure 1 (Scale 1), Features Figure 2 (Scale 2) and features Figure 3 (Scale 3); among which, features Figure 1 The dimension size is H*W*C, and the feature Figure 2 The dimension size is H / 2*W / 2*C / 2, and the feature Figure 3 The dimension size is H / 4*W / 4*C / 4. Furthermore, the dimension of the text features is set according to actual needs; in this embodiment, it is not limited.

[0045] S102. Generate self-attention features of the text features through a self-attention mechanism.

[0046] Specifically, the self-attention mechanism calculates the correlation between various positions in text features and assigns different weights to elements at different positions to better capture global dependencies. In the self-attention mechanism, each element in the text features can interact with other elements in the text features, thereby generating self-attention features of the text features. For the specific implementation principles and processes of the self-attention mechanism, please refer to the descriptions in related technologies, which will not be repeated here.

[0047] S103. For each scale of feature map, map the feature map of that scale to the weight space of the convolution kernel to generate the weights of the dynamic convolution kernel corresponding to that scale.

[0048] Specifically, for each scale of feature map, the weights of the corresponding dynamic convolutional kernel can be generated based on the feature map at that scale. It's important to note that the size and weights of the dynamic convolutional kernel at a given scale are generated based on the feature map at that scale, and are adapted to that scale. The size and weights of the dynamic convolutional kernels generated based on feature maps of different scales are different, and they change as the feature map changes. The size and weights of the dynamic convolutional kernel can be dynamically adjusted based on the feature map. For example, in one possible implementation, the larger the scale of the feature map, the larger the size of the dynamic convolutional kernel at that scale.

[0049] Based on the example above, for ease of explanation, the dynamic convolution kernels corresponding to the three scales are respectively denoted as dynamic convolution kernel K1 (dynamic convolution kernel corresponding to scale 1), dynamic convolution kernel K2 (dynamic convolution kernel corresponding to scale 2), and dynamic convolution kernel K3 (dynamic convolution kernel corresponding to scale 3).

[0050] Furthermore, combining the examples above, this step will involve defining the features. Figure 1 Mapping to the weight space of the convolution kernel generates the weights of the dynamic convolution kernel K1 corresponding to scale 1; the features are then... Figure 2 Mapping to the weight space of the dynamic convolution kernel generates the weights of the dynamic convolution kernel K2 corresponding to scale 2; [then, the features are...] Figure 3 Mapping to the weight space of the dynamic convolution kernel generates the weights of the dynamic convolution kernel K3 corresponding to scale 3. For example, Figure 2 This is a schematic diagram illustrating dynamic convolution kernels at different scales, as shown in an exemplary embodiment of this application. Please refer to... Figure 2Combining the examples above, for instance, in one possible implementation, the relevant information for convolutional kernel K1 is size=7, padding=3, and strike=1; the relevant information for convolutional kernel K2 is size=5, padding=2, and strike=1; and the size of convolutional kernel K3 is size=3, padding=1, and strike=1.

[0051] S104. Use the dynamic convolution kernel corresponding to this scale to perform convolution processing on the feature map of this scale to obtain the dynamic convolution feature at this scale.

[0052] In practice, for a feature map of a certain scale, the dynamic convolution kernel corresponding to that scale is applied to the feature map of that scale, and the feature map of that scale is convolved using the dynamic convolution kernel corresponding to that scale. The resulting convolution result is then determined as the dynamic convolution feature at that scale.

[0053] Based on the example above, in this step, a dynamic convolutional kernel K1 is used to process the features. Figure 1 Convolution processing is performed to obtain dynamic convolutional features 11 at scale 1; similarly, dynamic convolutional kernel K2 is used to process the features. Figure 2 Convolution processing is performed to obtain dynamic convolutional features 21 at scale 2; the features are then processed using dynamic convolution kernel K3. Figure 3 Convolution processing is performed to obtain dynamic convolutional features 31 at scale 3.

[0054] It should be noted that, for each scale of feature map, the weights of the dynamic convolution kernel corresponding to that scale are generated by mapping the feature map of that scale to the weight space of the convolution kernel. Then, the dynamic convolution kernel corresponding to that scale is used to perform convolution processing on the feature map of that scale to obtain the dynamic convolution features at that scale. In this way, the dynamic convolution kernel performs adaptive calculation for feature maps of different scales, which helps to reduce unnecessary calculations while maintaining efficient feature extraction.

[0055] For example, in practical implementation, the size of the convolutional kernel can be adaptively adjusted. When processing larger-scale feature maps, the dynamic convolutional kernel can choose a kernel with a larger receptive field, while when processing smaller-scale feature maps, it can choose a kernel with a smaller receptive field. In this way, compared to using a fixed convolutional kernel to process feature maps of all scales, the dynamic convolutional kernel can dynamically adjust the computational cost according to the feature map, avoiding unnecessary computational overhead and reducing computational cost.

[0056] S105. Target detection is performed using the multiple dynamic convolutional features at the multiple scales and the self-attention features to obtain the detection results.

[0057] In practice, multiple dynamic convolutional features at multiple scales can be used as multi-scale image features, and self-attention features can be used as text features. Then, target detection is performed based on multi-scale image features and text features to obtain detection results. The detection results can include the bounding boxes and category information of the detected targets, that is, the detection results include the bounding boxes and category information of the targets to be detected.

[0058] Referring to the preceding description, the image to be detected refers to the image for which object detection is required, the target to be detected refers to the target to be detected, and the prompt text for the target to be detected refers to the textual description information of the target to be detected. For example, in one possible implementation, the prompt text for the target to be detected may be the category description information of the target to be detected (e.g., in one embodiment, the prompt text is: "A gun has been detected in the image"); as another possible implementation, the prompt text for the target to be detected may be the attribute description information of the target to be detected (e.g., in one embodiment, the prompt text is: "A woman dancing in red has been detected").

[0059] Furthermore, by combining the prompt text of the target to be detected with the target text to perform target detection on the image, the position information of the target to be detected in the image can be identified and located. This position information can be represented by a coordinate frame, that is, the detection result includes the coordinate frame of the target to be detected.

[0060] Furthermore, based on the above introduction, it can be understood that in security monitoring scenarios, the image to be detected can be an image captured in real time by a security monitoring camera, and the prompt text can be: "Please detect the knife in the image." In this case, through target detection, the knife in the image can be identified and located. As another example, in autonomous driving scenarios, the image to be detected can be a real-time road scene image captured by a camera on an autonomous vehicle. This real-time road scene image contains real-time visual information about the vehicle's surrounding environment, such as roads, pedestrians, vehicles, and traffic lights. Furthermore, the prompt text can be: "Please detect traffic lights, road markings, vehicles, and pedestrians." In this case, through target detection, traffic lights, road markings, vehicles, and pedestrians in the real-time road scene image can be detected. That is, in this case, the detection result includes the location information and category information of each detected target.

[0061] The method provided in this embodiment, after obtaining feature maps at multiple scales, generates the weights of a dynamic convolutional kernel corresponding to each scale by mapping the feature map at that scale to the weight space of the convolutional kernel. Then, it uses the dynamic convolutional kernel corresponding to that scale to perform convolution processing on the feature map at that scale to obtain the dynamic convolutional features at that scale. In this way, the size and weights of the dynamic convolutional kernel at that scale can be dynamically adjusted according to the scale of the feature map. The dynamic convolutional kernel performs adaptive calculations for feature maps at different scales, which helps to reduce unnecessary calculations while maintaining efficient feature extraction, significantly reducing the amount of computation, and making the object detection method adaptable to edge devices with low computing power.

[0062] Figure 3 This is a flowchart of Embodiment 2 of the target detection method provided in this application. Please refer to... Figure 3 Based on the above embodiments, the target detection method provided in this embodiment is implemented based on a pre-trained target detection model. The target detection model includes an encoder, which includes a dynamically depthwise separable convolution module. The dynamically depthwise separable convolution module includes a first pointwise convolution module and a weight generation module. The step of mapping the feature map at that scale to the weight space of the convolution kernel to generate the weights of the dynamically convolution kernel corresponding to that scale includes:

[0063] S301. The feature map at this scale is processed using the first pointwise convolution module to obtain a hybrid feature map that integrates multi-channel information.

[0064] Specifically, Figure 4 This is a schematic diagram illustrating an exemplary target detection model of this application. (Refer to...) Figure 4 The object detection model includes a feature extraction module, which can extract features from the image to be detected and the prompt text. Furthermore, in one possible implementation, the feature extraction module can include an image backbone network and a text backbone network. The image backbone network can be used to extract features from the image to be detected, obtaining feature maps at multiple scales; similarly, the text backbone network can be used to extract features from the prompt text, obtaining text features.

[0065] It should be noted that the image backbone network can be a pre-trained image backbone network such as ResNet, Swin Transformer, or ViT; this embodiment does not limit it. Similarly, the text backbone network can be BERT, CLIP, etc.; this embodiment does not limit it. For details on the specific implementation principles and processes of image and text backbone networks, please refer to the descriptions in related technologies; they will not be repeated here.

[0066] For further details, please refer to [link / reference]. Figure 4The object detection model also includes an encoder, which includes a self-attention module connected to the text backbone network. The self-attention module can be used to process text features and output self-attention features of the text features.

[0067] For further details, please refer to [link / reference]. Figure 4 The encoder also includes a dynamic depthwise separable convolution module, which includes a first pointwise convolution module and a weight generation module. The dynamic depthwise separable convolution module can be used to generate weights for a dynamic convolution kernel corresponding to a certain scale.

[0068] The following explains the generation principle of the weights of the dynamic convolution kernel corresponding to a certain scale:

[0069] Specifically, the dynamic depthwise separable convolution module processes feature maps at various scales in a similar way. The following example, using a feature map at a specific scale, illustrates the principle behind the generation of the weights for the dynamic convolution kernel at that scale. Please refer to... Figure 4 Combining the previous examples, for instance, for features Figure 1 The first pointwise convolutional module performs this feature Figure 1 The data is processed to obtain a hybrid feature map that incorporates information from multiple channels. Specifically, the first pointwise convolutional module can process the features... Figure 1 Perform depthwise separable convolution, i.e., on features Figure 1 First, perform channel convolution (using k*k convolution kernels to perform convolution operations independently on each channel, generating feature maps with the same number of input channels), then perform pointwise convolution (using 1*1 convolution kernels to linearly combine the feature maps generated by channel convolution, generating the number of output channels) to obtain a mixed feature map.

[0070] It should be noted that pointwise convolution processing refers to linearly combining the features at each location of the feature map to fuse information from multiple channels.

[0071] Combining the examples above, features Figure 1 The dimension is H*W*C, meaning the size of the feature map input to the first pointwise convolutional module is H*W*C. For example, in one embodiment, the size of the convolutional kernel in the first pointwise convolutional module is 1*1*C*C. out Convolution kernel pairs with features Figure 1 After performing pointwise convolution, the features are obtained. Figure 1 The corresponding blended feature map has dimensions H*W*C. out .

[0072] Based on the above introduction, it can be understood that the location of the blended feature map (H, W, C) out The value of ) is calculated using the following formula:

[0073]

[0074] Y represents the mixed feature map;

[0075] X is the feature map;

[0076] K is the convolution kernel.

[0077] It should be noted that, compared to standard convolution, pointwise convolution significantly reduces the number of parameters and computation. For example, a standard 3x3 convolution kernel in C... in and C out There is 3*3*C in the channel. in *C out With only one parameter, pointwise convolution has only 1*1*C. in *C out One parameter.

[0078] Optional, Figure 5 This is a schematic diagram illustrating a first pointwise convolution module as shown in an exemplary embodiment of this application. Please refer to... Figure 5 In one possible implementation, the first pointwise convolutional module includes a channel convolutional layer, a first batch normalization layer, and a first activation function layer; wherein, the processing of the feature map at this scale using the first pointwise convolutional module to obtain a hybrid feature map incorporating multi-channel information includes:

[0079] (1) The feature map at this scale is convolved point by point using the channel convolution layer to obtain the first feature map.

[0080] In practical implementation, referring to the above example, for instance, regarding features... Figure 1 1*1*C can be used in *C out convolution kernel pairs of features Figure 1 Perform pointwise convolution to obtain the first feature map. It can be understood that the dimensions of the first feature map are H*W*C. out .

[0081] It should be noted that the dimension of the first feature map can be the same as or different from the dimension of the feature map at this scale; that is, the number of output channels can be the same as or different from the number of input channels. This embodiment does not impose any limitations on this. For example, when the dimension of the first feature map is the same as the dimension of the feature map at this scale, the dimension of the first feature map is H*W*C.

[0082] (2) The first feature map is batch-standardized using the first batch standardization processing layer to obtain the second feature map.

[0083] Specifically, batch standardization refers to the process of standardizing the first feature map. In practice, the mean and variance of each batch of input data can be calculated, and then the batch mean and variance can be used to standardize the input data. Finally, scaling and translation operations are performed on the standardized data. For details on the specific implementation principles and processes of batch standardization, please refer to the descriptions in related technologies; they will not be elaborated upon here.

[0084] It should be noted that the input and output dimensions of the first batch of standardized processing layers can be the same or different; this implementation does not impose any restrictions on this.

[0085] The following explanation uses the example of the first batch of standardized processing layers having the same input and output dimensions. Combining the example above, the dimensions of the second feature map are H*W*C.

[0086] (3) The second feature map is activated using the first activation function layer to obtain the hybrid feature map.

[0087] Specifically, the first activation function layer activates the second feature map using an activation function. It is understood that this activation function can be a ReLU (Rectified Linear Unit), a Sigmoid function, a Tanh function, etc., but in this embodiment, it is not limited to any particular function.

[0088] It should be noted that the dimensions of the input and output of the first activation function layer can be the same or different; in this embodiment, no limitation is imposed.

[0089] For example, in one possible implementation, when the dimensions of the input and output of the first activation function layer are the same, combining the example above, the dimension of the blended feature map is H*W*C.

[0090] The method provided in this embodiment gives a specific structural diagram of a first pointwise convolution module. Through the first pointwise convolution module, not only can feature maps at various scales be processed to obtain a hybrid feature map that integrates multi-channel information, but the amount of computation can also be further reduced.

[0091] S302. The mixed feature map is input into the weight generation module, and the weight generation module generates the weights of the dynamic convolution kernel corresponding to the scale based on the mixed feature map.

[0092] In practice, a neural network can be used to generate the weights of the dynamic convolutional kernel corresponding to that scale. For example, a fully connected layer can be used to map the mixed feature map to the weight space of the dynamic convolutional kernel and reshape it into the shape of the convolutional kernel. Alternatively, a set of linear networks can be used to dynamically generate the weights of the dynamic convolutional kernel.

[0093] Based on the above introduction, for example, the weights of the dynamic convolutional kernel corresponding to this scale can be generated according to the following process:

[0094] W = WeightGenerator(F)

[0095] Where W is the weight of the dynamic convolution kernel corresponding to this scale, and F is the feature map at this scale.

[0096] Optional, Figure 6 This is a schematic diagram illustrating a weight generation module as shown in an exemplary embodiment of this application. Please refer to... Figure 6 In one possible implementation, the weight generation module includes a pooling layer, a channel-reduced convolutional layer, a second batch normalization layer, a second activation function layer, a channel-incremented convolutional layer, and a shape reshaping layer; wherein, the step of inputting the mixed feature map into the weight generation module, and the weight generation module generating weights for the dynamic convolutional kernel corresponding to that scale based on the mixed feature map, includes:

[0097] (1) The pooling layer is used to perform average pooling on each channel of the hybrid feature map to obtain the pooling result.

[0098] Specifically, based on the above explanation, it can be understood that a hybrid feature map is a multi-channel feature map. For example, in the above example, when the dimensions of the hybrid feature map are H*W*C, it is a multi-channel feature map with C channels, and the size of the sub-feature map corresponding to each channel is H*W.

[0099] It should be noted that in this embodiment, the average pooling process generates the downsampled sub-feature map by sliding a fixed-size window on the sub-feature map corresponding to a certain channel and calculating the average value of all values ​​within the window.

[0100] Referring to the example above, in practical implementation, for a sub-feature map corresponding to a certain channel, average pooling can be performed on H*W elements of that sub-feature map to obtain the processing result corresponding to that channel (this processing result contains 1*1 elements). Furthermore, the processing results corresponding to all channels are concatenated together to obtain the final pooling result. Based on the example above, it can be understood that in this embodiment, the dimension of the pooling result is 1*1*C.

[0101] (2) The pooling result is reduced in dimensionality using the channel dimensionality reduction convolutional layer to obtain the channel dimensionality reduction result.

[0102] Specifically, channel-reduction convolution is used to reduce the number of channels in a feature map while maintaining its spatial dimensions. In practice, pointwise convolution can be used to achieve this reduction.

[0103] It should be noted that the degree of reduction in the number of channels is set according to actual needs, and is not limited in this embodiment. The following explanation uses a channel reduction of 1 / 4 as an example. Referring to the example above, for instance, in one possible implementation, the dimension of the pooling result is 1*1*C. After dimensionality reduction processing of the pooling result, the dimension of the channel dimensionality reduction result can be 1*1*C / 4. Specifically, a convolutional kernel of size 1*1*C*C / 4 can be used to perform pointwise convolution on the pooling result to achieve the dimensionality reduction.

[0104] (3) The second batch standardization processing layer is used to perform batch standardization processing on the channel dimensionality reduction result to obtain the batch standardization processing result.

[0105] For details on the specific implementation principles of batch standardization processing, please refer to the previous introduction, which will not be repeated here.

[0106] It should be noted that the dimensions of the input and output of the second batch normalization layer can be the same or different; this embodiment does not limit this. The following explanation uses the example of the second batch normalization layer having the same dimensions as its input and output. Referring to the example above, for instance, in one possible implementation, the dimension of the channel dimensionality reduction result input to the second batch normalization layer is 1*1*C / 4, and the dimension of the batch normalization result output by the second batch normalization layer is also 1*1*C / 4.

[0107] (4) The batch standardization result is activated using the second activation function layer to obtain the activation result.

[0108] Specifically, the second activation function layer activates the batch normalization results using an activation function. It is understood that this activation function can be a ReLU (Rectified Linear Unit), a Sigmoid function, a Tanh function, etc., but in this embodiment, it is not limited to any particular function.

[0109] It should be noted that the dimensions of the input and output of the second activation function layer can be the same or different; this embodiment does not limit this. The following explanation uses the example of the second activation function layer having the same dimensions as its input and output. Referring to the example above, for instance, in one possible implementation, the dimension of the batch normalization result input to the second activation function layer is 1*1*C / 4, and the dimension of the activation result output by the second activation function layer is also 1*1*C / 4.

[0110] (5) The activation processing result is processed by the channel-upgrading convolutional layer to obtain the upgrading result.

[0111] Specifically, channel-increasing convolution is used to increase the number of channels in a feature map while keeping its spatial dimensions unchanged. In practice, pointwise convolution can be used to achieve this goal.

[0112] It should be noted that the degree to which the number of channels increases is determined according to actual needs, and this embodiment does not limit it.

[0113] Furthermore, referring to the example above, for instance, in one possible implementation, the dimension of the activation processing result output by the second activation function layer is 1*1*C / 4. After performing pointwise convolution processing on it using a convolution kernel of 1*1*C / 4*36C, the dimension of the channel upscaling result is 1*1*9C, thus increasing the number of channels in the feature map.

[0114] For example, in one possible implementation, the activation result output by the second activation function layer has a dimension of 1*1*C / 4. After performing pointwise convolution on it using a 1*1*C / 4*64C convolution kernel, the dimension of the channel upscaling result is 1*1*16C, thus increasing the number of channels in the feature map.

[0115] (6) The shape reshaping layer is used to reshape the dimensionality increase result into the shape of the convolution kernel, and the weight of the dynamic convolution kernel corresponding to the feature map at this scale is obtained.

[0116] Specifically, shape reshaping refers to transforming a multidimensional array into an array with different dimensions while keeping the total number of elements unchanged. In practice, the Reshape operation can be used to reshape the result of the dimensionality increase into the shape of the convolution kernel.

[0117] Furthermore, referring to the examples above, for instance, in the first example, the dimension of the channel upscaling result is 1*1*9C, and after reshaping its shape, the size of the reshaped convolution kernel is C*1*3*3.

[0118] For example, in the second example, the dimension of the channel upscaling result is 1*1*16C. After reshaping its shape, the size of the reshaped convolution kernel is C*1*4*4.

[0119] In this embodiment, a specific structure of a weight generation module is given. Through this weight generation module, the weights of dynamic convolution kernels can be dynamically generated to perform convolution processing on feature maps based on dynamic convolution kernels, thereby reducing the amount of computation.

[0120] The method provided in this embodiment sets a dynamically depth-separable convolution module in the encoder, and then sets a first pointwise convolution module and a weight generation module in the dynamically depth-separable convolution module. In this way, the first pointwise convolution module can be used to process the feature map of the scale to obtain a hybrid feature map that integrates multi-channel information. The hybrid feature map is then input into the weight generation module, which generates the weights of the dynamic convolution kernel corresponding to the scale based on the hybrid feature map. In this way, the convolution operation can be adaptively adjusted for feature maps of different scales, effectively avoiding the redundant calculations that may be caused by fixed convolution kernels, reducing the amount of computation and improving the computational efficiency. In addition, by using the dynamically depth-separable convolution module to decompose the standard convolution into channel-wise convolution and pointwise convolution, the number of parameters and the amount of computation can be reduced.

[0121] Optional, please continue to refer to Figure 4 ,exist Figure 4 In the example shown, the dynamically depth-separable convolutional module also includes a convolutional processing layer; wherein, the feature map at this scale is convolved using the dynamic convolutional kernel corresponding to this scale to obtain the dynamic convolutional features at this scale, including:

[0122] The convolutional processing layer uses the dynamic convolutional kernel corresponding to this scale to perform convolution processing on the hybrid feature map output by the first pointwise convolutional module to obtain the dynamic convolutional features at this scale.

[0123] Specifically, for example, for features at a certain scale Figure 1 , will feature Figure 1 The corresponding hybrid feature map is denoted as F1, and the dynamic convolution kernel corresponding to this scale determined based on this feature map is denoted as B1. Referring to the previous introduction, it can be understood that the dimensions of F1 are H*W*C, and the dimensions of B1 are C*1*3*3. In this step, B1 is used to perform convolution processing on F1 to obtain the dynamic convolution feature at this scale.

[0124] Understandably, the first pointwise convolution module can fuse multi-channel original features into a hybrid feature map, capturing richer and more abstract feature information and improving feature expressiveness. Therefore, during convolution, the hybrid feature map output by the first pointwise convolution module is processed by a dynamic convolution kernel. Since the hybrid feature map after pointwise convolution has already fused the multi-channel information of the original features, when the dynamic convolution kernel is used to perform convolution processing on it, feature extraction can be performed in a higher dimension, further enhancing the expressiveness and accuracy of the features.

[0125] Figure 7 This is a flowchart of Embodiment 3 of the target detection method provided in this application. Please continue to refer to... Figure 4Based on the above embodiments, the encoder further includes a cross-attention module, and the object detection model further includes a decoder; wherein, the object detection using multiple dynamic convolutional features at multiple scales and the self-attention features includes:

[0126] S701. The cross-attention module processes the multiple dynamic convolutional features and the self-attention features to obtain text feature representations and image feature representations.

[0127] Specifically, Figure 8 This is a schematic diagram illustrating a cross-attention module as shown in an exemplary embodiment of this application. Please refer to... Figure 8 The cross-attention module includes a first cross-attention layer, a second cross-attention layer, a first feedforward network layer, and a second feedforward network layer; wherein, the process of processing the multiple channel convolutional features and the self-attention features through the cross-attention module to obtain text feature representations and image feature representations includes the following steps:

[0128] (1) The first cross-attention layer uses the image features as the query and the self-attention features as the key and value to perform attention calculation to obtain the first fusion feature; wherein the image features are the multiple dynamic convolution features.

[0129] Specifically, in the first cross-attention layer, image features are used for the query, and self-attention features extracted from text features are used as keys and values. The first cross-attention layer calculates attention by comparing the similarity between multiple dynamic convolutional features and self-attention features to obtain attention weights. Then, the self-attention features are weighted and summed according to the obtained attention weights to output the first fusion feature.

[0130] (2) The second cross-attention layer uses the self-attention feature as the query and the first fusion feature as the key and value to perform attention calculation to obtain the second fusion feature.

[0131] It should be noted that the second cross-attention layer has the same structure as the first cross-attention layer.

[0132] Specifically, the second cross-attention layer uses self-attention features as queries and the first fusion feature as keys and values. Attention is calculated based on the similarity between the self-attention features and the first fusion feature to obtain attention weights. Then, the first fusion feature is weighted and summed according to the obtained attention weights to output the second fusion feature.

[0133] (3) The first feedforward network layer generates the text feature representation based on the first fusion feature.

[0134] Specifically, the first feedforward network layer is used to further process the first fused features, generating a higher-level text feature representation through a series of operations.

[0135] (4) The second feedforward network layer generates the image feature representation based on the second fusion feature.

[0136] It should be noted that the second feedforward network layer has the same structure as the first feedforward network layer.

[0137] Specifically, the second feedforward network layer is used to further process the second fused features, generating a higher-level image feature representation through a series of operations.

[0138] S702. The text feature representation and the image feature representation are decoded by the decoder to obtain the detection result.

[0139] In practice, the decoder calculates the similarity between the text feature representation and the image feature representation, then selects the Top k query vectors from the image feature representation based on the similarity, and finally outputs the final detection result based on the query vectors and the text feature representation.

[0140] The method provided in this embodiment gives a specific method for target detection using multiple dynamic convolution features and self-attention features at multiple scales. Through this method, the target to be detected in the image can be accurately identified.

[0141] Figure 9 This is a flowchart of Embodiment 4 of the target detection method provided in this application. Figure 10 A schematic diagram of a target detection model shown in another exemplary embodiment of this application is provided below. Figure 10 ,exist Figure 10 In the example shown, the encoder further includes a second pointwise convolutional module and a cross-attention module, and the object detection model further includes a decoder; wherein, the object detection using multiple dynamic convolutional features at multiple scales and the self-attention features includes:

[0142] S901. For the dynamic convolution features at each scale, the second pointwise convolution module is used to process the dynamic convolution features to obtain the channel convolution features at each scale.

[0143] It should be noted that the structure and function of the second pointwise convolution module are the same as those of the first pointwise convolution module. In this embodiment, they will not be described in detail.

[0144] As explained above, the second pointwise convolution module can fuse dynamic convolutional features at each scale into channel convolutional features at that scale, capturing richer and more abstract feature information and improving feature expressiveness. Through the second pointwise convolution module, higher-level channel convolutional features can be generated.

[0145] S902. The cross-attention module processes the multiple channel convolutional features and the self-attention features to obtain text feature representations and image feature representations.

[0146] For details, please continue to refer to... Figure 8 The cross-attention module in this step has the same structure and processing procedure as the cross-attention module in step S701 above, so it will not be described again here.

[0147] S903. The text feature representation and the image feature representation are decoded by the decoder to obtain the detection result.

[0148] Specifically, the implementation process and principle of this step can be found in the description in the previous embodiments, and will not be repeated here.

[0149] This embodiment provides a specific method for target detection using multiple dynamic convolutional features and self-attention features at multiple scales. This method can accurately identify the target in the image to be detected. Furthermore, in this embodiment, for the dynamic convolutional features at each scale, a second pointwise convolutional module processes these features to obtain channel convolutional features at each scale. These channel convolutional features are then fed into a cross-attention module for further processing. Because the second pointwise convolutional module can capture richer and more abstract feature information, it improves the expressive power of the features, making subsequent processing more accurate and further enhancing the accuracy of target detection.

[0150] Corresponding to the aforementioned embodiment of a target detection method, this application also provides an embodiment of a target detection device.

[0151] An embodiment of the target detection device disclosed in this application can be applied to a target detection device. The device embodiment can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of the target detection device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 11 The diagram shown is a hardware structure diagram of the target detection equipment containing the target detection device provided in this application, except for... Figure 11In addition to the processor, memory, network interface, and non-volatile memory shown, the target detection device in the embodiment may also include other hardware depending on the actual function of the target detection device, which will not be described in detail here.

[0152] Figure 12 This is a schematic diagram of the structure of Embodiment 1 of the target detection device provided in this application. Please refer to... Figure 12 The apparatus provided in this embodiment includes an extraction module 121, a generation module 122, and a detection module 123; wherein,

[0153] The extraction module 121 is used to extract features from the image to be detected and the prompt text of the target to be detected, respectively, to obtain feature maps and text features at multiple scales;

[0154] The generation module 122 is used to generate self-attention features of the text features through a self-attention mechanism;

[0155] The generation module 122 is further configured to map the feature map at each scale to the weight space of the convolution kernel, and generate the weights of the dynamic convolution kernel corresponding to that scale.

[0156] The generation module 122 is further configured to perform convolution processing on the feature map of the scale using the dynamic convolution kernel corresponding to the scale, so as to obtain the dynamic convolution feature at the scale.

[0157] The detection module 123 is used to perform target detection using multiple dynamic convolutional features at multiple scales and the self-attention features, and obtain detection results.

[0158] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0159] Please continue to refer to Figure 11 This application also provides a target detection device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the methods provided in the first aspect of this application.

[0160] It should be noted that, in one possible implementation, the target detection device can be an edge low-computing-power device, which refers to a device deployed at the network edge with low computing power and energy consumption.

[0161] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods provided in this application.

[0162] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0163] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0164] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A target detection method, characterized in that, The method includes: Feature extraction is performed on the image to be detected and the prompt text of the target object to be detected separately to obtain feature maps and text features at multiple scales; The text features are generated using a self-attention mechanism; For each scale of feature map, the feature map at that scale is mapped to the weight space of the convolutional kernel to generate the weights of the dynamic convolutional kernel corresponding to that scale; the object detection method is implemented based on a pre-trained object detection model, which includes an encoder, a dynamically depth-separable convolutional module, and a first pointwise convolutional module and a weight generation module; wherein, mapping the feature map at that scale to the weight space of the convolutional kernel to generate the weights of the dynamic convolutional kernel corresponding to that scale includes: The feature map at this scale is processed using the first pointwise convolution module to obtain a hybrid feature map that incorporates multi-channel information; The mixed feature map is input into the weight generation module, which then generates the weights of the dynamic convolution kernel corresponding to that scale based on the mixed feature map. The feature map at this scale is convolved using a dynamic convolution kernel corresponding to this scale to obtain dynamic convolutional features at this scale; the dynamic depthwise separable convolution module further includes a convolution processing layer; wherein, the dynamic convolutional features at this scale are obtained by convolving the feature map at this scale using a dynamic convolution kernel corresponding to this scale, including: The convolutional processing layer uses the dynamic convolutional kernel corresponding to this scale to perform convolution processing on the hybrid feature map output by the first pointwise convolutional module to obtain the dynamic convolutional features at this scale. Target detection is performed using multiple dynamic convolutional features at multiple scales and the self-attention features to obtain detection results; the encoder further includes a second pointwise convolutional module and a cross-attention module, and the target detection model further includes a decoder; wherein, the target detection using multiple dynamic convolutional features at multiple scales and the self-attention features includes: For the dynamic convolutional features at each scale, the second pointwise convolution module is used to process the dynamic convolutional features to obtain the channel convolutional features at each scale. The cross-attention module processes the convolutional features of the multiple channels and the self-attention features to obtain text feature representations and image feature representations. The detection result is obtained by decoding the text feature representation and the image feature representation using the decoder.

2. The method according to claim 1, characterized in that, The first pointwise convolutional module includes a channel convolutional layer, a first batch normalization layer, and a first activation function layer; wherein, the process of using the first pointwise convolutional module to process the feature map at this scale to obtain a hybrid feature map that integrates multi-channel information includes: The first feature map is obtained by performing point-by-point convolution on the feature map at this scale using the channel convolutional layer. The first feature map is batch-normalized using the first batch of normalization layers to obtain the second feature map; The second feature map is activated using the first activation function layer to obtain the hybrid feature map.

3. The method according to claim 1 or 2, characterized in that, The weight generation module includes a pooling layer, a channel-reduced convolutional layer, a second batch normalization layer, a second activation function layer, a channel-incremented convolutional layer, and a shape reshaping layer; wherein, the step of inputting the mixed feature map into the weight generation module, and the weight generation module generating weights for the dynamic convolutional kernel corresponding to that scale based on the mixed feature map, includes: The pooling layer is used to perform average pooling on each channel of the hybrid feature map to obtain the pooling result. The pooling result is reduced in dimensionality using the channel-reducing convolutional layer to obtain the channel-reduced result. The second batch standardization layer is used to perform batch standardization on the channel dimensionality reduction result to obtain the batch standardization result. The batch normalization result is activated using the second activation function layer to obtain the activation result. The activation processing result is then subjected to dimensionality upscaling using the aforementioned channel-based dimensionality upscaling convolutional layer to obtain the dimensionality upscaling result. The shape reshaping layer is used to reshape the dimensionality increase result into the shape of the convolution kernel, thereby obtaining the weights of the dynamic convolution kernel corresponding to that scale.

4. The method according to claim 1, characterized in that, The encoder further includes a cross-attention module, and the object detection model further includes a decoder; wherein, the object detection using multiple dynamic convolutional features at multiple scales and the self-attention features includes: The cross-attention module processes the multiple dynamic convolutional features and the self-attention features to obtain text feature representations and image feature representations. The detection result is obtained by decoding the text feature representation and the image feature representation using the decoder.

5. The method according to claim 1 or 4, characterized in that, The cross-attention module includes a first cross-attention layer, a second cross-attention layer, a first feedforward network layer, and a second feedforward network layer; wherein, the process of processing the multiple channel convolutional features and the self-attention features through the cross-attention module to obtain text feature representations and image feature representations includes: The first cross-attention layer uses image features as queries and the self-attention features as keys and values ​​to perform attention calculations to obtain the first fusion feature; wherein, the image features are the multiple dynamic convolution features or the multiple channel convolution features; The second cross-attention layer uses the self-attention feature as the query and the first fusion feature as the key and value to perform attention calculation, thereby obtaining the second fusion feature; The first feedforward network layer generates the text feature representation based on the first fusion feature; The second feedforward network layer generates the image feature representation based on the second fusion feature.

6. A target detection device, characterized in that, The device includes an extraction module, a generation module, and a detection module; wherein, The extraction module is used to extract features from the image to be detected and the prompt text of the target to be detected, respectively, to obtain feature maps and text features at multiple scales; The generation module is used to generate self-attention features of the text features through a self-attention mechanism; The generation module is further configured to map the feature map at each scale to the weight space of the convolutional kernel, generating the weights of the dynamic convolutional kernel corresponding to that scale; the object detection device is implemented based on a pre-trained object detection model, the object detection model including an encoder, the encoder including a dynamic depthwise separable convolutional module, the dynamic depthwise separable convolutional module including a first pointwise convolutional module and a weight generation module; wherein, mapping the feature map at that scale to the weight space of the convolutional kernel to generate the weights of the dynamic convolutional kernel corresponding to that scale includes: The feature map at this scale is processed using the first pointwise convolution module to obtain a hybrid feature map that incorporates multi-channel information; The mixed feature map is input into the weight generation module, which then generates the weights of the dynamic convolution kernel corresponding to that scale based on the mixed feature map. The generation module is further configured to perform convolution processing on the feature map of the scale using the dynamic convolution kernel corresponding to the scale, to obtain dynamic convolution features at the scale; the dynamic depthwise separable convolution module further includes a convolution processing layer; wherein, performing convolution processing on the feature map of the scale using the dynamic convolution kernel corresponding to the scale to obtain dynamic convolution features at the scale includes: The convolutional processing layer uses the dynamic convolutional kernel corresponding to this scale to perform convolution processing on the hybrid feature map output by the first pointwise convolutional module to obtain the dynamic convolutional features at this scale. The detection module is used to perform target detection using multiple dynamic convolutional features at multiple scales and the self-attention features to obtain a detection result; the encoder further includes a second pointwise convolutional module and a cross-attention module, and the target detection model further includes a decoder; wherein, the target detection using multiple dynamic convolutional features at multiple scales and the self-attention features includes: For the dynamic convolutional features at each scale, the second pointwise convolution module is used to process the dynamic convolutional features to obtain the channel convolutional features at each scale. The cross-attention module processes the convolutional features of the multiple channels and the self-attention features to obtain text feature representations and image feature representations. The detection result is obtained by decoding the text feature representation and the image feature representation using the decoder.

7. A target detection device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the steps of the method according to any one of claims 1-5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1-5.

Images