A zero-shot image object detection method based on deep learning

Abstract

This paper designs a deep learning-based zero-shot image object detection method, which belongs to the field of large-model-based image object detection. The visual language model designed by this paper includes RO-Yolo-Backbone, HWD-RepVL-PAN, MCJ-Text Contrastive Head, and regression bounding box object detection modules. By optimizing the sharing of convolution kernel parameters and the receptive field range, the method enhances feature extraction capabilities, utilizes wavelet transforms to reduce downsampling texture loss, and introduces a multi-distance formula to weaken intra-class differences and enhance inter-class differences. The aim is to effectively improve zero-shot image object detection capabilities in the era of big data.

Description

Technical Field

[0001] The present invention belongs to the field of large-model-based target detection, and specifically relates to a zero-sample pedestrian detection method based on deep learning. Background Art

[0002] The development of deep neural networks has led to significant breakthroughs in many areas of object detection. However, regardless of the size of the deep neural network model, they still face a hidden drawback in real-world applications. Mainstream detection methods are severely restricted in the categories they can detect. When the model must detect hundreds of thousands of objects, or when there are few datasets for the objects, such as in rare medical cancers and autonomous driving scenarios, mainstream models face significant challenges, both in terms of dataset annotation and detection performance.

[0003] Regarding downsampling, Li et al. (Li Yandong, Hao Zongbo, Lei Hang. A review of convolutional neural network research [J]. Computer Applications, 2016, 36(09): 2508-2515+2565.) mentioned that traditional downsampling methods such as maximum pooling and average pooling have large errors. Maximum pooling will cause the loss of background information by discarding the parts with smaller pixel values, and average pooling will cause the loss of texture information by averaging the pixels. Therefore, this patent uses HWD wavelet transform instead of the maximum pooling layer, and adopts a mode of splicing high and low frequency components to reduce the loss of background and texture information.

[0004] Regarding convolution, Zhang et al. (Zhang Xin, Liu Chen, Yang Degang, et al. Rfaconv: Innovating spatital attention and standard convolutional operation [J]. arxiv preprint arxiv: 2304.03198, 2023.) mentioned that although existing spatial attention such as convolution block attention module and coordinated attention have been widely used to improve the performance of convolutional neural networks and solve the problem of convolution kernel parameter sharing, the information contained in the attention map generated by spatial attention is insufficient for large-size convolution kernels. Therefore, this patent uses RFA convolution to replace part of the ordinary convolution. RFA convolution not only focuses on the spatial features of the receptive field, but also provides effective attention weights for large-size convolution kernels, significantly improving network performance with almost negligible computational cost and parameter increment.

[0005] Regarding zero-shot detection, Pan et al. (Pan Chongyu, Huang Jian, Hao Jianguo, et al. A review of weakly supervised learning methods integrating zero-shot learning and small-shot learning [J]. Systems Engineering and Electronics, 2020, 42(10): 2246-2256.) mentioned that zero-shot learning attempts to identify new category concepts that have never been seen during the training process, which requires additional semantic feature auxiliary information, such as the semantic feature description vectors of the training set and the test set category to be classified, so as to achieve knowledge transfer from the training set to the test set category. Due to its inherent domain adaptation and hub degree problems, zero-shot learning also faces problems such as low recognition accuracy. Therefore, this patent adopts the open vocabulary detection architecture of Yolo-world to predict and embed the fused multi-level cross-modal text encoding information and image encoding information to achieve zero-shot detection efficiently.

[0006] In summary, while deep learning-based zero-shot image object detection methods have made some progress, they still need to be improved to better meet the needs of a large number of detection objects and very few dataset training samples. To this end, this patent proposes an efficient, accurate, and highly generalizable deep learning-based zero-shot image object detection method to meet the diverse needs of image object detection. Summary of the Invention

[0007] The present invention aims to overcome the shortcomings of the above existing technologies and designs a zero-shot image target detection method based on deep learning to improve the detection accuracy under different numbers of detection objects and training data sets.

[0008] Accuracy and speed.

[0009] In order to achieve the above invention content, the present invention provides a zero-sample image target detection method based on deep learning, such as Figure 1 As shown, the specific steps include:

[0010] (1) Generate region-text pairs using an automatic annotation algorithm for the dataset;

[0011] The method of generating region-text pairs by automatically annotating a data set comprises extracting noun objects from the data set, pseudo-annotating region-text pairs, and automatically generating region-text pairs by CLIP filtering.

[0012] Furthermore, the extraction of noun objects from the dataset refers to replacing the N-Gram model used in noun object extraction with a tagger pre-trained on a large corpus based on a hidden Markov model, and outputting the result by reducing the dimensionality, retaining only the word blocks of the noun part of speech;

[0013] Furthermore, the region-text pair pseudo-annotation refers to generating region-text pair pseudo-annotations for given object nouns using a pre-trained open vocabulary detector;

[0014] Furthermore, the CLIP filtering refers to re-storing or marking boxes and filtering based on region-text pair similarity using pre-trained CLIP.

[0015] (2) Extract text embeddings with zero-shot precision using the CLIP pre-trained Transformer text encoder;

[0016] The zero-shot extraction of text embeddings using the CLIP pre-trained Transformer text encoder refers to extracting the corresponding text embedding W = TextEncoder(T)∈R for a given text using the CLIP pre-trained Transformer text encoder. C∪D , where C is the number of nouns, D is the embedding dimension, and T is the text. Compared with the text-only language encoder, the CLIP text encoder provides better visual semantic capabilities to connect visual objects with text.

[0017] (3) Extract image codes through RO-Yolo-Backbone;

[0018] The image code extraction through the improved RO-Yolo-Backbone refers to the use of RFA convolution to replace the ordinary convolution of Yolo-Backbone, improving network performance with extremely small computational cost and parameter increment, optimizing convolution kernel parameter sharing, and introducing OD convolution in the Concat front end of CSPLayer_2Conv. At the same time, the stride connection between the convolution and Bottleneck in CSPLayer_2Conv and the stride connection between different Bottlenecks are increased, the gradient flow is increased, the gradient variance is reduced, the convergence speed and convergence effect are improved, the smoothness and generalization of the network are improved, and the subsequent HWD-RepVL-PAN module is more conducive to the bidirectional fusion of text coding information and image coding information.

[0019] (4) Bidirectional fusion of text coding and image coding through HWD-RepVL-PAN;

[0020] The bidirectional fusion of text coding and image coding through HWD-RepVL-PAN refers to using HWD to replace the downsampling layer, improving the internal convolution of T-CSPLayer and adding a downsampling layer to improve RepVL-PAN, thereby enhancing the network's bidirectional fusion capability for text coding and image coding;

[0021] Furthermore, the use of HWD to replace the downsampling layer refers to using the HWD downsampling module to replace the three maximum pooling layers in the image pooling attention layer. The HWD downsampling module consists of two parts. The first part uses Haar transform to reduce the spatial resolution, and the second part filters redundant information through convolution, batch normalization and ReLU operations.

[0022] Furthermore, the improved T-CSPLayer internal convolution refers to replacing the first convolution block internal convolution and the second convolution block internal convolution in the DarknetBottleneck in T-CSPLayer with RFA convolution and OD convolution respectively, to enhance the ability of bidirectional fusion of text encoding information and image encoding information;

[0023] Furthermore, the addition of the downsampling layer refers to adding an image pooling attention layer at the connection between Yolo-Backbone and HWD-RepVL-PAN to enrich the perception level of text embedding for image embedding information;

[0024] (5) Image object detection using MCJ-Text Contrastive Head and regression bounding box;

[0025] The image target detection by MCJ-Text Contrastive Head and regression bounding box refers to calculating the image-text similarity of the regional contrast head according to formulas (1) to (4). The similarity metric is mainly based on the Minkowski distance, supplemented by the cosine similarity coefficient and the Jaccard similarity coefficient.

[0026] S k，j =α1·F MD +α2·F CS +α3·F JS +β (1)

[0027]

[0028]

[0029]

[0030] Among them, the candidate value of p is an integer greater than or equal to 1. The value of p varies with the network density and distribution shape. At the same time, considering the defects of the difference in the calculation direction of the Minkowski distance, the cosine similarity coefficient and the Jaccard similarity coefficient are used as an auxiliary, aiming to weaken the differences in the encoding of objects of the same category in different images and enhance the ability to distinguish common features between individuals, thereby improving the accuracy; the initial coefficients of β, α1, α2, and α3 are 1, 1, 0.5, and 0.5, and the latter two values ​​are updated and optimized by back propagation. k represents the number of objects, and w h is the hth text embedding, e k is the k-th image embedding, F MD is the Minkowski distance formula after the encoded information is substituted, F CS is the cosine similarity coefficient formula after the encoding information is substituted, F JS It is the Jaccard similarity coefficient formula after the encoding information is substituted;

[0031] Finally, multiple targets with position confidence greater than a threshold are framed by the regression boundary, and non-maximum suppression is used to retain the highest-scoring bounding box to achieve zero-shot image target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a flow chart of an embodiment of the present invention;

[0033] Figure 2 This is a structural diagram of an embodiment of the present invention;

[0034] Figure 3 This is a diagram of RO-Yolo-Backbone according to an embodiment of the present invention;

[0035] Figure 4 This is a diagram of HWD-RepVL-PAN according to an embodiment of the present invention;

[0036] Figure 5 This is an example diagram of the improved CSPLayer_2Conv, Darknet Bottleneck, RFA convolution block, and OD convolution block in an embodiment of the present invention;

[0037] Figure 6 This is an example diagram of the improved T-CSPLayer, image pooling attention layer, and feature representation learning block in an embodiment of the present invention; DETAILED DESCRIPTION

[0038] The technical approach of the present invention is further described in detail below through specific embodiments and drawings.

[0039] like Figures 1 to 6 As shown, an embodiment of the method for zero-shot image target detection based on deep learning of the present invention includes the following steps:

[0040] 1. Use automatic annotation algorithm to generate region-text pairs for the dataset

[0041] The dataset is optimized by extracting noun objects, pseudo-annotating region-text pairs, and automatically generating region-text pairs using pre-trained CLIP filtering, while maintaining almost unchanged speed.

[0042] Specifically, considering that this method lacks long-term dependence when extracting nouns from short English texts, the N-Gram model cannot capture long-distance word relationships, and it is difficult to guarantee accuracy when the number of detected objects, i.e. nouns, increases. The tagger based on the hidden Markov model pre-trained on a large corpus is more accurate and efficient for this method's noun extraction from short English texts; therefore, the N-Gram model used in text noun extraction is replaced with a tagger based on the hidden Markov model pre-trained on a large corpus, and the results are output with reduced dimensionality, retaining only word blocks of noun parts of speech. Specifically, for a short English text, modeling is performed according to the hidden Markov model, and the specific steps are as follows:

[0043] (1) The state of the hidden Markov chain at any time t is only related to the previous state, where i is a state sequence of length T and o is the corresponding observation sequence;

[0044] P(i t |i t-1 ,o t-1 ,…,i1,o1)=P(i t |i t-1 ) (5)

[0045] (2) Implicit state: the part of speech of each English word or phrase can be roughly estimated;

[0046] P(o t |i t ,o t ,…,i1,o1)=P(o t |i t ) (6)

[0047] (3) Finally, the joint probability distribution established by the above two assumptions is:

[0048]

[0049] After completing the hidden Markov modeling of short English texts, the pre-trained open vocabulary detector GLIP-L (Grounded Language-Image Pre-training) is used to generate pseudo boxes for a given noun phrase in each image, thereby providing rough region-text pairs. The pre-trained CLIP is used to evaluate the relevance of image-text pairs and region-text pairs, and to filter out pseudo annotations and images with low relevance. Non-maximum suppression is then used to further filter out redundant bounding boxes.

[0050] 2. Extract text embeddings in zero-shot using CLIP pre-trained Transformer text encoder

[0051] For a given text, we use the Transformer text encoder pre-trained by CLIP to extract the corresponding text embedding W = TextEncoder(T)∈R in zero-shot. C∪D , where C is the number of nouns, D is the embedding dimension, and T is the text. The Transformer encoder fully utilizes the characteristics of its Self-Attention mechanism to establish long-distance modeling dependencies and global information of short English texts. The Self-Attention calculation formula is as follows:

[0052]

[0053] Where Q = x × W Q , K=x×W K , V=x×W V , d k is the feature dimension, B is the bias matrix, x is the input vector, W Q , W K , W V is the transformation matrix, which is determined by the initial parameters of the model.

[0054] 3. Extract image code through RO-Yolo-Backbone

[0055] Although Yolo-Backbone based on CNN has high speed and accuracy in traditional image object detection, its encoding information extraction and interaction capabilities for zero-shot image object detection tasks such as text encoding and image encoding need to be optimized. Therefore, this patent uses RFA convolution to replace the ordinary convolution of Yolo-Backbone, such as Figure 3 As shown in Figure 2, OD convolution is introduced in the Concat front end of CSPLayer_2Conv, and the stride connection between the convolution block and Bottleneck in CSPLayer_2Conv and the stride connection between different Bottlenecks are increased.

[0056] Specifically, for RFA convolution, the difference between RFA convolution and ordinary convolution is that RFA convolution emphasizes the importance of different features in the receptive field slider and gives priority to the receptive field space features, thereby solving the problem of parameter sharing in Yolo-Backbone ordinary convolution and improving the overall performance of the network. The RFA convolution module formula is expressed as follows:

[0057] F=Softmax(g 1×1 (AvgPool(X)))×ReLU(Norm(g k×k (X))) (9)

[0058] Among them, g i×i Represents a grouped convolution of size i×i, k represents the size of the convolution kernel, Norm represents normalization, X represents the input feature map, and F represents the multiplication result of the attention map and the transformed receptive field space feature.

[0059] For OD convolution, in each convolution layer, OD convolution uses a novel multi-dimensional attention mechanism to learn four types of attention of the convolution kernel along the four dimensions of the convolution kernel space in parallel and complement each other, and gradually apply them to the corresponding convolution kernel, significantly enhancing the feature extraction capability of the basic convolution operation of Yolo-Backbone. The OD convolution module formula is expressed as follows:

[0060]

[0061] Among them, X and Y represent the input features and output features respectively, W i represents the i-th convolution kernel composed of 4 filters, ⊙ represents the multiplication operation at different latitudes along the convolution kernel, α wi is the attention scalar, α fi , α ci and α si Represents the attention along the spatial dimension of the convolution kernel, the input channel dimension, and the output channel dimension respectively;

[0062] At the same time, if Figure 5 As shown in the figure, the stride connection between the convolution block and the Bottleneck in CSPLayer_2Conv is increased, and the stride connection between the i-th Bottleneck and the i+2-th Bottleneck is increased, where i∈1,2,…,n-2. This increases the gradient flow, reduces the gradient variance, improves the convergence speed and convergence effect, improves the smoothness and generalization of the network, and is more conducive to the subsequent HWD-RepVL-PAN module to bidirectionally fuse text encoding information with image encoding information.

[0063] 4. Bidirectional fusion of text encoding and image encoding through HWD-RepVL-PAN

[0064] like Figure 4 As shown in the figure, HWD is used to replace the downsampling layer, the internal convolution of T-CSPLayer is improved, and the downsampling layer is added to improve RepVL-PAN, thereby enhancing the network's bidirectional fusion capability for text encoding and image encoding.

[0065] Specifically, considering that the downsampling layer in RepVL-PAN is a maximum pooling layer, the maximum pooling layer will lose edge and texture information during the pooling process, which is not conducive to the bidirectional fusion of text coding and image coding. Therefore, this patent uses HWD to replace the downsampling layer. HWD consists of two modules, namely the lossless feature coding block and the feature representation learning block: 1) The lossless feature coding block uses the Haar wavelet transform layer to effectively reduce the spatial resolution of the feature map and retain all information. It contains a low-pass decomposition filter and a high-pass decomposition filter to generate approximate components, horizontal detail components, vertical detail components, and diagonal detail components, increasing the number of channels of the feature map by four times; 2) The feature representation learning block is used to filter redundant information. It contains standard 1x1 convolution, batch normalization, and ReLu operations.

[0066] Specifically, if Figure 5 As shown, considering the internal convolution of T-CSPLayer, that is, the DarknetBottleneck convolution part, the ordinary convolution inside it is difficult to give full play to the dual interaction of text encoding and image encoding, and the convolution kernel parameter sharing and receptive field are both defective. This patent changes the first ordinary convolution of Darknet Bottleneck to an RFA convolution block, and the second ordinary convolution to an OD convolution block. Specifically, the RFA convolution block and the OD convolution block have been expressed in detail in the specific embodiment step 3 of this patent (extracting image code through RO-Yolo-Backbone).

[0067] Specifically, if Figure 4 and Figure 6 As shown in the figure, considering that in the original RepVL-PAN, the three-size feature maps obtained by Yolo-Backbone are processed by a layer of T-CSPLayer before being processed by the image pooling attention layer to obtain the image code embedded in the text code, that is, there is a lack of interactive embedding of low-level image code into text code. This patent adds an image pooling attention layer at the connection between Yolo-Backbone and RepVL-PAN, and adds the corresponding interactive embedding of the image code processed by the image pooling attention layer into the text code, aiming to enrich the dual interactive capabilities of text and image codes.

[0068] 5. Image object detection through MCJ-Text Contrastive Head and regression bounding box

[0069] Considering that the image-text similarity formula of the Text Contrastive Head only uses the L2 normal form formula for distance measurement, it is difficult to reflect the differences in the encoding of objects of the same category in different images, and the ability to distinguish common features between individuals is weak. Therefore, this patent uses the MCJ-Text Contrastive Head and the regression bounding box to detect image targets. It means that the image-text similarity of the regional contrast head is calculated according to formulas (11) to (14). This similarity measurement is mainly based on the Minkowski distance, supplemented by the cosine similarity coefficient and the Jaccard similarity coefficient. Finally, the regression bounding box is used to define multiple targets whose position confidence is greater than the threshold, and non-maximum suppression is used to retain the highest-scoring bounding box to achieve zero-shot image target detection.

[0070] Specifically, the target text similarity formula is as follows:

[0071] S k,j =α1·F MD +α2·F CS +α3·F JS +β (11)

[0072]

[0073]

[0074]

[0075] Among them, the candidate value of p is an integer greater than or equal to 1. The value of p changes with the density and distribution shape of the network, and takes into account the defects of the difference in the calculation direction of the Minkowski distance. This patent uses the cosine similarity coefficient and the Jaccard similarity coefficient for assistance, aiming to weaken the differences in the encoding of objects of the same category in different images and enhance the ability to distinguish common features between individuals, thereby improving the accuracy. The initial coefficients of β, α1, α2, and α3 are 1, 1, 0.5, and 0.5. The latter two values ​​are updated and optimized by backpropagation. k represents the number of objects, and w h is the hth text embedding, e k is the k-th image embedding, F MD is the Minkowski distance formula after the encoded information is substituted, F CS is the cosine similarity coefficient formula after the encoding information is substituted, F JS It is the Jaccard similarity coefficient formula after the encoding information is substituted;

[0076] Finally, if Figure 2As shown in the figure, the target embedding is obtained by the text comparison head, the regional text matching is obtained by the target embedding and the image-aware text embedding, and the target embedding and the regressed bounding box are used to retain the highest scoring bounding box using non-maximum suppression to achieve zero-shot image target detection.

[0077] Compared with the prior art, the present invention has the following beneficial effects:

[0078] (1) This patent designs RO-Yolo-Backbone based on Yolo-World. By introducing RFA convolution and OD convolution, it optimizes the convolution kernel parameter sharing and receptive field range, enhances the image coding extraction capability, and increases the stride connection between the convolution block and Bottleneck in CSPLayer_2Conv and the stride connection between different Bottlenecks, increases the gradient flow, and facilitates the subsequent HWD-RepVL-PAN module to bidirectionally fuse text coding information with image coding information.

[0079] (2) This patent designs the HWD-RepVL-PAN module based on Yolo-World. By introducing HWD, RFA convolution and OD convolution, the downsampling module is optimized, and the information loss of image coding is reduced through Haar wavelet transform. At the same time, the image pooling attention layer is added to enhance the interactive embedding ability of low-level image coding into text coding.

[0080] (3) This patent designs the MCJ-Text Contrastive Head module based on Yolo-World. By introducing the Minkowski distance, cosine similarity coefficient and Jaccard similarity coefficient formula, the improved image-text similarity formula of the text contrast head aims to weaken the differences in different image encodings of objects of the same category and enhance the ability to distinguish common features between individuals, thereby improving accuracy.

[0081] (4) This patent provides an implementable zero-shot image target detection method based on deep learning, and designs a feature extraction module, an image-text encoding interaction module and a detection module. It has a significant improvement in rare medical cancer and autonomous driving scenarios, especially in the era of big data where there are many tasks that lack data sets and are difficult to label data sets. It provides a feasible method for zero-shot image target detection based on deep learning.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific implementation methods of the present invention can still be modified or replaced without departing from the spirit and scope of the present invention. Any modification or replacement should be included within the scope of protection of the claims of the present invention.

Claims

1. A zero-shot image target detection method based on deep learning, characterized by The noun phrases extracted by the hidden Markov model are embedded in text by the Transformer encoder pre-trained by CLIP, and feature extraction is performed using RO-Yolo-Backbone. Then, an interactive mapping between visual and language information is established through HWD-RepVL-PAN and region-text contrast loss to achieve zero-shot image target detection, including the following steps: (1) an automatic annotation algorithm is used to generate region-text pairs for the dataset: noun objects, region-text pairs pseudo-annotations are extracted from the dataset, and region-text pairs are automatically generated using CLIP filtering; (2) a Transformer text encoder pre-trained by CLIP is used to extract text embeddings in zero-shot mode: for a given text, a Transformer text encoder pre-trained by CLIP is used to extract the corresponding text embedding; (3) image encoding is extracted through RO-Yolo-Backbone, and the extraction of image encoding through RO-Yolo-Backbone refers to replacing the ordinary convolution of Yolo-Backbone with RFA convolution, improving network performance with minimal computational cost and parameter increment, optimizing convolution kernel parameter sharing, and The OD convolution is introduced into the Concat front end of PLayer_2Conv, and the stride connection between the convolution block and Bottleneck in CSPLayer_2Conv and the stride connection between different Bottlenecks are increased to increase the gradient flow, which is conducive to the subsequent HWD-RepVL-PAN module to bidirectionally fuse the text encoding information with the image encoding information; (4) bidirectional fusion of text encoding and image encoding through HWD-RepVL-PAN, the bidirectional fusion of text encoding and image encoding through HWD-RepVL-PAN refers to using HWD to replace the downsampling layer, improving the internal convolution of T-CSPLayer and adding a downsampling layer to improve RepVL-PAN; the use of HWD to replace the downsampling layer refers to using the HWD downsampling module to replace the three maximum pooling layers in the image pooling attention layer. The HWD downsampling module consists of two parts. The first part uses Haar transform to reduce the spatial resolution, and the second part filters redundant information through convolution, batch normalization and ReLU operations; the improved internal convolution of T-CSPLayer refers to the Darknet in T-CSPLayer. The convolution inside the first convolutional block and the convolution inside the second convolutional block in Bottleneck are replaced by RFA convolution and OD convolution respectively; the added downsampling layer refers to adding an image pooling attention layer at the connection between Yolo-Backbone and HWD-RepVL-PAN; (5) Image target detection is performed through MCJ-Text Contrastive Head and regression bounding box.

2. The method for zero-shot image target detection based on deep learning according to claim 1, characterized in that: The image target detection by MCJ-Text Contrastive Head and regression bounding box refers to calculating the image-text similarity of the regional contrast head according to formulas (1) to (4). The similarity metric is mainly based on Minkowski distance, supplemented by cosine similarity coefficient and Jaccard similarity coefficient. ， in, p The candidate value of is an integer greater than or equal to 1, p The value changes with the density and distribution shape of the network. Considering the defects of the difference in the calculation direction of the Minkowski distance, the cosine similarity coefficient and the Jaccard similarity coefficient are used as an aid. The initial coefficients are 1, 1, 0.5, and 0.5 respectively. The latter two values ​​are updated and optimized by back propagation. k Indicates the number of objects, It is h text embeddings, It is k Image embedding, is the Minkowski distance after the encoded information is substituted, is the cosine similarity coefficient after the encoding information is substituted, It is the Jaccard similarity coefficient after the encoding information is substituted; finally, multiple targets whose position confidence is greater than the threshold are framed by the regression boundary, and the highest scoring boundary box is retained by non-maximum suppression to achieve zero-shot image target detection.

Images