Zero sample image target detection method based on deep learning

Abstract

The invention designs a zero sample image target detection method based on deep learning, and belongs to the field of image target detection based on a large model. The visual language model designed by the invention comprises a RO-Yolo-Backbone, an HWD-RepVL-PAN, an MCJ-Text Control Head and a regression bounding box target detection module, the feature extraction capability is enhanced by sharing convolution kernel parameters and optimizing a receptive field range, down-sampling texture loss is reduced by utilizing wavelet transform, intra-class differences are weakened and inter-class differences are enhanced by introducing a multi-distance formula, and the robustness of the target detection model is improved. The objective of the invention is to effectively improve the zero sample image target detection capability in the big data era.

Description

Technical Field

[0001] The present invention belongs to the field of large model-based target detection, and specifically, relates to a zero-sample pedestrian detection method based on deep learning. Background Art

[0002] With the development of deep neural networks, the field of object detection has made significant breakthroughs in many aspects. However, regardless of the size of the deep neural network model, they have hidden shortcomings in actual application scenarios, that is, mainstream detection methods have strict restrictions on the categories of detection. When the model's detection objects are at the level of hundreds of thousands or there are almost no corresponding data sets for the objects to be detected, such as rare medical cancers and autonomous driving scenarios, it is a huge challenge for mainstream models both in terms of data set annotation and detection results.

[0003] Regarding downsampling, Li et al. (Li Yandong, Hao Zongbo, Lei Hang. A review of convolutional neural network research [J]. Computer Applications, 2016, 36(09): 2508-2515+2565.) mentioned that traditional downsampling methods such as maximum pooling and average pooling have large errors. Maximum pooling will cause the loss of background information due to discarding the part with smaller pixel values, and average pooling will cause the loss of texture information due to the pixel averaging process. Therefore, this patent uses HWD wavelet transform instead of the maximum pooling layer, and adopts the mode of splicing high and low frequency components to reduce the loss of background information and texture information.

[0004] Regarding convolution, Zhang et al. (Zhang Xin, Liu Chen, Yang Degang, et al. Rfaconv: Innovating spatital attention and standard convolutional operation [J]. arxiv preprint arxiv: 2304.03198, 2023.) mentioned that although existing spatial attention such as convolution block attention module and coordinated attention have been widely used to improve the performance of convolutional neural networks and solve the problem of convolution kernel parameter sharing, the information contained in the attention map generated by spatial attention is not enough for large-size convolution kernels. Therefore, this patent uses RFA convolution to replace part of ordinary convolution. RFA convolution not only pays attention to the spatial features of the receptive field, but also provides effective attention weights for large-size convolution kernels, significantly improving network performance with almost negligible computational cost and parameter increment.

[0005] Regarding zero-shot detection, Pan et al. (Pan Chongyu, Huang Jian, Hao Jianguo, etc. A review of weakly supervised learning methods integrating zero-shot learning and small-shot learning [J]. Systems Engineering and Electronics, 2020, 42(10): 2246-2256.) mentioned that zero-shot learning attempts to identify new category concepts that have never been seen during the training process, which requires additional semantic feature auxiliary information, such as semantic feature description vectors of the training set and the test set category to be classified, so as to achieve knowledge transfer from the training set to the test set category. Due to its inherent domain adaptation and hub degree problems, zero-shot learning also faces problems such as low recognition accuracy. Therefore, this patent adopts the open vocabulary detection architecture of Yolo-world to predict and embed the fused multi-level cross-modal text encoding information with the image encoding information, and efficiently achieve zero-shot detection.

[0006] In summary, although the zero-shot image target detection method based on deep learning has made some progress, it still needs to be improved to better meet the needs of more detection objects and very few data set training samples. To this end, this patent proposes an efficient, accurate, and highly generalized zero-shot image target detection method based on deep learning to meet the diverse image target detection needs. Summary of the invention

[0007] The present invention aims to overcome the deficiencies in the above prior art and designs a zero-sample image target detection method based on deep learning to improve the detection performance under different numbers of detection objects and training data sets. Accuracy and speed.

[0008] In order to achieve the above invention, the present invention provides a zero-sample image target detection method based on deep learning, such as Figure 1 As shown, the specific steps include:

[0009] (1) Using an automatic annotation algorithm to generate region-text pairs for the dataset;

[0010] The method of using an automatic annotation algorithm to generate region-text pairs for a data set refers to extracting noun objects from the data set, pseudo-annotating region-text pairs, and automatically generating region-text pairs by CLIP filtering;

[0011] Furthermore, the extracting of noun objects from the data set refers to replacing the N-Gram model used in extracting noun objects with a tagger pre-trained on a large number of corpora based on a hidden Markov model, and outputting the result by reducing the dimension, and only retaining the word blocks of the noun part of speech;

[0012] Furthermore, the region-text pair pseudo-annotation refers to generating region-text pair pseudo-annotations of given object nouns using a pre-trained open vocabulary detector;

[0013] Furthermore, the CLIP filtering refers to re-storing or marking boxes and filtering based on region-text pair similarity using pre-trained CLIP.

[0014] (2) Extract text embeddings with zero-shot precision using the CLIP pre-trained Transformer text encoder;

[0015] The zero-shot extraction of text embedding using the Transformer text encoder pre-trained by CLIP refers to the extraction of the corresponding text embedding W = TextEncoder(T)∈R using the Transformer text encoder pre-trained by CLIP for a given text. C∪D , where C is the number of nouns, D is the embedding dimension, and T is the text. Compared with the text-only language encoder, the CLIP text encoder provides better visual semantic capabilities to connect visual objects with text.

[0016] (3) Extract image codes through RO-Yolo-Backbone;

[0017] The image coding extraction through the improved RO-Yolo-Backbone refers to using RFA convolution to replace the ordinary convolution of Yolo-Backbone, improving network performance with extremely small computational cost and parameter increment, optimizing convolution kernel parameter sharing, and introducing OD convolution in the Concat front end of CSPLayer_2Conv. At the same time, the stride connection between the convolution and Bottleneck in CSPLayer_2Conv and the stride connection between different Bottlenecks are increased, the gradient flow is increased, the gradient variance is reduced, the convergence speed and convergence effect are improved, the smoothness and generalization of the network are improved, and the subsequent HWD-RepVL-PAN module is more conducive to the bidirectional fusion of text coding information and image coding information.

[0018] (4) Bidirectional fusion of text coding and image coding through HWD-RepVL-PAN;

[0019] The bidirectional fusion of text coding and image coding through HWD-RepVL-PAN refers to using HWD to replace the downsampling layer, improving the internal convolution of T-CSPLayer and adding a downsampling layer to improve RepVL-PAN, thereby enhancing the network's bidirectional fusion capability for text coding and image coding;

[0020] Furthermore, the use of HWD to replace the downsampling layer refers to using the HWD downsampling module to replace the three maximum pooling layers in the image pooling attention layer. The HWD downsampling module includes two parts. The first part uses Haar transform to reduce the spatial resolution, and the second part filters redundant information through convolution, batch normalization and ReLU operations.

[0021] Furthermore, the improved T-CSPLayer internal convolution refers to replacing the first convolution block internal convolution and the second convolution block internal convolution in DarknetBottleneck in T-CSPLayer with RFA convolution and OD convolution respectively, so as to enhance the bidirectional fusion capability of text encoding information and image encoding information;

[0022] Furthermore, the added downsampling layer refers to adding an image pooling attention layer at the connection between Yolo-Backbone and HWD-RepVL-PAN to enrich the perception level of text embedding for image embedding information;

[0023] (5) Image object detection using MCJ-Text Contrastive Head and regression bounding box;

[0024] The image target detection by MCJ-Text Contrastive Head and regression bounding box refers to calculating the image-text similarity of the regional contrast head according to formulas (1) to (4). The similarity measurement is mainly based on the Minkowski distance, supplemented by the cosine similarity coefficient and the Jaccard similarity coefficient.

[0025] S k,j =α1·F MD +α2·F CS +α3·F JS +β (1)

[0029] Among them, the candidate value of p is an integer greater than or equal to 1, and the value of p changes with the density and distribution shape of the network. At the same time, considering the defects of the difference in the calculation direction of the Minkowski distance, the cosine similarity coefficient and the Jaccard similarity coefficient are used to assist, aiming to weaken the differences in the encoding of objects of the same category in different images and enhance the ability to distinguish common features between individuals, thereby improving the accuracy; the initial coefficients of β, α1, α2, and α3 are 1, 1, 0.5, and 0.5, and the latter two values ​​are updated and optimized by back propagation. k represents the number of objects, and w h is the hth text embedding, ek is the k-th image embedding, F MD is the Minkowski distance formula after the encoded information is substituted, F CS is the cosine similarity coefficient formula after the encoding information is substituted, F JS It is the Jaccard similarity coefficient formula after the encoding information is substituted;

[0030] Finally, multiple targets whose position confidence is greater than the threshold are framed by the regression boundary, and non-maximum suppression is used to retain the highest scoring boundary box to achieve zero-shot image target detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of an embodiment of the present invention;

[0032] Figure 2 It is a structural diagram of an embodiment of the present invention;

[0033] Figure 3 This is a RO-Yolo-Backbone diagram of an embodiment of the present invention;

[0034] Figure 4 This is a diagram of HWD-RepVL-PAN according to an embodiment of the present invention;

[0035] Figure 5 This is an example diagram of the improved CSPLayer_2Conv, DarknetBottleneck, RFA convolution block and OD convolution block in the embodiment of the present invention;

[0036] Figure 6 This is an example diagram of the improved T-CSPLayer, image pooling attention layer, and feature representation learning block in an embodiment of the present invention; DETAILED DESCRIPTION

[0037] The technical route of the present invention is further described in detail below through specific embodiments and drawings.

[0038] like Figures 1 to 6 As shown, an embodiment of the zero-sample image target detection method based on deep learning of the present invention includes the following steps:

[0039] 1. Generate region-text pairs using automatic annotation algorithms for the dataset

[0040] The algorithm for extracting noun objects, pseudo-annotating region-text pairs, and automatically generating region-text pairs is optimized for the dataset by extracting noun objects, pseudo-annotating region-text pairs, and using pre-trained CLIP filtering to optimize the algorithm for extracting noun objects while keeping the speed almost unchanged.

[0041] Specifically, considering that the N-Gram model lacks long-term dependence when extracting nouns from short English texts, it cannot capture long-distance word relationships, and it is difficult to ensure accuracy when the number of detected objects, i.e. nouns, increases. The tagger based on the hidden Markov model pre-trained in a large corpus is more accurate and efficient for the extraction of nouns from short English texts by this method; therefore, the N-Gram model for text noun extraction is replaced with a tagger based on the hidden Markov model pre-trained in a large corpus, and the result is outputted by reducing the dimension, retaining only the word blocks of the noun part of speech. Specifically, for a short English text, modeling is performed according to the hidden Markov model, and the specific steps are as follows:

[0042] (1) The state of the hidden Markov chain at any time t is only related to the previous state, where i is a state sequence of length T and o is the corresponding observation sequence; P(i t |i t-1 ,o t-1 ,…,i1,o1)=P(i t |i t-1 ) (5)

[0043] (2) Implicit state: the part of speech corresponding to each English word or phrase can be roughly estimated; P(i t |i t ,o t ,…,i1,o1)=P(o t |i t ) (6)

[0044] (3) Finally, the joint probability distribution established by the above two assumptions is:

[0045] After completing the hidden Markov modeling of short English texts, the pre-trained open vocabulary detector GLIP-L (Grounded Language-Image Pre-training) is used to generate pseudo boxes for a given noun phrase for each image, thereby providing rough region-text pairs. The pre-trained CLIP is used to evaluate the relevance of image-text pairs and region-text pairs, and to filter out pseudo annotations and images with low relevance, and non-maximum suppression is combined to further filter redundant bounding boxes.

[0046] 2. Extract text embeddings with zero-shot using CLIP pre-trained Transformer text encoder

[0047] For a given text, we use the Transformer text encoder pre-trained by CLIP to extract the corresponding text embedding W = TextEncoder(T)∈R in zero-shotC∪D , where C is the number of nouns, D is the embedding dimension, and T is the text; the Transformer encoder makes full use of the characteristics of its Self-Attention mechanism to establish long-distance modeling dependencies and global information of short English texts. The Self-Attention calculation formula is as follows:

[0048]

[0049] Where, Q = x × W Q , K = x × W K , V = x × W V , d k is the feature dimension, B is the bias matrix, x is the input vector, W Q , W K , W V is the transformation matrix, which is determined by the initial parameters of the model.

[0050] 3. Extract image code through RO-Yolo-Backbone

[0051] Although Yolo-Backbone based on CNN has high speed and accuracy in traditional image object detection, its encoding information extraction and interaction capabilities for zero-shot image object detection tasks such as text encoding and image encoding need to be optimized. Therefore, this patent uses RFA convolution to replace the ordinary convolution of Yolo-Backbone, such as Figure 3 As shown in the figure, OD convolution is introduced in the Concat front end of CSPLayer_2Conv, and the stride connection between the convolution block and Bottleneck in CSPLayer_2Conv and the stride connection between different Bottlenecks are increased.

[0052] Specifically, for RFA convolution, the difference between RFA convolution and ordinary convolution is that RFA convolution emphasizes the importance of different features in the receptive field slider and gives priority to the receptive field space features, thereby solving the problem of parameter sharing in Yolo-Backbone ordinary convolution and improving the overall performance of the network. The RFA convolution module formula is expressed as follows:

[0053] F = Softmax(g 1×1 (AvgPool(X)))×ReLU(Norm(g k×k (X)))(9)

[0054] Among them, g i×i represents a grouped convolution of size i×i, k represents the size of the convolution kernel, Norm represents normalization, X represents the input feature map, and F represents the multiplication result of the attention map and the transformed receptive field space feature.

[0055] For OD convolution, in each convolution layer, OD convolution uses a novel multi-dimensional attention mechanism to learn four types of attention of the convolution kernel along the four dimensions of the convolution kernel space in parallel and complement each other, and gradually apply them to the corresponding convolution kernels, significantly enhancing the feature extraction capability of the basic convolution operation of Yolo-Backbone. The OD convolution module formula is expressed as follows:

[0056]

[0057] Among them, X and Y represent the input features and output features respectively, and W i represents the i-th convolution kernel composed of 4 filters, ⊙ represents the multiplication operation at different latitudes along the convolution kernel, α wi is the attention scalar, α fi , α ci and α si Respectively represent the attention along the spatial dimension of the convolution kernel, the input channel dimension, and the output channel dimension;

[0058] At the same time, if Figure 5 As shown in the figure, the stride connection between the convolution block and the Bottleneck in CSPLayer_2Conv is increased, and the stride connection between the i-th Bottleneck and the i+2-th Bottleneck is increased, where i∈1, 2, ..., n-2, which increases the gradient flow, reduces the gradient variance, improves the convergence speed and convergence effect, improves the smoothness and generalization of the network, and is more conducive to the subsequent HWD-RepVL-PAN module to bidirectionally fuse the text encoding information with the image encoding information.

[0059] 4. Bidirectional fusion of text encoding and image encoding through HWD-RepVL-PAN

[0060] like Figure 4 As shown, HWD is used to replace the downsampling layer, the internal convolution of T-CSPLayer is improved, and the downsampling layer is added to improve RepVL-PAN, thereby enhancing the network's bidirectional fusion capability for text encoding and image encoding.

[0061] Specifically, considering that the downsampling layer in RepVL-PAN is a maximum pooling layer, the maximum pooling layer will lose edge and texture information during the pooling process, which is not conducive to the bidirectional fusion of text coding and image coding. Therefore, this patent uses HWD to replace the downsampling layer. HWD contains two modules, namely, lossless feature coding block and feature representation learning block: 1) The lossless feature coding block uses the Haar wavelet transform layer to effectively reduce the spatial resolution of the feature map and retain all information. It contains low-pass decomposition filters and high-pass decomposition filters to generate approximate components, horizontal detail components, vertical detail components and diagonal detail components, and increase the number of channels of the feature map by four times; 2) The feature representation learning block is used to filter redundant information. It contains standard 1x1 convolution, batch normalization and ReLu operations.

[0062] Specifically, Figure 5 As shown, considering the internal convolution of T-CSPLayer, that is, the DarknetBottleneck convolution part, the ordinary convolution inside it is difficult to give full play to the dual interaction of text encoding and image encoding, and the convolution kernel parameter sharing and receptive field are defective. This patent changes the first ordinary convolution of Darknet Bottleneck to an RFA convolution block, and the second ordinary convolution to an OD convolution block. Specifically, the RFA convolution block and the OD convolution block have been expressed in detail in step 3 of the specific embodiment of this patent (extracting image encoding through RO-Yolo-Backbone).

[0063] Specifically, Figure 4 and Figure 6 As shown, considering that in the original RepVL-PAN, the three-size feature maps obtained by Yolo-Backbone are processed by a layer of T-CSPLayer before being processed by the image pooling attention layer to obtain the image code embedded in the text code, that is, there is a lack of interactive embedding of low-level image code to text code. This patent adds a layer of image pooling attention layer at the connection between Yolo-Backbone and RepVL-PAN, and adds the corresponding interactive embedding of the image code processed by the image pooling attention layer to the text code, aiming to enrich the dual interactive capabilities of text code and image code.

[0064] 5. Image object detection through MCJ-Text Contrastive Head and regression bounding box

[0065] Considering that the image-text similarity formula of the Text Contrastive Head only uses the L2 normal form formula for distance measurement, it is difficult to reflect the differences in the same category of objects in different image encodings, and the ability to distinguish common features between individuals is weak. Therefore, this patent uses MCJ-Text Contrastive Head and regression bounding box to detect image targets, which means that the image-text similarity of the regional contrast head is calculated according to formulas (11) to (14). The similarity measurement is mainly based on the Minkowski distance, supplemented by the cosine similarity coefficient and the Jaccard similarity coefficient. Finally, the regression bounding box defines multiple targets whose position confidence is greater than the threshold, and non-maximum suppression is used to retain the highest scoring bounding box to achieve zero-sample image target detection.

[0066] Specifically, the target text similarity formula is as follows:

[0067] S k,j =α1·F MD +α2·F CS +α3·F JS +β (11)

[0068]

[0071] Among them, the candidate value of p is an integer greater than or equal to 1, and the value of p changes with the density and distribution shape of the network, taking into account the defects of the difference in the calculation direction of the Minkowski distance. This patent uses the cosine similarity coefficient and the Jaccard similarity coefficient for assistance, aiming to weaken the differences in the encoding of objects of the same category in different images and enhance the ability to distinguish common features between individuals, thereby improving the accuracy. The initial coefficients of β, α1, α2, and α3 are 1, 1, 0.5, and 0.5. The latter two values ​​are updated and optimized by back propagation. k represents the number of objects, and w represents the number of objects. h is the hth text embedding, e k is the k-th image embedding, F MD is the Minkowski distance formula after the encoded information is substituted, F CS is the cosine similarity coefficient formula after the encoding information is substituted, F JS It is the Jaccard similarity coefficient formula after the encoding information is substituted;

[0072] Finally, if Figure 2 As shown in the figure, the target embedding is obtained by the text comparison head, the regional text matching is obtained by the target embedding and the image-aware text embedding, and the highest scoring bounding box is retained by the target embedding and the regressed bounding box using non-maximum suppression to achieve zero-shot image target detection.

[0073] Compared with the prior art, the present invention has the following beneficial effects:

[0074] (1) This patent designs RO-Yolo-Backbone based on Yolo-World. By introducing RFA convolution and OD convolution, the convolution kernel parameter sharing and receptive field range are optimized to enhance the image coding extraction capability. At the same time, the stride connection between the convolution block and the Bottleneck in CSPLayer_2Conv and the stride connection between different Bottlenecks are increased to increase the gradient flow, which is conducive to the subsequent HWD-RepVL-PAN module to bidirectionally fuse text coding information with image coding information.

[0075] (2) This patent designs the HWD-RepVL-PAN module based on Yolo-World. By introducing HWD, RFA convolution and OD convolution, the downsampling module is optimized. The information loss of image coding is reduced through Haar wavelet transform. At the same time, the image pooling attention layer is added to enhance the interactive embedding capability of low-level image coding into text coding.

[0076] (3) This patent designs the MCJ-Text Contrastive Head module based on Yolo-World. By introducing the Minkowski distance, cosine similarity coefficient and Jaccard similarity coefficient formula, the improved image-text similarity formula of the text contrast head aims to weaken the differences in the encoding of objects of the same category in different images and enhance the ability to distinguish common features between individuals, thereby improving the accuracy.

[0077] (4) This patent provides an implementable zero-shot image target detection method based on deep learning, designs a feature extraction module, an image-text encoding interaction module and a detection module, and has significant improvements in rare medical cancer and autonomous driving scenarios, especially in the era of big data where there are many tasks that lack data sets and are difficult to annotate. This provides a feasible method for zero-shot image target detection based on deep learning.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the specific implementation methods of the present invention can still be modified or replaced without departing from the spirit and scope of the present invention. Any modification or replacement should be included in the scope of protection of the claims of the present invention.

Claims

1. A zero-shot image target detection method based on deep learning, characterized in that The noun phrases extracted by the hidden Markov model are embedded in the text by the Transformer encoder pre-trained by CLIP (Contrastive Language-Image Pre-training), and feature extraction is performed by RO-Yolo-Backbone (Receptive-Field Attention Convolution Omni-Dimensional Dynamic Convolution Yolo-Backbone), and then HWD-RepVL-PAN (Haar Wavelet Downsampling Re-Parameterizable Vision-Language PAN) and region-text contrast loss, establish an interactive mapping between visual and language information, and realize zero-shot image object detection, including the following steps: (1) Generate region-text pairs by using an automatic annotation algorithm for the dataset: extract noun objects, region-text pair pseudo-annotations and automatically generate region-text pairs by using CLIP filtering; (2) Use the Transformer text encoder pre-trained by CLIP to extract text embeddings in zero-shot mode: For a given text, use the Transformer text encoder pre-trained by CLIP to extract the corresponding text embedding; (3) Extract image encoding through RO-Yolo-Backbone; (4) Bidirectionally fuse text encoding and image encoding through HWD-RepVL-PAN; (5) Perform image object detection through MCJ-TextContrastive Head (Minkowski Cosine Jaccard Text Contrastive Head) and regress bounding boxes.

2. The method for zero-sample image target detection based on deep learning according to claim 1, characterized in that: The image coding extraction through RO-Yolo-Backbone refers to using RFA (Receptive-Field Attention) convolution to replace the ordinary convolution of Yolo-Backbone, improving the network performance with extremely small computational cost and parameter increment, optimizing the convolution kernel parameter sharing, and introducing OD (Omni-Dimensional) convolution in the Concat front end of CSPLayer_2Conv. At the same time, the stride connection between the convolution block and the Bottleneck in CSPLayer_2Conv and the stride connection between different Bottlenecks are increased, and the gradient flow is increased, which is conducive to the subsequent HWD-RepVL-PAN module to bidirectionally fuse the text coding information with the image coding information.

3. The method for zero-shot image target detection based on deep learning according to claim 1, characterized in that: The bidirectional fusion of text encoding and image encoding through HWD-RepVL-PAN refers to using HWD to replace the downsampling layer, improving the internal convolution of T-CSPLayer and adding a downsampling layer to improve RepVL-PAN; the use of HWD to replace the downsampling layer refers to using the HWD downsampling module to replace the three maximum pooling layers in the image pooling attention layer. The HWD downsampling module includes two parts. The first part uses Haar transform to reduce the spatial resolution, and the second part filters redundant information through convolution, batch normalization and ReLU operations; the improved internal convolution of T-CSPLayer refers to replacing the internal convolution of the first convolution block and the internal convolution of the second convolution block in DarknetBottleneck in T-CSPLayer with RFA convolution and OD convolution respectively; the added downsampling layer refers to adding an image pooling attention layer at the connection between Yolo-Backbone and HWD-RepVL-PAN.

4. The method for zero-shot image target detection based on deep learning according to claim 1, characterized in that: The image target detection by MCJ-Text Contrastive Head and regression bounding box refers to calculating the image-text similarity of the regional contrast head according to formulas (1) to (4). The similarity measurement is mainly based on the Minkowski distance, supplemented by the cosine similarity coefficient and the Jaccard similarity coefficient. S k,j =α1·F MD +α2·F CS +α3·F JS +b (1) Among them, the candidate value of p is an integer greater than or equal to 1, and the value of p changes with the density and distribution shape of the network. Considering the defects of the difference in the calculation direction of the Minkowski distance, the cosine similarity coefficient and the Jaccard similarity coefficient are used for assistance; the initial coefficients of β, α1, α2, and α3 are 1, 1, 0.5, and 0.5 respectively, and the latter two values ​​are updated and optimized by back propagation. k represents the number of objects, and w h is the hth text embedding, e k is the k-th image embedding, F MD is the Minkowski distance after the encoding information is substituted, F CS is the cosine similarity coefficient after the encoding information is substituted, F JS It is the Jaccard similarity coefficient after the encoding information is substituted; finally, multiple targets whose position confidence is greater than the threshold are framed by the regression boundary, and the highest scoring boundary box is retained by non-maximum suppression to achieve zero-sample image target detection.

Images