Deep learning-oriented zero sample target detection method and system

Abstract

The invention belongs to the field of zero sample target detection, and discloses a deep learning-oriented zero sample target detection method and system. The method comprises the following steps: establishing a zero sample target detection model comprising a feature extraction module, a cross-modal alignment module and a course learning module; extracting image features and text features through a feature extraction module; inputting the image features and the text features into a cross-modal alignment module; carrying out cross-modal similarity calculation on image feature embedding and text feature embedding; the obtained cross-modal similarity is imported into a course learning module; fusion of image features and text features is enhanced through a course learning module, and the difficulty of course learning is dynamically adjusted through knowledge distillation in the back propagation process; and obtaining a zero sample target detection model and a detection result. According to the method, the advantages of high performance and high generalization ability of a zero sample learning large model are reserved, mutual complementation and enhancement among cross-modal features are realized, and the detection performance and robustness of the model are improved.

Description

Technical Field

[0001] The present invention belongs to the field of zero-sample target detection and discloses a zero-sample target detection method and system for deep learning. Background Art

[0002] The object detection task is one of the cornerstones of the field of computer vision. However, most existing object detection methods are under the paradigm of supervised learning. The success of the model often depends on a large number of labeled data sets, which severely limits the scalability of the object detection system. In most real-world application scenarios, it is unrealistic to obtain enough labeled samples covering all categories. In some extreme scenarios, it is even difficult to obtain any samples of the detection object.

[0003] In recent years, the Zero-shot Object Detection (ZSD) method has been proposed, which aims to identify unseen target categories during the test phase, thereby improving the bottleneck problem of existing systems that rely on large amounts of labeled data. In the ZSD model, the visual-semantic embedding space is usually used to bridge visual information with semantic information. This space can represent the mapping from visual features to semantic attributes. Even when the algorithm encounters objects of unseen categories, it can rely on the knowledge learned from the mapping space for effective detection.

[0004] As an emerging field, zero-shot object detection has made some progress in theory and practice, but it still faces many challenges. Most existing methods encode text and image modalities separately, ignoring the complex correlation between modalities. In addition, the model learning strategy is too simple and direct, making it difficult to achieve accurate alignment of cross-modal features, and the visual-semantic gap is too large. Therefore, it is urgent for technicians in this field to solve the above technical problems. Summary of the invention

[0005] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned prior art and to provide a method and system for zero-sample target detection in deep learning. The method and system for zero-sample target detection under the condition that samples are difficult to obtain in deep learning solve the problem of scarcity of zero samples in deep learning, significantly improve the performance and practicality of zero-sample target detection, and bring important technological progress to the field of cross-modal learning and target detection.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A zero-sample target detection method for deep learning, the method comprising:

[0008] S1: Establish a zero-shot object detection model including feature extraction module, cross-modal alignment module and curriculum learning module;

[0009] S2: Extract image features and text features through feature extraction module;

[0010] S3: Input the image features and text features extracted in step S2 into the cross-modal alignment module to obtain image feature embedding and text feature embedding for calculation;

[0011] S4: Calculate the cross-modal similarity between the image feature embedding and the text feature embedding in step S3, and import the obtained cross-modal similarity into the course learning module;

[0012] S5: Enhance the fusion of image features and text features through the progressive visual semantic loss function in the course learning module, and dynamically adjust the difficulty of course learning through knowledge distillation during the back-propagation process;

[0013] S6: Get the test result.

[0014] Furthermore, in step S2, the image features and text features are extracted by a feature extraction module, specifically:

[0015] S21: The YOLOv5 model receives the preprocessed image and adjusts the image size to match the input requirements of the YOLOv5 model;

[0016] The YOLOv5 model includes an input layer, a Backbone network, a Neck network, and a Head detection head;

[0017] The Backbone network adopts a variant of CSPNet including multiple convolutional layers and residual structures;

[0018] The convolutional layers of the CSPNet include standard convolutional layers and depth-separable convolutional layers. The standard convolutional layers use several 3×3 convolutional layers, and batch normalization and activation function operations are performed after convolution to extract low-level features in the image.

[0019] The depth-wise separable convolutional layer uses several 3×3 convolutional layers to perform spatial convolution on each input channel, and then uses a 1x1 convolution to fuse the outputs of these channels, thereby reducing the number of parameters and computational complexity;

[0020] The residual structure includes several 3×3 convolutional layers, and the residual structure introduces a split-connection strategy based on the basic residual unit. Specifically, the input feature map is first split into two parts in the channel, one part is directly passed to the output, and the other part is passed through a series of 3×3 convolution and residual operations, and finally the two parts are merged again in the channel dimension;

[0021] The Neck network includes several 1×1 convolutional layers and is connected using the PANet structure. The specific workflow is as follows:

[0022] Bottom-up path: Starting from the last output of the Backbone network, the feature is compressed through a 1x1 convolutional layer, and then the size of the feature map is enlarged by upsampling to match the size of the feature map of the previous layer, and then the upsampled feature map is fused with the previous feature layer;

[0023] Top-down path: After all bottom-up feature fusion is completed, the top-down path starts working, compressing the features through a 1x1 convolutional layer, and then using downsampling to reduce the spatial dimension of the feature map to match the size of the next layer, and the feature map will be fused with the next feature layer.

[0024] Feature fusion: Feature fusion uses additive fusion and serial fusion. Additive fusion can keep the number of channels unchanged, while serial fusion increases the number of channels. The number of channels is then adjusted using the subsequent 1x1 convolutional layer according to the specific situation in order to unify the feature depth of each layer.

[0025] The Head detection head includes several 3x3 convolutional layers and multiple 1x1 convolutional layers. These convolutional layers are connected sequentially, followed by batch normalization and activation functions. After feature extraction, multiple 1x1 convolutional layers are used to generate the final output. The number of channels of this output layer is set according to the prediction requirements. The calculation formula is: (num_classes+5)*num_anchors, where num_classes is the number of categories, num_anchors is the number of anchor boxes used for each grid unit, and 5 represents the 4 coordinate values ​​of each anchor box plus a confidence level.

[0026] S22: In the text encoder of the CLIP model, the text is first segmented into vocabulary tokens using tokenization, and then each token is mapped into a text embedding vector in a high-dimensional space and positionally encoded.

[0027] The CLIP model includes a Transformer network structure, a self-attention mechanism, and a pooling layer. The text embedding vector is sent to a multi-layer Transformer structure, and the context information of the text is processed through the self-attention mechanism.

[0028] The output result is converted into a semantic embedding vector represented by a single vector through a pooling layer;

[0029] The text encoder of the CLIP model is used to encode these descriptions to generate semantic embedding vectors representing text features. This process is usually completed when the CLIP model is offline and saved as a lookup table.

[0030] YOLOv5, with its advantages of high accuracy, multi-target detection capability and easy deployment in industrial applications, ensures the efficiency and practicality of image feature extraction. This means that when faced with complex scenes, the model can quickly and accurately locate and identify multiple targets in the image, laying a solid foundation for subsequent zero-shot target detection. As an advanced semantic-image pre-training large model, the CLIP model combines text and image information for cross-modal learning, and has excellent generalization ability, robustness and interpretability. Using CLIP as a text feature extractor can ensure the high quality of text features. These features are rich in semantic information, which is conducive to understanding image content and category reasoning. By adjusting the YOLOv5 output structure to make it similar to the CLIP embedding space, the image features and text features are structurally comparable, which is conducive to the alignment of cross-modal features and subsequent joint modeling. This consistency enhances the model's ability to comprehensively understand image and text information, and promotes the effective integration of visual information and semantic information. Through such a design, even in the absence of samples of a specific category, the model can recognize new category targets to a certain extent based on the joint representation of image features and text features, breaking through the traditional supervised learning's reliance on large amounts of labeled data and improving the model's flexibility and generalization ability in practical applications.

[0031] Furthermore, the image feature embedding and text feature embedding calculated in step S3 are specifically:

[0032] The semantic embedding vector generated by the CLIP model text encoder is used as text feature embedding;

[0033] By adjusting the Head detection head of the YOLOv5 model to fuse image and text features, a fully connected layer is added to the Head detection head to transform the feature space, so that the image feature embedding dimension output by the detection head of the YOLOv5 model is consistent with the text feature embedding vector dimension generated by the CLIP model. represents text feature embedding, where d t Represents the dimension of text feature embedding, and transforms the dimension of image feature embedding to obtain Image feature embedding of shapes.

[0034] By transforming the dimensions of image features, we ensure that they are consistent in shape with text features, which is the premise of cross-modal fusion. This consistency enables visual information and semantic information to be compared and integrated in the same feature space, which helps the model to more effectively understand the correspondence between image content and text description. The consistency of cross-modal features allows the model to use both visual and semantic information for joint modeling. Through technologies such as cross-modal attention mechanisms, we can deeply explore the association information between image regions and text descriptions, and enhance the model's understanding and expression capabilities of complex scenes. After the dimensions are consistent, a unified loss function (such as progressive visual semantic matching loss) can be used to guide model learning, which helps the model to adjust weights more accurately during training and optimize the matching of cross-modal features, thereby improving the model's performance in zero-shot object detection tasks. The introduction of the curriculum learning strategy further stabilizes the training process and improves the generalization and robustness of the model by gradually increasing the learning difficulty. The precise alignment of cross-modal features helps to narrow the gap between visual features and semantic features (i.e., the visual-semantic gap), so that the model can more accurately identify targets based on previously learned visual-semantic mapping relationships when faced with unseen categories, reducing its dependence on large amounts of labeled data.

[0035] Furthermore, in step S4, the image feature embedding and the text feature embedding in step S3 are subjected to cross-modal similarity calculation, specifically:

[0036] The similarity between image feature embedding and text feature embedding is calculated through the softmax function, which includes the following steps:

[0037] S41: Construct a cross-attention layer: The cross-attention layer accepts the input image features and text features, and passes them through three different unbiased linear layers to generate the required query (Query, Q), key (Key, K) and value (Value, V). Let the transposed matrix K of Q and K be T Perform a dot product operation and process it through the softmax function to obtain an attention weight matrix to describe the relationship between image features and text features; multiply the weight matrix by V to obtain the final image feature embedding F affected by the text feature information v '; This process is described as:

[0038] Among them, F v 、F t are the input image feature vector, text feature vector, and d k is a scaling factor, which is the inverse of the square root of the key vector dimension and is used to avoid gradient problems caused by higher dimensions in dot product operations;

[0039] S42: Construct self-attention layer: Use three unbiased linear layers to simultaneously generate its own Q′, K′ and V′; the new image feature embedding can be calculated through the self-attention weight matrix;

[0040] The output of the cross-modal alignment module adopts a residual connection mechanism: F v =F v +f self (F v ′).

[0041] By constructing a cross-attention layer, image features and text features interact at a higher level, achieving a deep fusion of visual information and semantic information. The dot product operation and the attention weight matrix generated by the softmax function can highlight the part of the image that is most relevant to the text description, thereby improving the accuracy and depth of the model's recognition of the target object. The self-attention layer further refines the contextual relationship within the image features through the query, key and value generated by itself, and strengthens the model's ability to distinguish the importance of image regions. This helps the model to understand the image content more finely and improve the ability to locate and identify targets through the guidance of semantic information without direct samples. The use of the scaling factor dk (i.e., the inverse of the square root of the key vector dimension) effectively alleviates the gradient vanishing problem that may be encountered in the dot product of high-dimensional vectors, ensures the stability and effectiveness of the training process, and enables the model to learn the complex associations between cross-modal features more smoothly. The use of the residual connection mechanism in the self-attention layer helps the model retain the original feature information while learning new features, avoids the gradient explosion or learning stagnation problems that may occur during training, and enhances the stability of the model and its resistance to noise.

[0042] Further, in step S41, when the unbiased linear layer is linearly transformed, it is specifically as follows:

[0043] Q=W Q *F v

[0044] K=W k *F t

[0045] V=W V *F t

[0046] Where W Q , W k and W V is the weight matrix of the linear layer, and the output dimension of each linear layer matches the expected Q, K, V dimensions;

[0047] The linear transformation in step S42 indicates that the self-attention mechanism uses three unbiased linear layers to simultaneously generate its own Q′, K′, and V′:

[0048] Q′=W q′ *F v '

[0049] K′=W k′ *F v '

[0050] V′=W v′ *F v ′.

[0051] Through linear transformation and self-attention mechanism, the model can extract and highlight key features in the input data, enhance the deep understanding of image regions and text descriptions, and allow features to transfer and aggregate information between different positions, promote the interaction between features, and help capture global context and local details. The self-attention weight intuitively shows how the model allocates attention and which parts contribute most to the prediction results, thereby enhancing the interpretability of the model. Through the deep integration of cross-modal information, the model can learn complex cross-modal associations and improve the detection performance on unseen categories, that is, zero-sample target detection capabilities. The residual connection mechanism of the self-attention mechanism helps stabilize the training process, avoid gradient vanishing or explosion problems, accelerate convergence, and improve learning efficiency.

[0052] Furthermore, in step S5, the fusion of image features and text features is enhanced through the progressive visual semantic loss function in the course learning module, and the difficulty of course learning is dynamically adjusted through knowledge distillation during the back propagation process, specifically:

[0053] S51: Given an image feature f i v And all text feature sets in this mini-batch Use the softmax function to calculate f i v and its text features is the predicted probability p of a matching pair i,j :

[0054]

[0055] In this formula, τ is the temperature hyperparameter, which controls the prediction result f i v and f j t The smoothness of the probability distribution of i,j ∈{0,1} is the real matching label, and the matching probability p is predicted by the model i,j∈[0,1], and then introduce a very small number ∈ to avoid numerical problems, integrated into the relative entropy divergence, and obtain the cross-modal matching loss from image to text in this small batch of data

[0056] The progressive visual semantic loss function pushes away the mismatching negative samples while increasing the relevance of the matching samples:

[0057] S52: Dynamic adjustment during training is achieved through the temperature hyperparameter τ in knowledge distillation;

[0058] Increase the total model loss through stochastic gradient descent;

[0059] By increasing the loss value of the zero-shot object detection model, the zero-shot object detection model pays more attention to difficult samples;

[0060] The update method of τ can be expressed as:

[0061] Among them, μ represents the learning rate, λ is the coefficient that gradually increases according to the cosine scheduling as the training progresses, so as to achieve the purpose of increasing task difficulty; τ continues to increase with the increase of training rounds, and the adversarial effect is also strengthened accordingly, so as to achieve the purpose of course learning; finally, when τ increases to the preset maximum threshold, the learning difficulty of the target detection model reaches its peak within the range of human control.

[0062] By dynamically adjusting the temperature hyperparameter τ, the model can gradually transition from a relatively loose (smooth probability distribution) learning environment to a more stringent (sharp probability distribution) environment. This design allows the model to more easily explore possible matching relationships in the early stages of training, and then focus on accurately distinguishing matching and non-matching samples, which improves the adaptability and learning efficiency of the model at different training stages. Dynamically adjusting the loss function and learning rate strategy helps the model maintain sensitivity to sample diversity during the learning process and avoid falling into local optimal solutions too early, especially in scenarios with scarce data or high noise. This method can effectively alleviate the overfitting problem and improve the generalization ability of the model. By increasing the focus on difficult samples (i.e., by increasing the loss value), the model can more effectively utilize training resources and focus on solving difficulties and ambiguities in the learning process, which not only improves learning efficiency, but also helps to discover and correct potential errors in the model, further improving the overall performance. Gradually increasing the learning difficulty according to the principle of "from simple to complex" is similar to the human learning process, which helps the model gradually accumulate knowledge and gradually improve its ability to solve complex problems. In the cross-modal matching task, it is equally important to effectively push away unmatched negative samples and attract matching positive samples. The loss function design adopted not only increases the correlation between matching samples, but also effectively separates unmatched samples, which is crucial to improving the accuracy of cross-modal retrieval, recognition and other tasks.

[0063] In addition, the present invention also discloses a zero-sample target detection system for deep learning, which is based on the zero-sample target detection method for deep learning as the operating principle, and includes a feature extraction module, a cross-modal alignment module and a course learning module; the feature extraction module is used to extract image features and text features of the sample, and the cross-modal alignment module realizes that the image features and the text features are consistent in dimension, and the cross-modal alignment module also includes a visual semantic relationship enhancement module, and the visual semantic relationship enhancement module realizes the complementarity and enhancement between image features and text features, effectively narrowing the gap between vision and semantics, and the course learning module accurately controls the learning difficulty in the cross-modal alignment process, and realizes the effect of gradually increasing learning difficulty.

[0064] The present invention has the following beneficial effects:

[0065] 1. The present invention proposes a zero-sample target detection method and system for deep learning. By combining the efficient target detection capability of YOLOv5 and the cross-modal understanding capability of the CLIP model, the method can detect new category targets without specific category training samples, greatly reducing the dependence on large-scale annotated data. The visual semantic relationship enhancement module uses the cross-modal attention mechanism to promote the deep interaction and complementarity between image features and text descriptions, and enhance the model's ability to understand complex scenes.

[0066] 2. The local semantic alignment strategy further ensures that the model can accurately capture subtle visual-language correspondences, improving the accuracy and robustness of detection. The introduced loss function and curriculum learning strategy not only consider the distinction between positive and negative samples, but also dynamically adjust the learning difficulty through knowledge distillation, providing the model with a learning path from easy to difficult, which helps the model to converge quickly in the early stages of training, while deeply exploring complex patterns in the data in the later stages, ultimately achieving better generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a flow chart of a zero-sample target detection method based on cross-modal learning provided by an embodiment of the present invention.

[0068] Figure 2 It is an overall framework diagram of the system provided by an embodiment of the present invention.

[0069] Figure 3 It is a structural diagram of a visual semantic relationship enhancement module provided in an embodiment of the present invention.

[0070] Figure 4 It is the zero-sample detection result on the COCO dataset provided by an embodiment of the present invention.

[0071] Figure 5 It is a visualization diagram of the generalized zero-sample detection result of the present invention. DETAILED DESCRIPTION

[0072] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific preferred embodiments.

[0073] In the description of the present invention, it should be understood that the terms "left side", "right side", "upper part", "lower part" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. "First", "second" and the like do not indicate the importance of the components, and therefore cannot be understood as limiting the present invention. The specific dimensions used in this embodiment are only for illustrating the technical solution, and do not limit the scope of protection of the present invention.

[0074] Reference Figure 1 It can be seen that the zero-sample target detection method and system provided by the embodiment of the present invention for deep learning under the condition that samples are difficult to obtain include the following steps:

[0075] In the feature extraction module, the input image is processed by the feature extractor of YOLOv5 to generate an image feature vector, which is then processed by the offline text encoder of CLIP to obtain a text feature vector, and the shape of the vector is made consistent in one dimension through dimensional transformation;

[0076] A visual semantic relationship enhancement module is introduced for the object detector. The cross-modal attention mechanism is used to integrate visual and semantic information for joint modeling, and the overall relevance is inferred through local semantic alignment, thereby achieving the purpose of mutual complementation and enhancement of cross-modal relationships.

[0077] A progressive visual semantic matching loss is introduced into the training process. By calculating the KL divergence of the projection compatible distribution and the normalized matching distribution defined by positive and negative samples and imitating the human curriculum learning process, the learning difficulty in the cross-modal alignment process is precisely controlled, thereby achieving the purpose of improving the stability and performance of the target detection model in the zero-sample migration process.

[0078] Reference Figure 2 and Figure 3 ,It can be seen that, first, through the feature extraction module, it consists of two parts: visual feature extractor (TOLR) block and text feature extractor (CLIP). These two modules are responsible for extracting useful feature information from images and texts respectively;

[0079] Then, these extracted features are fed into the intermediate cross-modal alignment module. The function of this module is to align data of different modalities (such as images and text) so that they can match each other;

[0080] Next, these aligned features are fed into the relational reasoning module on the right, where the relationships between them are inferred based on these features.

[0081] Finally, these inferred relationships are fed into the top deep learning module, where the backpropagation algorithm is used for training and the loss function is calculated to evaluate the model’s performance.

[0082] In one embodiment, the present invention proposes a zero-sample target detection method and system for deep learning to address the problems that existing methods ignore the correlation between cross-modal information and the low efficiency of zero-sample migration. Specifically, first, YOLOv5 and CLIP are respectively used to extract image feature embedding and text feature embedding of the input image. Then, the visual and semantic information are integrated through a cross-modal attention mechanism for joint modeling to mine the correlation information between the two. Finally, the progressive visual semantic matching loss is used to optimize the model training process through a curriculum learning strategy. The present invention retains the advantages of high performance and high generalization ability of the large zero-sample learning model, realizes the mutual complementation and enhancement between cross-modal features, and improves the detection performance and robustness of the model.

[0083] In one embodiment, YOLOv5 is the target detection algorithm with the strongest comprehensive performance in the current industrial application field, with the advantages of high accuracy, multi-target detection, and easy deployment. Therefore, the present invention selects the YOLOv5 model as an image feature extractor. The YOLOv5 detection head output structure is modified by imitating the CLIP encoder structure, so that the input image passes through YOLOv5 to generate an image feature embedding. CLIP is the most advanced semantic-image pre-training large model, which combines the information of text and image for cross-modal learning, and has the advantages of high generalization ability, high robustness, and interpretability. Therefore, the present invention selects the CLIP model as a text feature extractor. The input image generates a text feature embedding through the CLIP offline encoder.

[0084] In order to facilitate the alignment of the two and the calculation of loss minimization, the shape of the output image embedding feature must be similar to the embedding space of CLIP. Let P∈RH×W×d represent the image feature vector, where H and W represent the spatial dimensions of the feature map, and d represents the number of feature channels at each spatial position; Represents text features, where d t Represents the dimension of text embedding. It is necessary to transform the dimension of the image feature P and collapse the spatial dimension H×W into a single dimension d v , and adjust the number of channels to match the dimension d of the text features t , and then get the transformed image features

[0085] In one embodiment, the criss-cross attention layer takes the input features and passes them through three different unbiased linear layers to generate the required query (Query, Q), key (Key, K), and value (Value, V). This linear transformation can be expressed as:

[0086] Q=W Q *F v

[0087] K=W k *F t

[0088] V=W V *F t

[0089] Where W Q , W k and W V is the weight matrix of the linear layer, and the output dimension of each linear layer matches the expected dimensions of Q, K, and V. Let the transposed matrix K of Q and K be TPerform a dot product operation and process it through the softmax function to obtain the attention weight matrix, which describes the relationship between image features and text features. Finally, multiply the weight matrix by V to obtain the final image feature embedding F with weighted text feature information. v ′. This process can be formally described as:

[0090]

[0091] Among them, d k is a scaling factor, which is the inverse of the square root of the key vector dimension, and is used to avoid gradient problems caused by higher dimensions in dot product operations. v ′ has captured the dependency information between the two modalities, and the self-attention layer can deepen the model's understanding of the contextual information in the sequence. Unlike cross attention, the self-attention mechanism uses three unbiased linear layers to simultaneously generate its own Q′, K′, and V′:

[0092] Q′=W q′ *F v '

[0093] K′=W k′ *F v '

[0094] V′=W v′ *F v '

[0095] Similarly, the new image feature embedding can be calculated through the self-attention weight matrix. In order to ensure the stability of the training process, prevent gradient explosion, and maintain the diversity of features at different levels, the output of the module adopts a residual connection mechanism:

[0096] F v =F v +f self (F v ′)

[0097] In one embodiment, given an image feature f i v And all text feature sets in this mini-batch You can use a softmax function to calculate f i v and one of the text features is the predicted probability p of a matching pair i,j :

[0098]

[0099] In this formula, τ is the temperature hyperparameter, which controls the prediction result fi v and The smoothness of the probability distribution of τ. In the context of knowledge distillation, the size of τ directly affects the difficulty of model learning, and thus affects the distillation results: the smaller the τ value (1>τ>0), the sharper the distribution and the lower the learning difficulty; the larger the τ value (τ>1), the smoother the distribution and the higher the learning difficulty.

[0100] Known y i,j ∈{0,1} is the real matching label, and the matching probability p is predicted by the model i,j ∈[0,1], and then introduce a very small number ∈ to avoid numerical problems, and integrate them into the KL divergence, we can get the cross-modal matching loss from image to text in this small batch of data This loss function can increase the correlation of matching samples while pushing away mismatched negative samples:

[0101]

[0102] According to the idea of ​​curriculum learning, the learning difficulty of the target detector should also change dynamically in the process of learning cross-modal mapping, and follow the principle of going from easy to difficult. To this end, this method starts with the temperature hyperparameter τ in the formula, so that it can be dynamically adjusted during the training process. Its update strategy is similar to the adversarial learning mechanism in the generative adversarial network (GAN). Specifically, the cross-modal matching loss from image to text is used. For example.

[0103] During the training process, the overall optimization goal of the model is to minimize the loss Ideally, it meets:

[0104]

[0105] However, the update direction of the hyperparameter τ is opposite to that of the model as a whole, and it is intentionally increased by stochastic gradient descent (SGD). The "adversarial effect" of τ here does not reduce the learning efficiency of the model. On the contrary, it increases the loss value and forces the model to pay more attention to samples that are more difficult for the teacher model. The update method of τ can be expressed as:

[0106]

[0107] Among them, μ represents the learning rate, and λ is a coefficient that gradually increases with the training process according to the cosine schedule, which also plays the role of increasing the difficulty of the task. τ continues to increase with the increase of training rounds, and the adversarial effect is also strengthened accordingly, so as to achieve the purpose of course learning. Finally, when τ increases to the preset maximum threshold, the learning difficulty of the target detection model reaches the peak within the range of human control.

[0108] In one embodiment, in order to demonstrate the performance of the present invention, the algorithm is tested on the ZSD task and the GZSD task on the 65 / 15 and 48 / 17 splits of the COCO dataset, respectively. The results are shown in Tables 1 and 2:

[0109] Table 1 ZSD experimental results on two segmentations of COCO dataset

[0110]

[0111] Table 2 GZSD experimental results on two segmentations of COCO dataset

[0112]

[0113] In order to verify the effectiveness of this research method, Table 1 compares it with several state-of-the-art ZSD methods. It can be seen from Table 1 that this method has improved recall and precision compared with existing methods such as PL, BLC, ZSI, GRAN, ContrastZSD, etc., especially compared with the baseline model ZSD-YOLO, the mAP indicators are increased by 1.8 and 2.0 respectively, achieving an improvement of 13.4% and 10.9%.

[0114] In the GZSD task, the proposed method also significantly outperforms existing research work. As shown in Table 2, under the 48 / 17 category division, the mAP of the unseen class is improved by 1.9, and the harmonic mean accuracy is improved by 2.0; under the 65 / 15 category division, the mAP of the unseen class is improved by 2.8, and the mAP of HM is improved by 2.4.

[0115] Generally speaking, the performance improvement of the method of the present invention on unseen categories is more significant. This is mainly because this study aims to enhance the model's ability to mine and utilize semantic information of unseen categories, thereby enhancing the cross-modal relevance of unseen categories. In comparison, although it is also helpful for the detection of visible categories, this study did not perform special optimization processing on it, and the detection of visible categories still mainly depends on the performance of the zero-sample classifier and the target detector itself.

[0116] Visualization effects such as Figure 4 , Figure 5 As shown:

[0117] The three columns from left to right are: original image, ZSD-YOLO model effect diagram, and this method effect diagram. From Figures (1) and (2), we can see that the existing model has a low detection sensitivity for black and white or grayscale images, while this method effectively improves the recognition ability in this regard; Figures (1) and (3) prove the advantage of this method in capturing inter-class association information, and can effectively identify and associate the interaction between humans and sports equipment; in Figure (4), the model successfully detects the difficult example - potted plants, reflecting the progress of this method in processing difficult samples; in Figure (5), this method avoids possible category confusion by optimizing inter-class association information; finally, Figure (6) demonstrates the superiority of this method in high-density target detection scenarios. It can be seen that the proposed method can significantly improve the performance of zero-shot target detection.

[0118] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A zero-shot target detection method for deep learning, used for zero-shot target detection under the condition that samples are difficult to obtain in deep learning, characterized in that: The method comprises: S1: Establish a zero-shot object detection model including feature extraction module, cross-modal alignment module and curriculum learning module; S2: Extract image features and text features through feature extraction module; S3: Input the image features and text features extracted in step S2 into the cross-modal alignment module to obtain image feature embedding and text feature embedding for calculation; S4: Calculate the cross-modal similarity between the image feature embedding and the text feature embedding in step S3, and import the obtained cross-modal similarity into the course learning module; S5: Enhance the fusion of image features and text features through the progressive visual semantic loss function in the course learning module, and dynamically adjust the difficulty of course learning through knowledge distillation during the back-propagation process; S6: Get the test result.

2. The method for zero-sample target detection in deep learning according to claim 1, characterized in that: In step S2, the image features and text features are extracted by the feature extraction module, specifically: S21: The YOLOv5 model receives the preprocessed image and adjusts the image size to match the input requirements of the YOLOv5 model; The YOLOv5 model includes an input layer, a Backbone network, a Neck network, and a Head detection head; The Backbone network adopts a variant of CSPNet including multiple convolutional layers and residual structures; The convolutional layers of the CSPNet include standard convolutional layers and depth-separable convolutional layers. The standard convolutional layers use several 3×3 convolutional layers, and batch normalization and activation function operations are performed after convolution to extract low-level features in the image. The depth-wise separable convolutional layer uses several 3×3 convolutional layers to perform spatial convolution on each input channel, and then uses a 1x1 convolution to fuse the outputs of these channels, thereby reducing the number of parameters and computational complexity; The residual structure includes several 3×3 convolutional layers, and the residual structure introduces a split-connection strategy based on the basic residual unit. Specifically, the input feature map is first split into two parts in the channel, one part is directly passed to the output, and the other part is passed through a series of 3×3 convolution and residual operations, and finally the two parts are merged again in the channel dimension; The Neck network includes several 1×1 convolutional layers and is connected using the PANet structure. The specific workflow is as follows: Bottom-up path: Starting from the last output of the Backbone network, the feature is compressed through a 1x1 convolutional layer, and then the size of the feature map is enlarged by upsampling to match the size of the feature map of the previous layer, and then the upsampled feature map is fused with the previous feature layer; Top-down path: After all bottom-up feature fusion is completed, the top-down path starts working, compressing the features through a 1x1 convolutional layer, and then using downsampling to reduce the spatial dimension of the feature map to match the size of the next layer, and the feature map will be fused with the next feature layer. Feature fusion: Feature fusion uses additive fusion and serial fusion. Additive fusion can keep the number of channels unchanged, while serial fusion increases the number of channels. The number of channels is then adjusted using the subsequent 1x1 convolutional layer according to the specific situation in order to unify the feature depth of each layer. The Head detection head includes several 3x3 convolutional layers and multiple 1x1 convolutional layers. These convolutional layers are connected sequentially, followed by batch normalization and activation functions. After feature extraction, multiple 1x1 convolutional layers are used to generate the final output. The number of channels of this output layer is set according to the prediction requirements. The calculation formula is: (num_classes+5)*num_anchors, where num_classes is the number of categories, num_anchors is the number of anchor boxes used for each grid unit, and 5 represents the 4 coordinate values ​​of each anchor box plus a confidence level. S22: In the text encoder of the CLIP model, the text is first segmented into vocabulary tokens using tokenization, and then each token is mapped into a text embedding vector in a high-dimensional space and positionally encoded. The CLIP model includes a Transformer network structure, a self-attention mechanism, and a pooling layer. The text embedding vector is sent to a multi-layer Transformer structure, and the context information of the text is processed through the self-attention mechanism. The output result is converted into a semantic embedding vector represented by a single vector through a pooling layer; The text encoder of the CLIP model is used to encode these descriptions to generate semantic embedding vectors representing text features. This process is usually completed when the CLIP model is offline and saved as a lookup table.

3. The method for zero-sample target detection in deep learning according to claim 1, characterized in that: The image feature embedding and text feature embedding calculated in step S3 are specifically: The semantic embedding vector generated by the CLIP model text encoder is used as text feature embedding; By adjusting the Head detection head of the YOLOv5 model to fuse image and text features, a fully connected layer is added to the Head detection head to transform the feature space, so that the image feature embedding dimension output by the detection head of the YOLOv5 model is consistent with the text feature embedding vector dimension generated by the CLIP model. represents text feature embedding, where d t Represents the dimension of text feature embedding, and transforms the dimension of image feature embedding to obtain Image feature embedding of shapes.

4. The zero-sample target detection method for deep learning under the condition that samples are difficult to obtain according to claim 1, characterized in that: In step S4, the image feature embedding and the text feature embedding in step S3 are subjected to cross-modal similarity calculation, specifically: The similarity between image feature embedding and text feature embedding is calculated through the softmax function, which includes the following steps: S41: Construct a cross-attention layer: The cross-attention layer accepts the input image features and text features, and passes them through three different unbiased linear layers to generate the required query (Query, Q), key (Key, K) and value (Value, V). Let the transposed matrix K of Q and K be T Perform a dot product operation and process it through the softmax function to obtain an attention weight matrix to describe the relationship between image features and text features; multiply the weight matrix by V to obtain the final image feature embedding F affected by the text feature information v '; This process is described as: Among them, F v 、F t are the input image feature vector, text feature vector, and d k is a scaling factor, which is the inverse of the square root of the key vector dimension and is used to avoid gradient problems caused by higher dimensions in dot product operations; S42: Construct self-attention layer: Use three unbiased linear layers to simultaneously generate its own Q′, K′ and V′; the new image feature embedding can be calculated through the self-attention weight matrix; The output of the cross-modal alignment module adopts a residual connection mechanism: F v =F v +f self (F v ′).

5. The method for zero-sample target detection in deep learning according to claim 4, characterized in that: In step S41, when the unbiased linear layer is linearly transformed, specifically: Q=W Q *F v K=W k *F t V=W V *F t Where W Q , W k and W V is the weight matrix of the linear layer, and the output dimension of each linear layer matches the expected Q, K, V dimensions; The linear transformation in step S42 indicates that the self-attention mechanism uses three unbiased linear layers to simultaneously generate its own Q′, K′, and V′: Q′=W q′ *F v ′ K′=W k′ *F v ′ V′=W v′ *F v ′。 6. The method for zero-sample target detection in deep learning according to claim 1, characterized in that: In step S5, the fusion of image features and text features is enhanced through the progressive visual semantic loss function in the course learning module, and the difficulty of course learning is dynamically adjusted through knowledge distillation during the back propagation process, specifically: S51: Given image features And all text feature sets in this mini-batch Use the softmax function to calculate and its text features is the predicted probability p of a matching pair i,j : In this formula, τ is the temperature hyperparameter that controls the prediction results. and The smoothness of the probability distribution of i,j ∈{0,1} is the real matching label, and the matching probability p is predicted by the model i,j ∈[0,1], and then introduce a very small number ε to avoid numerical problems, integrated into the relative entropy divergence, and obtain the cross-modal matching loss from image to text in this small batch of data The progressive visual semantic loss function pushes away the mismatching negative samples while increasing the relevance of the matching samples: S52: Dynamic adjustment during training is achieved through the temperature hyperparameter τ in knowledge distillation; Increase the total model loss through stochastic gradient descent; By increasing the loss value of the zero-shot object detection model, the zero-shot object detection model pays more attention to difficult samples; The update method of τ can be expressed as: Among them, μ represents the learning rate, λ is the coefficient that gradually increases according to the cosine scheduling as the training progresses, so as to achieve the purpose of increasing task difficulty; τ continues to increase with the increase of training rounds, and the adversarial effect is also strengthened accordingly, so as to achieve the purpose of course learning; finally, when τ increases to the preset maximum threshold, the learning difficulty of the target detection model reaches its peak within the range of human control.

7. A zero-sample target detection system for deep learning, implemented based on a zero-sample target detection method for deep learning according to any one of claims 1 to 6, characterized in that: It includes a feature extraction module, a cross-modal alignment module and a course learning module; the feature extraction module is used to extract the image features and text features of the sample, the cross-modal alignment module realizes the consistency of the image features and the text features in dimension, the cross-modal alignment module also includes a visual semantic relationship enhancement module, the visual semantic relationship enhancement module realizes the complementarity and enhancement between the image features and the text features, effectively narrowing the gap between vision and semantics, and the course learning module accurately controls the learning difficulty in the cross-modal alignment process, achieving the effect of gradually increasing learning difficulty.

Images