An open set target detection method based on fusion text coding

Abstract

The present application relates to the technical field of image processing, solves the technical problem that open set detection cannot be realized when facing unknown targets newly appearing in a detection image, and particularly relates to an open set target detection method based on fused text information, comprising: obtaining text template information preset according to a detection scene, user text information input by a user corresponding to detection requirements, and an original image; and establishing an open set target detection model for fusing text modalities for target samples with unknown categories to realize open set detection. The present application provides rich detection information for solving unknown targets newly appearing in a detection image, realizes more fine-grained understanding of an image, improves the accuracy of target detection in an open world, can overcome the limitation that traditional target detection methods need to limit targets, realizes unknown category detection in an open world, and better adapts to real world scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an open set target detection method based on fusion text coding. BACKGROUND

[0002] With the continuous improvement of the intelligent degree of industry, the industrial production process has a large demand for target detection technology. A large amount of target detection technology is used in product defect detection, production workshop target positioning and tracking, and target detection technology is playing an increasingly important role in the industrial field. The research object of target detection is mainly two-dimensional images, and the predicted bounding box of the target to be detected in the image is given, and the classification is correct, so as to represent the detected object.

[0003] The traditional target detection model mainly focuses on the recognition of pre-defined known categories, which limits the usability in real-world applications. In actual industrial scenarios, some unknown, unlabeled defects, abnormalities or new products are often encountered. The emergence of these unknown categories poses a challenge to traditional closed set detection. Open set detection technology, as a technology that can identify unknown category samples, provides a new way to solve this problem. Unlike traditional closed set detection, open set detection not only accurately classifies known category targets, but also provides as much information as possible about samples that do not belong to any known category. Some existing methods also attempt to detect unknown category targets, but can only provide simple information, such as classifying them into unknown categories, so there is still a problem of not being able to identify new unknown targets. SUMMARY

[0004] The purpose of the present application is to address the performance deficiencies of the prior art, and to provide an open set target detection method based on fusion text information, which provides rich detection information for target detection to solve the detection of new unknown targets in the detection image, realizes a more fine-grained understanding of the image, and improves the accuracy of target detection in an open world.

[0005] To solve the above technical problems, the present application provides the following technical scheme: an open set target detection method based on fusion text coding, the method comprising the following steps:

[0006] S1, obtaining text template information preset according to a detection scene, user text information input by a user according to detection requirements, and an original image;

[0007] S2, establishing an open set target detection model for fusing text modalities for target samples with unknown categories to realize open set detection;

[0008] S3. Based on the open set target detection model, feature extraction is performed on text template information, user text information, and original image to obtain original text features and multi-scale image features.

[0009] S4. Interact and enhance the original text features and multi-scale image features to obtain multimodal query features with text awareness capabilities and updated text features.

[0010] S5. Decode the multimodal query features to obtain the target region in the original image, including the coordinate information and category information of the target sample;

[0011] S6. Establish a contrastive focus classification loss for matching target regions with updated text features and assigning labels. ;

[0012] S7. Contrast Focus-Based Classification Loss Establish an overall loss function for updating the open-set object detection model. And set hyperparameters to obtain an open-set object detection model for fusing text modalities.

[0013] Furthermore, the open set target detection model includes a text feature extractor for extracting features from text template information and user text information to obtain original text features;

[0014] It also includes a multi-scale image feature extractor that extracts features from the original image to obtain multi-scale features, and a text perception module that interactively outputs target sample coordinate information and category information based on the original text features and multi-scale image features.

[0015] Furthermore, the text perception module includes a text perception encoder and a text perception decoder with text feature perception capabilities. The text perception encoder is used to realize the interaction between multi-scale image features and original text features, and output multimodal query features with text feature perception capabilities. The text perception decoder is used to decode and output the coordinate information and category information of the target sample based on the input multimodal query features.

[0016] Furthermore, the method for constructing the text-aware encoder and text-aware decoder with text feature awareness capabilities includes:

[0017] The language-aware query selection module selects object features by evaluating the similarity between multi-scale image features and original text features. Specifically:

[0018] Calculate multi-scale image features and original text features The similarity is then used to select the most relevant proposal embedding. and target embedding the most relevant proposal embedding for initializing the reference anchor, the selected target embedding for the subsequent language-aware fusion, the expression is:

[0019] ;

[0020] wherein, denotes the kronecker product; denotes the first target under the selection order; denotes the transpose;

[0021] the language-aware query fusion module, which fuses the language features and object features while preserving the original semantics of the content query, finally outputs the multi-modal query features with text-aware capability, the expression is:

[0022] ;

[0023] wherein, denotes the number of layers of the language-aware query fusion module; denotes the output of each layer in the language-aware query module; denotes the query, key and value of the attention layer respectively; denotes the attention layer, which can be divided into self-attention layer or cross-attention layer according to the input; denotes the feedforward layer, which consists of two perception machines and performs nonlinear transformation operation; is an activation function using a gate mechanism.

[0024] Further, the expression of the contrastive focus classification loss is:

[0025] ;

[0026] wherein, is the probability that the target region output by the open set target detection model belongs to the corresponding text feature; is an adjustment factor.

[0027] Further, the overall loss function contains loss and loss, that is:

[0028] ;

[0029] wherein, Y is the real label data; X is the coordinate information output by the text-aware decoder; is the intersection over union of the real box and the predicted box; an area of a minimum closed set of the real box and the predicted box; an area of an intersection of the real box and the predicted box;

[0030] the overall loss function The expression is:

[0031] ;

[0032] In the formula, respectively represent the weight factor of the corresponding sub-loss function.

[0033] Further, the training of the open set target detection model and the setting of the hyperparameters are as follows:

[0034] The adaptive momentum estimation optimizer with weight decay is The total batch size is 128, the base learning rate of the text feature extractor, the multi-scale image feature extractor and the text-aware decoder is The learning rate of the text-aware encoder is 0.1 times the base learning rate, and the training is performed for 24 rounds, using a step learning rate plan, in which the learning rate is reduced to 0.1 and 0.01 times the base learning rate at the 16th and 22nd epochs, respectively.

[0035] By the above technical solutions, the present application provides an open set target detection method based on fusion text coding, which has at least the following beneficial effects:

[0036] 1. The present application can overcome the limitations of traditional target detection methods that require limited targets, realize open world unknown class detection, and better adapt to real world scenarios by combining language modalities. Thus, it provides rich detection information for solving new unknown targets in the detection image in target detection, realizes a more fine-grained understanding of the image, and improves the accuracy of target detection in the open world.

[0037] 2. The present application aims to improve the performance of image open set detection, provides more rich information for open set detection by fusing text modalities, divides each region into a new class in the perception semantic space, realizes the alignment of image targets and text information, and further improves the performance of open set target detection, providing more rich detection information for industrial scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings, which are included to provide a further understanding of the application, constitute a part of this application and illustrate certain illustrative embodiments of the application and its description, which are used to explain the application, and do not constitute an improper limitation on the application. In the drawings:

[0039] Figure 1 is a flowchart of the open set target detection method in the present application;

[0040] Figure 2 The network structure diagram of the open set target detection model in the application;

[0041] Figure 3 The network structure diagram of the text perception module in the application. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned purposes, features and advantages of the application more obvious and easy to understand, the application will be further described in detail below in combination with the drawings and specific embodiments. The implementation process of how to apply technical means to solve technical problems and achieve technical effects can be fully understood and implemented by the application.

[0043] The embodiment utilizes the current pre-trained text feature extractor to obtain the original text features in the text template information and the user text information, which can inject rich semantic information into the open set target detection model; and utilizes the existing pre-trained image feature extractor to obtain the multi-scale image features of the original image.

[0044] On this basis, a text perception encoder with text perception capability is constructed to realize the interaction and feature updating of the original text features and the multi-scale image features, output the multi-modal query features with text perception capability and the updated text features, and then send them together into the text perception decoder with text perception capability to decode and obtain the open set detection target, so as to obtain the target region in the original image; by establishing a contrast focus classification loss The matching and alignment of the target region and the text features are performed to provide more fine-grained label information for the detection target, and finally the open set target detection model is updated together with other losses to complete the training and obtain the final open set target detection model fused with the text mode.

[0045] Please refer to Figures 1-3 The embodiment proposes an open set target detection method based on fusion text coding, which can overcome the limitation of traditional target detection methods that need to limit the target, realize open world unknown class detection, and better adapt to real world scenes. As shown in Figure 1 The implementation process of the method includes the following steps:

[0046] S1, obtaining the text template information preset according to the detection scene, the user text information input by the user according to the detection requirement, and the original image. The preset text template information is a further supplement and enrichment of the semantics of the deployment scene, which is generally the annotation of the scene picture and can be obtained from the predefined knowledge base. The user text information is the text information input by the user according to the detection requirement, or the further supplemented text information made by the user.

[0047] S2, an open-set target detection model for fusing text modalities to realize open-set detection for target samples with unknown categories is established. As shown in Figure 2 The open-set target detection model includes a text feature extractor for extracting features of text template information and user text information to obtain original text features, and a multi-scale image feature extractor for extracting features of original images to obtain multi-scale features, and a text perception module for outputting target sample coordinate information and category information according to the original text features and the multi-scale image features. In this embodiment, the text feature extractor uses BERT or other types of text feature extractors. The multi-scale image feature extractor uses ResNet or other types of image multi-scale feature extractors.

[0048] The text perception module includes a text perception encoder with text feature perception capability and a text perception decoder. The text perception encoder is used to realize the interaction of multi-scale image features and original text features, and output multi-modal query features with text feature perception capability. The text perception decoder is used to decode and output the coordinate information and category information of the target sample according to the input multi-modal query features.

[0049] As shown in Figure 3 The construction method of the text perception encoder with text feature perception capability and the text perception decoder includes:

[0050] ;

[0051] The similarity of the multi-scale image features and the original text features is calculated, and the most relevant proposal embedding and the target embedding are selected. The most relevant proposal embedding is used to initialize the reference anchor point, and the selected target embedding is used for subsequent language perception fusion. denotes the kronecker product; denotes the first target under the selection order.

[0052] The language perception query fusion module fuses language features and object features while preserving the original semantics of the content query. This module is also an important part of the text perception decoder layer, which contains self-attention, cross-attention operation sub-layers, etc. Finally, it outputs multi-modal query features with text perception capability, and the expression is:

[0053] ;

[0054] wherein, denotes the number of layers of the language perception query fusion module. represents the output of each layer in the language-aware query module; respectively represent the query, key, and value of the attention layer; represents the attention layer, which can be divided into self-attention layer or cross-attention layer according to different inputs, and there are three kinds of inputs corresponding to query, key, and value, respectively. Each attention layer is set according to its position. q is usually the input of the previous layer k and v are usually the input of the previous layer multi-scale image features and target embedding ; represents that the feedforward layer is composed of two perception machines, and performs nonlinear transformation operation; is an activation function using a gate mechanism, and the specific calculation formula is as follows:

[0055] ;

[0056] wherein, represents the input; is a model parameter to be learned; and represent the activation function operation.

[0057] The embodiment constructs a text-aware encoder with text-aware capability, can realize the interaction and update of multi-scale image features and original text features, and obtain multi-modal query features. The text-aware decoder is used to obtain the positioning and classification of the target in the image, and the open set detection capability of the model is improved.

[0058] S3, based on the open set target detection model, the text template information and the user text information, and the original image are respectively extracted to obtain the original text features and the multi-scale image features. In this embodiment, the original text features and the multi-scale image features are extracted by the text feature extractor and the multi-scale image feature extractor in the open set target detection model. There are many types of models supporting text feature extraction, and BERT type model is preferred to provide good original text features. There are many models supporting image multi-scale feature extraction, such as ResNet native type and related variant models, or Swin-transformer type models, which can all provide good multi-scale image features.

[0059] In this embodiment, the text feature extractor and the multi-scale image feature extractor are used to extract image and text features respectively, which provides a good feature basis for the open set target detection model, and injects rich semantic information, so that the application has good feature extraction function and ensures the open set detection accuracy of the model.

[0060] S4, interact and feature enhancement are performed on the original text features and the multi-scale image features to obtain multi-modal query features with text perception capability and updated text features. In this embodiment, a text perception module with text feature perception capability is constructed, which includes a text perception encoder with text feature perception capability and a text perception decoder. By inputting the multi-scale image features and the original text features into the text perception encoder, the interaction and feature enhancement between the multi-scale image features and the original text features are realized, and multi-modal query features with text perception capability and updated text features are output.

[0061] S5, decoding the multi-modal query features to obtain the target region in the original image, including the coordinate information and the category information of the target sample; by inputting the multi-modal query features with text perception capability into the text perception decoder, the coordinate information and the category information of the target sample are decoded and output.

[0062] S6, a contrast focal point classification loss is established for matching the target region with the updated text features and assigning labels ; in this embodiment, the expression of the contrast focal point classification loss is:

[0063] ;

[0064] In the formula, is the probability that the target region output by the open set target detection model belongs to the corresponding text feature; is an adjustment factor for dynamically reducing the weight of easily distinguishable samples in the training process and focusing on difficult samples.

[0065] S7, an overall loss function is established for updating the open set target detection model based on the contrast focal point classification loss , and a hyperparameter is set to obtain an open set target detection model for fusing the text modal. In this embodiment, the gradient calculation and updating are performed by using the overall loss function , and the training of the open set target detection model is completed, wherein the overall loss function includes loss and loss.

[0066] Specifically, according to the coordinate information of the target sample output by the text perception decoder and the real coordinate, the loss and the loss are calculated, that is:

[0067] ;

[0068] In the formula, Y is the real label data; X is the coordinate information output by the text perception decoder. is the intersection over union of the ground truth box and the predicted box; is the minimum closed area of the ground truth box and the predicted box; is the area of the ground truth box and the predicted box after performing the AND operation;

[0069] Therefore, the expression of the overall loss function is as follows:

[0070]

[0071] In the formula, w1, w2, and w3 respectively represent the weight factors of the corresponding sub-loss functions.

[0072] The embodiment establishes multiple losses for training, specifically including the loss of the localization loss, the loss of the classification loss, and the contrastive focus classification loss of the image-text alignment to optimize the open-set object detection model, so as to enhance the text perception ability of the open-set object detection model, and enrich the text semantics, so that the present application can provide more abundant target information for unknown class target samples in the real world, thereby breaking the predefined closed set and realizing open-set detection.

[0073] The training of the open-set object detection model and the related hyperparameter settings are as follows:

[0074] In order to maintain the simplicity of the open-set object detection model, the adaptive momentum estimator optimizer with the weight decay of 0.05 is adopted in the embodiment, the total batch size is 128, the base learning rate of the text feature extractor, the multi-scale image feature extractor, and the text perception decoder is 0.01, the learning rate of the text perception encoder is 0.001 times the base learning rate, which is specifically set to 0.001 in the embodiment, 24 rounds of training are performed, and a step learning rate plan is used, in which the learning rate is reduced to 0.001 and 0.0001 times the base learning rate at the 16th and 22nd epochs respectively. After the training of the open-set object detection model is completed, the trained model weight is saved and loaded, and then the image is input to obtain the open-set detection result.

[0075] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiment methods can be completed by programs instructing related hardware, therefore, the present application can adopt a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) containing computer usable program codes. ​​​​​

[0076] The above embodiments have been described in detail, and the principles and embodiments of the present application are described by using specific examples. The above examples are only used to help understand the method of the present application and its core idea. Meanwhile, for those skilled in the art, according to the idea of the present application, the specific embodiments and application scope will be changed, and the above description should not be understood as a limitation of the present application.

Claims

1. An open set object detection method based on fusion text encoding, characterized in that, The method comprises the following steps: S1, obtaining preset text template information according to a detection scene, user text information input by a user according to detection requirements, and an original image; S2, establishing an open-set target detection model for fusing text modalities for a target sample with an unknown category to realize open-set detection; S3, performing feature extraction on the text template information and the user text information, and the original image based on the open-set target detection model to obtain original text features and multi-scale image features; S4, interacting and enhancing the original text features and the multi-scale image features to obtain multi-modal query features with text perception ability and updated text features; S5, decoding the multi-modal query features to obtain a target region in the original image, including coordinate information and category information of the target sample; S6, a contrastive focal classification loss is established for matching the target region with the updated text features and assigning a label , the expression is: ； In the formula, is an open set target detection model outputting a probability that a target region belongs to a corresponding text feature; is an adjustment factor; S7、based on the contrast focus classification loss establishing an overall loss function for updating the open set object detection model , and setting the hyperparameters to obtain the open set object detection model for fusing the text modal.

2. The open set target detection method of claim 1, wherein, The open-set target detection model comprises a text feature extractor for extracting features of the text template information and the user text information to obtain the original text features; a multi-scale image feature extractor for extracting features of the original image to obtain multi-scale features, and a text perception module for interacting the original text features and the multi-scale image features to output the coordinate information and the category information of the target sample.

3. The open set target detection method of claim 2, wherein, The text perception module comprises a text perception encoder with text feature perception ability and a text perception decoder, the text perception encoder is used to realize interaction of the multi-scale image features and the original text features, and output multi-modal query features with text feature perception ability, and the text perception decoder is used to decode and output the coordinate information and the category information of the target sample according to the input multi-modal query features.

4. The open set target detection method of claim 3, wherein, The construction method of the text perception encoder with text feature perception ability and the text perception decoder comprises: A language perception query selection module selects object features by evaluating the similarity between the multi-scale image features and the original text features, specifically: Computing multiscale image features and original text features a similarity score between the most relevant proposal embedding and the target embedding the most relevant proposal embedding for initializing the reference anchor, the selected target embedding for subsequent language-aware fusion, expressed as: ； wherein denotes the kronecker product; denotes the first target under selection order; denotes the transpose; A language perception query fusion module fuses the language features and the object features while retaining the original semantics of the content query, and finally outputs multi-modal query features with text perception ability, and the expression is: ； wherein, represents the layer number of the language-aware query fusion module; represents the output of each layer in the language-aware query module; respectively represent the query, key, and value of the attention layer; represents the attention layer, which can be divided into self-attention layer or cross-attention layer according to the input; represents that the feedforward layer is composed of two perception machines, and performs a nonlinear transformation operation; is an activation function using a gate mechanism.

5. The open set target detection method of claim 1, wherein, The overall loss function comprising losses and losses, i.e.: ； In the formula, Y is real label data; X is coordinate information output by a text perception decoder; is an intersection over union of a real box and a predicted box; is a minimum closed area of a real box and a predicted box; is an area obtained by performing an and operation on a real box and a predicted box; Overall loss function The expression for the overall loss function is: ； In the formula, respectively represent the weight factors of the corresponding sub-loss functions.

6. The open set target detection method of claim 1, wherein, The training of the open-set target detection model and the setting of the hyperparameters are as follows: Using weight decay The adaptive momentum estimation optimizer has a total batch size of 128, and the base learning rate for the text feature extractor, multi-scale image feature extractor, and text-aware decoder is... The text-aware encoder was trained for 24 epochs with a learning rate of 0.1 times the base learning rate, using a step learning rate plan where the learning rate was reduced to 0.1 times the base learning rate in the 16th and 22nd epochs, respectively.

Images