Open set target detection model training method, target detection method, equipment and medium

Abstract

The invention discloses a training method of an open set target detection model, a target detection method, equipment and a medium. The method comprises the following steps: acquiring decoder query of an input image and a target boundary prediction result; obtaining text embedding characteristics of the text information and an alignment score matrix between the text information and decoder query; determining a vision-language prompt according to the decoder query, and generating a target name prediction result based on the vision-language prompt; and constructing a joint loss item, a mask alignment loss item and a distillation loss item, and training based on the joint loss item, the mask alignment loss item and the distillation loss item. According to the invention, while the dependence on a large-scale data set is reduced, the training convergence is accelerated, the training efficiency is improved, and the detection performance is improved, so that the model performance of the open-set target detection model in pedestrian detection analysis and detection of any category of targets is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of pedestrian analysis, and in particular to a training method of an open-set target detection model, a target detection method, equipment and a medium. BACKGROUND

[0002] Pedestrian analysis is an important research direction in the field of computer vision, aiming to detect, identify and analyze the behavior of pedestrians in images or videos through image processing and pattern recognition technology. The core tasks of pedestrian analysis include pedestrian detection and identification, and are widely used in intelligent video monitoring, security, traffic management and other fields.

[0003] Open vocabulary object detection (OVD) is an important basic vision task in computer vision, which effectively extends the detection capability of the model from closed-set detection (COD) to open-set detection, can locate and identify new categories, and exceeds the fixed category of objects contained in the model training data, greatly expanding the application scenarios and detection capabilities of the detection model. However, OVD still relies on additional vocabulary input to obtain detection results during inference, and this dependency requires the model to rely on artificially supplemented language knowledge prior, which greatly limits the detection capability of the model in an open world scenario.

[0004] In recent years, researchers have proposed an open-ended object detection method (OED), which enables the model to directly detect targets without providing predefined object category text, and generate the corresponding names of the targets. However, although the model has the characteristics of open-ended detection, it is still difficult to effectively align the semantic information between the visual and language modalities in the high-dimensional feature space, affecting the accuracy of model training and object name generation, resulting in the model training relying on large-scale datasets and a large number of GPU resources, thereby reducing the training efficiency and affecting the detection performance.

[0005] Therefore, the prior art still needs to be improved and improved. SUMMARY

[0006] The technical problem to be solved by the present application is to provide a training method of an open-set target detection model, a target detection method, equipment and a medium to solve the problems of the prior art.

[0007] In order to solve the above technical problems, the first aspect of the present application provides a training method for an open-set object detection model, which is used to train an open-set object detection model to be trained to determine the open-set object detection model, wherein the open-set object detection model to be trained includes an object detector, a prompt distiller, an object name generator, and a vision-language aligner; the training method for the open-set object detection model specifically includes:

[0008] Obtaining a decoder query and an object boundary prediction result of an input image through an object detector, and obtaining a text embedding feature of text information corresponding to the input image and an alignment score matrix between the decoder query and the text embedding feature through a vision-language aligner;

[0009] determining, by the cue distiller, visual-linguistic cues from the decoder query, and generating, by the target name generator, target name predictions based on the visual-linguistic cues;

[0010] constructing a joint loss term based on the object boundary prediction result and the alignment score matrix, constructing a mask alignment loss term based on the alignment score matrix and the text embedding feature, and constructing a distillation loss term based on the visual-linguistic cue and the text embedding feature;

[0011] The open set object detection model to be trained is trained based on the joint loss term, the mask alignment loss term, and the distillation loss term to obtain an open set object detection model.

[0012] The training method of the open set object detection model, wherein the process of obtaining the alignment score matrix specifically includes:

[0013] Obtaining, by the vision-language aligner, a first alignment score from the decoder query to the text embedding feature and a second alignment score from the text embedding feature to the decoder query;

[0014] An alignment score matrix between the decoder query and the text embedding feature is determined based on the first alignment score and the second alignment score.

[0015] The method for training an open-set object detection model, wherein determining visual-linguistic cues based on the decoder query by the cue distiller specifically comprises:

[0016] reading, by the hint distiller, backbone features and encoder features extracted by the object detector in the process of obtaining the decoder query;

[0017] Interacting information between the decoder query and the backbone features through the prompt distiller to obtain a first query;

[0018] Interacting the decoder query with the encoder features via the hint distiller to obtain a second query;

[0019] Projecting the first query, the second query, and the decoder query to the channel dimension space where the text embedding features are located by the prompt distiller;

[0020] The projected first query, the projected second query, and the projected decoder query are fused by the prompt distiller to obtain a visual-linguistic prompt.

[0021] The training method of the open set object detection model, wherein generating the object name prediction result based on the visual-linguistic prompt by the object name generator specifically includes:

[0022] determining, by a large language model in the target name generator, a name embedding feature based on the visual-linguistic cue and a noisy text embedding feature, wherein the noisy text embedding feature is formed by adding Gaussian noise to the text embedding feature;

[0023] A target name prediction result is generated according to the name embedding feature by a low-rank prediction head in the target name generator.

[0024] The training method of the open set object detection model, wherein the step of constructing a mask alignment loss term based on the alignment score matrix and the text embedding feature specifically includes:

[0025] generating a binary mask based on the text embedding feature;

[0026] A mask alignment loss term is constructed based on the alignment score matrix and the binary mask.

[0027] The training method of the open set object detection model, wherein the step of constructing a mask alignment loss term based on the alignment score matrix and the text embedding feature specifically includes:

[0028] generating a binary mask based on the text embedding feature;

[0029] A mask alignment loss term is constructed based on the alignment score matrix and the binary mask.

[0030] The training method of the open set object detection model, wherein the constructing of the distillation loss term based on the visual-linguistic cue and the text embedding feature specifically includes:

[0031] Matching the visual-linguistic cue with the text embedding feature to obtain a matched visual-linguistic cue and a matched text embedding feature;

[0032] A distillation loss term is constructed based on the matched visual-linguistic cues and the matched text embedding features.

[0033] A second aspect of the present application provides a target detection method, using an open set target detection model trained using the above-mentioned open set target detection model training method, the target detection method specifically comprising:

[0034] Obtain the image to be detected;

[0035] The image to be detected is input into the open set target detection model, and the detection result corresponding to the image to be detected is output through the open set target detection model.

[0036] The third aspect of the present application provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the training method of the open set target detection model as described above.

[0037] A fourth aspect of the present application provides a terminal device, comprising: a processor and a memory;

[0038] The memory stores a computer-readable program executable by the processor;

[0039] When the processor executes the computer-readable program, the steps in the training method of the open set target detection model as described above are implemented.

[0040] Beneficial effect: Compared with the prior art, the present application provides a training method, target detection method, device and medium for an open-set target detection model, the method comprising obtaining a decoder query and a target boundary prediction result of an input image through a target detector; obtaining text embedding features of text information corresponding to the input image and an alignment score matrix between the decoder query and the text embedding feature through a visual-linguistic aligner; determining a visual-linguistic cue according to the decoder query through the cue distiller, and generating a target name prediction result based on the visual-linguistic cue through the target name generator; constructing a joint loss term based on the target boundary prediction result and the alignment score matrix, constructing a mask alignment loss term based on the alignment score matrix and the text embedding feature, and constructing a distillation loss term based on the visual-linguistic cue and the text embedding feature; training the open-set target detection model to be trained based on the joint loss term, the mask alignment loss term and the distillation loss term to obtain an open-set target detection model. This application uses a prompt distiller to migrate the visual-language alignment information in the visual-language aligner to the decoder query to obtain visual-language prompts, effectively reducing the multimodal information difference between vision and language, and improving the prediction accuracy of the target name prediction results, thereby improving the convergence speed and model performance of the open-set object detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0042] Figure 1 Flowchart of the training method of the open set target detection model provided in the embodiment of the present application.

[0043] Figure 2 Schematic diagram of the framework structure of the training method for the open set target detection model.

[0044] Figure 3 Schematic diagram of the framework structure of the vision-language distillation module.

[0045] Figure 4 A visualization of the joint loss term.

[0046] Figure 5 This diagram explains the causes of conflicting losses.

[0047] Figure 6 Model training performance curves for GenerateU and Open-Det.

[0048] Figure 7 A comparison chart of the test results.

[0049] Figure 8 This is a block diagram of the principles of the terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The embodiments of the present application provide a training method for an open-set object detection model, an object detection method, an apparatus, and a medium. To make the objectives, technical solutions, and effects of the present application more clear and explicit, the present application is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are intended only to explain the present application and are not intended to limit the present application.

[0051] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the specification of the present application refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or wireless couplings. The term "and / or" used herein includes all or any units and all combinations of one or more associated listed items.

[0052] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0053] It should be understood that the sequence numbers and sizes of the steps in this embodiment do not imply the order of execution. The order of execution of each process is determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of this application.

[0054] The application content will be further explained below through description of embodiments in conjunction with the accompanying drawings.

[0055] The present invention addresses the problems of current open detection frameworks, such as models relying on large-scale datasets and a large amount of GPU resources for training, low model training efficiency, slow convergence, and poor detection performance. A training method for an open-set object detection model is proposed. The method is used to train an open-set object detection model to determine the open-set object detection model, effectively improving the detection accuracy of the open-set object detection model and expanding the application flexibility of the open-set object detection model. This method is of great significance in fields such as pedestrian analysis and object detection. The open-set object detection model to be trained includes an object detector, a prompt distiller, an object name generator, and a visual-linguistic aligner. In other words, during the training phase, textual information corresponding to the input image is used as a priori linguistic knowledge, enabling the model to grasp the alignment information between vision and language. Subsequently, this visual-linguistic alignment information is transferred to the decoder's query process to accelerate the convergence of the open-set object detection model and improve model performance. During the inference phase, the open-set object detection model does not require additional object category text and directly detects objects and generates corresponding names.

[0056] like Figure 1 As shown, the training method of an open set target detection model provided in this embodiment specifically includes:

[0057] S10, obtaining a decoder query of the input image and an object boundary prediction result through the object detector.

[0058] Specifically, the object detector is used for decoder query of bounding box prediction, and performs bounding box prediction based on the decoder query to obtain the object boundary prediction result. Figure 2 As shown, the target detector may include a backbone network model, a Transformer model, and a prediction head. The backbone network model is used to extract features from the input image I to obtain backbone features B; the encoder is used to form encoder features E based on the backbone features B, and the decoder is used to form decoder queries Q based on the encoder features E. d ; The prediction head is used to query Q based on the decoder d Determine the target boundary prediction result. That is, extracting the decoder query, backbone features, and encoder features through the target detector, and determining the target boundary prediction result based on the decoder query can be specifically as follows: inputting the input image I into the backbone network model, outputting the backbone feature B through the backbone network model, inputting the backbone feature B into the encoder, and outputting the encoder feature E through the encoder; inputting the encoder feature E into the decoder, and outputting the decoder query Q through the decoder. d ; Query the decoder Q d The input prediction head outputs the target boundary prediction result through the prediction head.

[0059] Furthermore, the target boundary prediction result may include a predicted bounding box and a predicted binary score. Accordingly, the prediction head may include a binary prediction head and a bounding box prediction head. The binary prediction head is used to predict the binary score based on the decoder query, and the bounding box prediction head is used to predict the bounding box based on the decoder query. Of course, in actual applications, the target boundary prediction result may also include only the predicted bounding box, and the prediction head may include only the bounding box prediction head.

[0060] S20. Obtaining, through a vision-language aligner, text embedding features of the text information corresponding to the input image and an alignment score matrix between the decoder query and the text embedding features.

[0061] Specifically, the visual-language aligner is used to align the text embedding features of the visual-language features, and performs bidirectional alignment on the text embedding features and the decoder query to obtain an alignment score matrix between the text embedding features and the decoder query and a visual-language alignment index. The visual-language aligner may include a visual-language model (VLM) and a bidirectional visual-language alignment module, wherein the visual-language model (VLM) is used to extract the text embedding features T e The bidirectional visual-language alignment module is used to align the text embedding features with the decoder query to obtain the alignment score matrix S between the text embedding features and the decoder query. align , alignment score matrix S align It is used to reflect the similarity between the decoder query and the text embedding feature. In addition, when bidirectionally aligning the text embedding feature with the decoder query, the visual-language alignment index VL between the decoder query and the text embedding feature can be recorded. align , so as to index VL based on the visual-language alignment align Quickly determine correspondences between decoder queries and text embedding features.

[0062] Furthermore, due to Q d The channel dimension is usually smaller than T e The channel dimension, for example, Q d There are generally 256 channels, and T e Generally, there are 768 channels. However, this dimensionality difference may result in insufficient information in the visual query to match the text embedding features. d Mapping to a higher-dimensional space along the channel dimension to directly align the two features may produce suboptimal visual-linguistic alignment results. Therefore, in the embodiment of the present application, the bidirectional visual-linguistic alignment module calculates the alignment scores from the perspectives of vision to language (V-to-L) and language to vision (L-to-V) to obtain the alignment score matrix S align, enhance the alignment effect between visual and language modalities. Among them, the alignment score matrix S align It can be expressed as:

[0063] S align =cos(Q d ×M LV ,T e )+cos(Q d ,T e ×M VL ),

[0064] Among them, S align ∈R N×M represents the alignment score matrix, N represents the number of queries, M represents the number of text embeddings, Denotes decoder query, d q represents the channel dimension of the query feature, represents the text embedding feature, d t Represents the channel dimension of text embedding features, cos() represents cosine similarity, M LV represents the vision-to-language (V-to-L) transformation matrix, M VL Represents the language to vision (L-to-V) transformation matrix.

[0065] The embodiment of the present application calculates alignment scores from both vision-to-language (V-to-L) and language-to-vision (L-to-V) perspectives to obtain an alignment score matrix, thereby solving the problem in the prior art that it is still difficult to effectively align semantic information between visual and language modalities in high-dimensional feature space, effectively improving the matching accuracy of the two modal information, and thereby improving the accuracy of target name prediction.

[0066] Of course, it should be noted that in actual applications, the alignment score can be calculated only from the vision-to-language (V-to-L) perspective, or only from the language-to-vision (L-to-V) perspective, according to implementation requirements. Alternatively, after calculating the alignment scores from both the vision-to-language (V-to-L) and language-to-vision (L-to-V) perspectives, the alignment scores calculated from the two perspectives are weighted to determine the alignment score matrix.

[0067] S30 , determining visual-linguistic cues according to the decoder query by the cue distiller.

[0068] Specifically, the hint distiller is used to transfer the visual-language alignment knowledge learned by the visual-language aligner to the decoder query. The hint distiller includes a visual-language distillation module, which transfers the visual-language alignment knowledge to the decoder query Q dTo generate visual-linguistic cues. Specifically, the visual-linguistic distillation module enhances the decoder query by converting it into visual-linguistic cues that are similar to the image. These visual-linguistic cues are more easily aligned with image-level text embedding features and can effectively narrow the gap between visual queries and image-level text embeddings in the VLM, improving the text generation quality of the target name generator and accelerating its convergence.

[0069] Exemplarily, determining the visual-linguistic cue according to the decoder query by the cue distiller specifically includes:

[0070] reading, by the hint distiller, backbone features and encoder features extracted by the object detector in the process of obtaining the decoder query;

[0071] Interacting information between the decoder query and the backbone features through the prompt distiller to obtain a first query;

[0072] Interacting the decoder query with the encoder features via the hint distiller to obtain a second query;

[0073] Projecting the first query, the second query, and the decoder query to the channel dimension space where the text embedding features are located by the prompt distiller;

[0074] The projected first query, the projected second query, and the projected decoder query are fused by the prompt distiller to obtain a visual-linguistic prompt.

[0075] Specifically, the backbone features B and encoder features E are used to extract the decoder query Q d The first query is to query the decoder Q d The second query is obtained by interactive learning with the backbone feature B. d It is obtained by interactive learning with the encoder feature E. Among them, Figure 3 As shown, the prompt distiller may include a first interactive branch, a second interactive branch and a fusion unit, the first interactive branch and the second interactive branch are both connected to the fusion unit, the first interactive branch is used to convert the decoder query Q d Interactive learning with backbone feature B, the second interactive branch is used to query the decoder Q d and the encoder feature E for interactive learning, and the fusion unit is used to generate visual-linguistic cues based on the first query, the second query and the decoder query.

[0076] Further, the first interaction branch and the second interaction branch have the same structure, and each includes a deformable cross-attention module, a normalization layer, a feed-forward network (FFN), and a normalization layer; the deformable cross-attention module in the first interaction branch performs cross-attention operation on Q d and the backbone feature B to interact information, the deformable cross-attention module in the second interaction branch performs cross-attention operation on Q d and the encoder feature E to interact information, and adaptively enriches the surrounding background information of the query corresponding target, wherein the deformable cross-attention module can adaptively perform bias sampling on the query object Q d surrounding token, so as to enable it to learn relevant background information. The feed-forward network is used to weight the decoder query after interaction and the decoder query to obtain the first query Q b and the first query Q e .

[0077] The first query Q b , the second query Q e , and the decoder query Q d are projected along the channel dimension to the same channel dimension space as the text embedding feature T e , and then the projected first query, the projected second query, and the projected decoder query are fused to generate the visual-linguistic prompt P vl . As shown in Figure 3 , the process of generating the visual-linguistic prompt according to the first query Q b , the second query Q e , and the decoder query Q d can be implemented by a fusion unit. Specifically, the fusion unit can include three parallel multi-layer perceptrons (MLPs) and a linear layer. The three multi-layer perceptrons can include two linear layers and an activation layer. The three multi-layer perceptrons are respectively used to project the first query, the second query, and the decoder query along the channel dimension to the same channel dimension space as the text embedding feature T e , and the linear layer is used to fuse the projected first query, the projected second query, and the projected decoder query.

[0078] It should be noted that in actual application, the prompt distiller can also adopt other network structures, for example, only including one interaction branch, through which the decoder query and the backbone feature are sequentially interacted and learned, or the decoder query and the encoder feature are sequentially interacted and learned, or only including one multi-layer perceptron (MLP), through which the first query, the second query, and the decoder query are sequentially projected along the channel dimension to the same channel dimension space as the text embedding feature T eThe same channel dimension space; or, it only includes a deformable cross-attention module and a feedforward network, etc. There is no specific restriction here, as long as the network structure can generate visual-language cues according to the above process.

[0079] S40: Generate a target name prediction result based on the visual-linguistic prompt by the target name generator.

[0080] Specifically, the target name generator is configured to generate a target name prediction result based on the visual-linguistic cues. In other words, the target name generator generates a target name based on the visual-linguistic cues. The target name generator may include a generated large language model, and the target name prediction result is generated using the large language model.

[0081] Furthermore, since the open set object detection model is in the early stage of training, the decoder queries Q d Not yet fully trained, it is related to the text embedding feature T e There is a large semantic gap between them, which will cause the alignment of visual-linguistic alignment information (i.e., visual-linguistic cues) to be noisy (i.e., misaligned) and have low alignment quality. If the visual-linguistic alignment information is used for model training, the pre-trained weights of the large language model (especially the prediction head with a large amount of parameters responsible for text generation) may be destroyed, thereby seriously reducing the training efficiency of the open set target detection model. For this reason, in an embodiment of the present application, the pre-trained head in the large language model is frozen, and a low-rank prediction head is introduced to reduce the trainable parameters of the head to speed up the training of the large language model.

[0082] Based on this, generating a target name prediction result based on the visual-linguistic prompt by the target name generator specifically includes:

[0083] determining name embedding features based on the visual-linguistic cues and the noisy text embedding features using a large language model in the target name generator;

[0084] A target name prediction result is generated according to the name embedding feature by a low-rank prediction head in the target name generator.

[0085] Specifically, the noise text embedding feature is formed by adding Gaussian noise to the text embedding feature. That is, after obtaining the text embedding feature, Gaussian noise can be added to the text embedding feature to form a noise text embedding feature, and the noise text embedding feature is used as the input of the large language model to perform text reconstruction through the large language model to generate the name embedding feature. This can stabilize the training process and improve the model's ability to handle noise input, thereby effectively improving the robustness of the large language model. Among them, Gaussian noise obeys the normal distribution N(0,σ 2), where σ is composed of the text embedding feature T e The standard deviation is calculated.

[0086] The embodiment of the present application freezes the weights of the pre-trained LLM head and introduces a low-rank prediction head (LoRaHead), which can significantly reduce the trainable parameters of the head. This can avoid the destruction of pre-trained weights caused by noisy data in the early stages of training, and significantly improve training efficiency while maintaining model performance. For example, taking the generative language model T5 as an example, the head of the T5 model responsible for generating object names contains 24.7M parameters, that is, mapping the input dimension 768 to the output dimension 32,128 (32,128 is the vocabulary size of the T5 model). By replacing the original head with a low-rank prediction head, the number of trainable parameters is reduced to 0.526M, which is only 2.12% of the original number of parameters.

[0087] S50. Constructing a joint loss term based on the target boundary prediction result and the alignment score matrix, constructing a mask alignment loss term based on the alignment score matrix and the text embedding feature, and constructing a distillation loss term based on the visual-linguistic cue and the text embedding feature.

[0088] Specifically, the joint loss term is used to improve the consistency of binary classification predictions, the mask alignment loss term is used to enhance the alignment between query and text embeddings, and the distillation loss term is used to achieve effective knowledge distillation.

[0089] In one embodiment, since the open set target detection model automatically detects targets and generates target names during the inference phase, it is impossible to calculate the similarity between the detected query and the text embedding features of the text information, and it is impossible to obtain the visual-language alignment score, thereby failing to match and select the results of a large number of detection boxes. To this end, the target detector of the embodiment of the present application includes a binary prediction head, which predicts a binary score based on the decoder query to perform binary classification (positive or negative) on the decoder query, identify and filter out duplicate detections.

[0090] Based on this, constructing a joint loss term based on the target boundary prediction result and the alignment score matrix specifically includes:

[0091] Obtaining the IoU score between the predicted bounding box and the true bounding box in the target boundary prediction result;

[0092] A joint loss term is constructed based on the IoU score, the predicted binary score in the object boundary prediction result, and the alignment score matrix.

[0093] Specifically, the binary label of each query is dynamically assigned based on the visual-linguistic alignment and matching process, and the matched query is labeled as a positive sample, while the unmatched query is labeled as a negative sample. Dynamic label assignment increases the difficulty of the binary classification prediction of the binary prediction head. When matching the decoder query and the text embedding feature, the IoU score and the alignment score between the predicted bounding box and the real bounding box are evaluated, and the query with a higher IoU score and alignment score is more likely to match the real bounding box, while the query with a lower IoU score and alignment score is more likely to remain unmatched. Embodiments of the present application combine the predicted binary score with the IoU score and the alignment score to construct a joint loss term to improve the consistency of binary classification prediction.

[0094] In one implementation, the joint loss term can be represented as:

[0095]

[0096] wherein L JL represents the joint loss term, y i ∈{0,1} represents the binary label, p i represents the predicted binary score in the target boundary prediction result, s i represents the alignment score between the decoder query and the text embedding feature, u i represents the IoU score of the predicted bounding box in the target name prediction result and the real bounding box, N represents the number of queries, and a (default value is 0.25) represents a weight adjustment factor. The square root operation is mainly used to prevent the product of the three scores from being too small, thereby helping to maintain the numerical stability of model training.

[0097] To further illustrate the effectiveness of the joint loss term, the present application visualizes the positive weight function, the negative sample weight function, and the loss function of the joint loss term with respect to the numerical changes of the predicted binary score p i , the alignment score s i , and the IoU score u i . In the three-dimensional visualization process, it is assumed that the IoU score u i and the alignment score s i in the joint loss term have the same value, and the IoU score u i and the alignment score s i are denoted as IAScore to simplify the ternary function problem (p i , u i , s i ) into a binary function problem, thereby facilitating three-dimensional visualization, obtaining a three-dimensional visualization diagram as shown in Figure 4 . Figure 4 It can be seen from

[0098] (1) Positive weight: When IAScore and predicted binary score p i When the IAScore and the predicted binary score p are consistent, it indicates that the binary prediction head accurately predicts the binary classification results and matches well with the localization accuracy and semantic alignment. In this case, the positive weight is relatively small, which helps the training stability. This consistency enables the model to quickly adjust the gradient and converge when the matching state changes. On the contrary, when the IAScore and the predicted binary score p are consistent, it indicates that the binary prediction head accurately predicts the binary classification results and matches well with the localization accuracy and semantic alignment. In this case, the positive weight is relatively small, which helps the training stability. i Inconsistent, which means that the predicted binary score p i This inconsistency will significantly increase the number of positive samples, thereby effectively increasing the loss weight of positive samples.

[0099] (2) Negative weight: During model training, positive samples are usually scarce, while negative samples are very abundant, which leads to a serious imbalance between positive and negative samples. This imbalance makes it difficult for the model to effectively detect rare positive samples. In addition, the larger the negative sample weight, the easier it is to drive the predicted binary score p output by the binary prediction head. i Approaching 0 may further reduce the number of positive samples detected by the model. To solve this problem, the embodiment of the present application sets the negative sample weight in the joint loss term. Specifically, when the predicted binary score p i When it is consistent with IAScore, it indicates that the predicted binary score output by the binary prediction head is relatively accurate, regardless of whether the current dynamic matching is correct. Specifically, if the predicted binary score p i If the value of IAScore is high or low, the probability prediction is relatively accurate; if the IAScore is high but the predicted binary score p is i If it is low, it may indicate that the matching and labeling are incorrect, and the negative weight should be small to avoid introducing wrong supervision signals. In both cases, smaller negative weights help alleviate the training instability caused by label changes in dynamic matching and maintain the model's ability to detect positive samples; if and only if IAScore is small and the predicted binary score p is i When the value is high, the negative weight increases significantly. This indicates that the current matches and labels are correct, but the output of the binary prediction head is wrong. In this case, the negative weight increases to reduce the predicted probability scores of negative samples, thereby improving the model's predictions.

[0100] (3) Joint loss term: When IAScore and predicted binary score p i Consistent, indicating that the predicted binary score p i The prediction is accurate and the loss is small; 2) When IAScore and predicted binary score p i Inconsistency indicates that the predicted binary score p i The prediction is inaccurate and the loss value is large.

[0101] In summary, the joint loss term effectively combines the alignment score, IoU score, and predicted binary score, alleviating the incorrect supervision and training instability caused by dynamic matching, thereby significantly improving model performance.

[0102] Of course, in practical applications, other methods can be used to construct the joint order according to actual needs. For example, the joint loss can be directly constructed based on the difference between the predicted bounding box and the true bounding box.

[0103] In one embodiment, the mask alignment loss is used to calculate the visual-language alignment loss. Currently, the BCE loss is commonly used for visual-language alignment. The BCE loss strengthens the alignment between query and text embeddings by increasing the similarity between matching query-text pairs and imposing a negative constraint on the query and all unmatched text embeddings in a mini-batch to increase dissimilarity. However, the BCE loss fails to consider the relationships between object names within an image or within the same mini-batch, resulting in conflicting losses and gradients, ultimately reducing training efficiency.

[0104] Specifically, for each query, the matching text embedding is regarded as a positive sample, while all other text embeddings in the same mini-batch are regarded as negative samples, thus constructing a contrastive learning framework: the model maximizes the similarity between the query and its matching text embedding while minimizing the similarity with the unmatched text embedding. However, this visual-language alignment loss introduces potential conflicts in the optimization process, resulting in contradictory losses. For example, Figure 5 As shown in the figure, an image contains multiple instances of the same category. Therefore, when calculating the loss, each query is matched with only one text embedding, while all other text embeddings are considered unmatched. Therefore, the query calculates a positive loss with the matching text embedding and a negative loss with the multiple unmatched text embeddings. However, the text embedding features of the same text are identical. Therefore, this loss generates both positive and negative losses for the same text embedding, a contradictory loss. Furthermore, during model training, backpropagation can generate contradictory gradient information.

[0105] Based on this, the mask alignment loss term constructed based on the alignment score matrix and the text embedding feature specifically includes:

[0106] generating a binary mask based on the text embedding feature;

[0107] A mask alignment loss term is constructed based on the alignment score matrix and the binary mask.

[0108] Specifically, the binary mask is a text embedding feature generator, which is used to calculate the similarity between all paired text embeddings to perform label correction on the matching of the same text. The binary mask can be expressed as:

[0109]

[0110] Where M∈R M×M represents a binary mask, M represents the number of text embeddings in the text embedding feature, Represents the transpose of text embedding features.

[0111] Accordingly, the mask alignment loss term can be expressed as:

[0112]

[0113] Where τ represents the mask passing threshold, which is used to binarize the similarity score, assigning a value of 1 to text embeddings of the same category and 0 to others; N represents the number of queries; VL align ∈R N×M represents the alignment index between the decoder query and the text embedding feature (in one-hot form); S align ∈R N×M Represents the alignment score matrix.

[0114] The embodiment of the present application performs label correction on the matching of the same text by combining binary masks, which can effectively remove the contradictory losses and contradictory gradients generated by the model during the training process, and improve the efficiency and accuracy of model training.

[0115] In one embodiment, constructing a distillation loss term based on the visual-linguistic cue and the text embedding feature specifically includes:

[0116] Matching the visual-linguistic cue with the text embedding feature to obtain a matched visual-linguistic cue and a matched text embedding feature;

[0117] A distillation loss term is constructed based on the matched visual-linguistic cues and the matched text embedding features.

[0118] Specifically, when matching the visual-linguistic hint and the text embedding feature, the visual-linguistic alignment index VL align The query features in the decoder query are matched with the text embeddings in the text embedding features to obtain matched visual-linguistic cues and matched text embeddings. A distillation loss term is then constructed using cosine similarity loss. This distillation loss term transfers the visual-linguistic alignment information contained in the visual-linguistic model in the visual-linguistic aligner to the decoder query, achieving distillation from the visual-linguistic model to the visual-linguistic cues.

[0119] S60: Training the open set object detection model to be trained based on the joint loss term, the mask alignment loss term, and the distillation loss term to obtain an open set object detection model.

[0120] Specifically, after obtaining the joint loss term, mask alignment loss term, and distillation loss term, the joint loss term, mask alignment loss term, and distillation loss term can be weighted to obtain a total loss term, which is then used to train the open-set object detection model to be trained to obtain a trained open-set object detection model. Of course, in the process of obtaining the trained intermediate open-set object detection model, multiple rounds of iterative training can be performed. Only one round of training is described here, and the number of training iterations can be set according to actual needs and is not specifically limited here.

[0121] Furthermore, after obtaining the open-set object detection model, the visual-linguistic aligner does not participate in the reasoning process when no textual information is provided (such as in object detection scenarios). However, when textual information is provided (such as in specified object recognition scenarios), the visual-linguistic aligner participates in the reasoning process. Of course, in practical applications, when the open-set object detection model is applied to scenarios that do not require additional textual auxiliary information, the visual-linguistic aligner in the open-set object detection model can be omitted to simplify the model structure of the open-set object detection model.

[0122] In summary, this embodiment provides a training method for an open-set target detection model. By adding a visual-language aligner to the open-set target detection model to be trained, the method can effectively improve the matching accuracy of the two modal information; and the visual-language alignment information contained in the visual-language model in the visual-language aligner is migrated to the decoder query, thereby realizing distillation from the visual-language model to the visual-language prompt, effectively reducing the multimodal information difference of the visual-language, improving the text generation quality of the target name generator, and accelerating the convergence speed of the target name generator. At the same time, the method effectively accelerates the convergence speed and model accuracy of the open-set target detection model by introducing a low-rank prediction head and a noisy text embedding feature in the target name generator. In addition, a mask alignment loss term and a joint loss term are introduced during the training process. The mask alignment loss term effectively removes the contradictory loss and contradictory gradient generated by the model during the training process, thereby improving the model training efficiency and accuracy; the joint loss term makes the classification of positive and negative samples of the query more accurate, thereby improving the model accuracy.

[0123] To demonstrate the effectiveness of the present invention, the model was trained on the small-scale Visual Genome (VG) detection dataset. This dataset contains 77,398 training images. Following the common experimental setup for the current Open Object Detection (OED) task, the model was evaluated for zero-shot performance on the commonly used LVIS MiniVal dataset, which contains 1,203 categories.

[0124] Model detection accuracy evaluation index (Evaluation Criteria): Following the current evaluation method of the open object detection (OED) task, the embodiment of this application uses a fixed pre-trained VLM model's text encoder to calculate the similarity score between the generated object name and the labeled category name (Ground Truth). Specific evaluation indicators include: Average Precision (AP) of rare categories r ), the average precision (AP) of common categories c ), Average Precision (AP) of frequent categories f ) and the average precision (AP) of all categories.

[0125] Experimental results: Compared with OVD models (such as GLIP), the open-set target detection model Open-Det provided by the present application demonstrates significant advantages by eliminating the reliance on additional text vocabulary priors in the reasoning stage and achieving higher model efficiency. As shown in Table 1, Open-Det achieves better performance with less training data. When trained with only 0.077M training data (i.e., 11.7% of the data used by the GLIP(A) model), Open-Det outperforms GLIP(A) by +6.8% and +8.5%. Even compared with the GLIP(C) model trained with a larger dataset (5.456M), Open-Det still achieves a +1.0% accuracy improvement using only 1.4% training data.

[0126] Compared with the OED framework model GenerateU, Open-Det demonstrates faster convergence speed and higher performance. Specifically, Open-Det achieves significant performance improvements with only 20.8% of the training times of GenerateU, an improvement of +3.6% and an improvement of +1.6%. In addition, even if GenerateU uses additional grounding data from GRIT5M for model training, Open-Det still outperforms it by +1.0% in terms of indicators when using only 1.5% of its training data. The experimental results show that Open-Det has significant model accuracy and efficiency advantages in the same task. When experiments were conducted on larger backbone models, such as Swin-Smal l and Swin-Large, the advantages of the Open-Det framework were even more significant. For example, the Open-DetSwin-Small model surpassed the GenerateU-Swin-Large model by +3.4% AP in accuracy using only 1.5% of the data. r When using the same backbone model Swin-Large, the Open-Det model significantly outperforms GenerateU by +8.9% AP. r As well as +5.2% AP, it fully verifies the effectiveness of Open-Det provided by this application.

[0127] Table 1 Comparison results of zero-shot domain transfer on the LVIS MiniVal dataset

[0128]

[0129] Figure 6 Performance curves of GenerateU and Open-Det trained on the VG dataset and evaluated for zero-shot domain transfer on the LVIS MiniVal dataset are presented. Experimental results demonstrate that Open-Det significantly outperforms GenerateU in training efficiency and convergence speed. Specifically, Open-Det achieves higher performance in fewer training epochs. These results further validate the efficiency and superiority of Open-Det in the open object detection task.

[0130] As an efficient open object detection (OED) framework, Open-Det can detect potential objects in any image without additional vocabulary priors (vocabulary-free), and has significant advantages and high practicality and applicability in real-world scenarios. Figure 7The visualization shows the detection results of Open-Det on the LVIS MiniVal dataset. Compared with the results of manual annotation (Ground Truth) and the detection results of GenerateU, the model of the present invention demonstrates the ability to detect a wider range of potential targets (as shown by the yellow arrows), covering large-scale targets, small targets and fine-grained targets. For example, in Figure 7 In (a), Open-Det successfully detects the cabinet, carpet, lamp, radiator, and fireplace; Figure 7 In (b), Open-Det detects sidewalks, shadows, streets, and small signs; in Figure 7 In (c), Open-Det recognizes the sky, cable car, head, and leaves, while the GenerateU model fails to detect these objects.

[0131] These results demonstrate that Open-Det has significant advantages in detecting diverse and fine-grained objects, further validating its efficiency and robustness in open object detection tasks. These characteristics give it a wide range of potential in real-world applications, such as pedestrian analysis, autonomous driving, intelligent surveillance, and medical image analysis.

[0132] Based on the above-mentioned training method for the open set target detection model, this embodiment provides a target detection method, which uses the open set target detection model trained using the above-mentioned training method for the open set target detection model. The target detection method specifically includes:

[0133] Obtain the image to be detected;

[0134] The image to be detected is input into the open set target detection model, and the detection result corresponding to the image to be detected is output through the open set target detection model.

[0135] Specifically, after obtaining the image to be detected, the image to be detected can be directly input into the open set target detection model, and the open set target detection model outputs the detection result corresponding to the detection image, wherein the detection result may include the target detection box and the target name. In addition, in practical applications, the visual-language aligner in the open set target detection model can determine whether to participate in the reasoning process according to the detection task type. That is, in the reasoning process of the open set target detection model, the visual-language aligner can determine whether to participate in the reasoning process according to the detection task requirements. For example, when the detection task type is the all-target detection task, the visual-language aligner does not participate in the reasoning process; when the detection task type is the specified target detection task (such as detecting a specified target in an image), the visual-language aligner participates in the reasoning process, that is, the input information input into the open set target detection model includes the image to be detected and the specified target information.

[0136] Based on the above-mentioned open set target detection model training method and target detection method, this embodiment provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the open set target detection model training method as described in the above-mentioned embodiment, and / or to implement the steps in the target detection method as described above.

[0137] Based on the above training method of the open set target detection model, the present application also provides a terminal device, such as Figure 8 As shown, it includes at least one processor 20; a display screen 21; and a memory 22. It may also include a communications interface 23 and a bus 24. The processor 20, display screen 21, memory 22, and communications interface 23 can communicate with each other via bus 24. The display screen 21 is configured to display a preset user guidance interface in the initial setup mode. The communications interface 23 can transmit information. The processor 20 can call the logic instructions in the memory 22 to execute the method in the above embodiment.

[0138] In addition, the logic instructions in the memory 22 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0139] The memory 22, as a computer-readable storage medium, can be configured to store software programs or computer-executable programs, such as program instructions or modules corresponding to the methods in the embodiments of the present disclosure. The processor 20 executes the software programs, instructions, or modules stored in the memory 22 to perform functional applications and data processing, thereby implementing the methods in the above embodiments.

[0140] The memory 22 may include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the terminal device. In addition, the memory 22 may include high-speed random access memory and non-volatile memory. For example, various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, may also be transient storage media.

[0141] In addition, the specific process of loading and executing the multiple instructions in the storage medium and the processor in the terminal device has been described in detail in the above method and will not be described here one by one.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A training method for an open-set object detection model, characterized in that: for training an open-set object detection model to be trained to determine an open-set object detection model, wherein the open-set object detection model to be trained includes an object detector, a prompt distiller, an object name generator, and a vision-language aligner; The training method of the open set target detection model specifically includes: Obtaining a decoder query and an object boundary prediction result of an input image through an object detector, and obtaining a text embedding feature of text information corresponding to the input image and an alignment score matrix between the decoder query and the text embedding feature through a vision-language aligner; determining, by the cue distiller, visual-linguistic cues from the decoder query, and generating, by the target name generator, target name predictions based on the visual-linguistic cues; constructing a joint loss term based on the object boundary prediction result and the alignment score matrix, constructing a mask alignment loss term based on the alignment score matrix and the text embedding feature, and constructing a distillation loss term based on the visual-linguistic cue and the text embedding feature; The open set object detection model to be trained is trained based on the joint loss term, the mask alignment loss term, and the distillation loss term to obtain an open set object detection model.

2. The method for training an open set object detection model according to claim 1, wherein: The process of obtaining the alignment score matrix specifically includes: Obtaining, by the vision-language aligner, a first alignment score from the decoder query to the text embedding feature and a second alignment score from the text embedding feature to the decoder query; An alignment score matrix between the decoder query and the text embedding feature is determined based on the first alignment score and the second alignment score.

3. The method for training an open set target detection model according to claim 1, wherein: Determining, by the cue distiller, visual-verbal cues based on the decoder query specifically includes: reading, by the hint distiller, backbone features and encoder features extracted by the object detector in the process of obtaining the decoder query; Interacting information between the decoder query and the backbone features through the prompt distiller to obtain a first query; Interacting the decoder query with the encoder features via the hint distiller to obtain a second query; Projecting the first query, the second query, and the decoder query to the channel dimension space where the text embedding features are located by the prompt distiller; The projected first query, the projected second query, and the projected decoder query are fused by the prompt distiller to obtain a visual-linguistic prompt.

4. The method for training an open set object detection model according to claim 1, wherein: Generating a target name prediction result based on the visual-linguistic prompt by the target name generator specifically includes: determining, by a large language model in the target name generator, a name embedding feature based on the visual-linguistic cue and a noisy text embedding feature, wherein the noisy text embedding feature is formed by adding Gaussian noise to the text embedding feature; A target name prediction result is generated according to the name embedding feature by a low-rank prediction head in the target name generator.

5. The method for training an open set object detection model according to claim 1, wherein: Constructing a joint loss term based on the target boundary prediction result and the alignment score matrix specifically includes: Obtaining the IoU score between the predicted bounding box and the true bounding box in the target boundary prediction result; A joint loss term is constructed based on the IoU score, the predicted binary score in the object boundary prediction result, and the alignment score matrix.

6. The method for training an open set object detection model according to claim 1, wherein: The step of constructing a mask alignment loss item based on the alignment score matrix and the text embedding feature specifically includes: generating a binary mask based on the text embedding feature; A mask alignment loss term is constructed based on the alignment score matrix and the binary mask.

7. The method for training an open set object detection model according to claim 1, wherein: The constructing of the distillation loss term based on the visual-linguistic cue and the text embedding feature specifically includes: Matching the visual-linguistic cue with the text embedding feature to obtain a matched visual-linguistic cue and a matched text embedding feature; A distillation loss term is constructed based on the matched visual-linguistic cues and the matched text embedding features.

8. A target detection method, characterized in that: An open set target detection model trained using the open set target detection model training method according to any one of claims 1 to 7 is used, wherein the target detection method specifically comprises: Obtain the image to be detected; The image to be detected is input into the open set target detection model, and the detection result corresponding to the image to be detected is output through the open set target detection model, wherein the detection result includes a target detection box and a target name.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the training method of the open set target detection model as described in any one of claims 1-7, and / or to implement the steps in the target detection method as described in claim 8.

10. A terminal device, characterized in that: include: processor and memory; The memory stores a computer-readable program executable by the processor; When the processor executes the computer-readable program, it implements the steps in the training method of the open set target detection model as described in any one of claims 1-7, and / or implements the steps in the target detection method as described in claim 8.

Images