Class incremental target detection method and related equipment

Abstract

The invention discloses a category incremental target detection method and related equipment, and the method comprises the steps: firstly extracting the multi-scale features of an input image, coding a known category text label to generate an embedded feature, calculating an average feature, and then generating an unknown category pseudo text embedded feature; a cross-modal feature pyramid is formed through a cross-modal fusion network, so that a known target is predicted and an unknown target is positioned; self-supervised training is carried out on an unknown target to generate a pseudo label to finish category fine tuning; and finally, model expansion is realized through incremental updating of a low-rank matrix and a shared matrix. According to the scheme, through pseudo-text embedding features and cross-modal fusion, the model is endowed with the unknown target recognition capability, and the application limitation of the open world is broken; by means of self-supervised false label fine tuning and matrix increment updating, full-category retraining is avoided, computing resource consumption and training duration are reduced, interference on original category weights is prevented, and stability and reliability of a target detection system during category expansion are guaranteed.

Description

Technical Field

[0001] The present application relates to the technical field of target detection, and more specifically, to a category-incremental target detection method and related equipment. Background Art

[0002] In today's era of rapid digital and intelligent development, target detection technology has been deeply integrated into multiple fields such as security monitoring, autonomous driving, industrial quality inspection, and medical image analysis. Its performance directly affects the operating results and application value of related systems.

[0003] However, traditional object detection methods have exposed numerous problems in practical applications. Existing technologies rely on training models from image datasets with limited categories, and can only recognize fixed objects predefined in the training set. In security monitoring scenarios, urban environments are complex and ever-changing, with new objects or special scene targets constantly emerging. Traditional methods are unable to learn and recognize unknown objects, resulting in numerous security vulnerabilities in monitoring. In the field of autonomous driving, unknown targets such as unconventional obstacles and new traffic signs frequently appear, making it difficult for traditional technologies to effectively identify them, posing a serious threat to driving safety and greatly limiting their application in open-world scenarios. Furthermore, when expanding detection categories, to avoid catastrophic forgetting, traditional methods require comprehensive retraining of all categories of data. This not only consumes significant computing resources and prolongs training time, but also interferes with the original category weights due to the overall weight update. Taking industrial quality inspection as an example, comprehensive retraining for newly added product defect types can easily lead to false detections and missed detections of existing defect types, seriously affecting detection accuracy.

[0004] In view of the dual bottlenecks of traditional methods in unknown object recognition and category expansion, this application scheme is proposed to meet the requirements of practical applications for adaptability and reliability of target detection systems. Summary of the Invention

[0005] The present application provides a category-incremental target detection method and related equipment, which achieves the beneficial effects of accurate detection of unknown targets, efficient category expansion, and improved system stability and reliability.

[0006] A category-incremental target detection method, comprising:

[0007] Extract multi-scale image features of the input image through an image feature extraction network;

[0008] Encode the text labels of known categories, generate text embedding features of known categories, and calculate the average embedding features;

[0009] generating a pseudo text embedding feature of an unknown category based on the average text embedding feature and the general target embedding feature;

[0010] fusing the multi-scale image features with the text embedding features and the pseudo-text embedding features through a cross-modal fusion network to generate a cross-modal feature pyramid;

[0011] Based on the cross-modal feature pyramid, the bounding box and category of the known target are predicted by similarity calculation, and the unknown target is located, using the universal label as the initial category;

[0012] Performing self-supervised training on the located unknown target to generate pseudo labels and performing fine-tuning training on the unknown category based on the pseudo labels to form a new category;

[0013] An independent low-rank matrix is allocated to each of the newly added categories, and model expansion is achieved by combining incremental updates of the shared matrix.

[0014] Optionally, the formula for generating the pseudo-text embedding feature of the unknown category is:

[0015]

[0016] in, Embed features for general purpose, is the average text embedding feature, and μ is a trainable parameter.

[0017] Optionally, the process of fusing the multi-scale image features with the text embedding features and the pseudo-text embedding features to generate a cross-modal feature pyramid includes:

[0018] Performing cross-modal interaction between the multi-scale image features, the text embedding features, and the pseudo-text embedding features;

[0019] Align image features with the spatial semantic information of text embeddings through the attention mechanism to generate multi-level fusion features;

[0020] The multi-level fusion features are spliced according to scale to construct a cross-modal feature pyramid.

[0021] Optionally, based on the cross-modal feature pyramid, bounding boxes and categories of known targets are predicted by similarity calculation, and unknown targets are located, using universal labels as initial categories, including:

[0022] For each spatial position of the cross-modal feature pyramid, the cosine similarity between it and the embedding features of the known category text and the embedding features of the unknown category pseudo-text is calculated respectively;

[0023] Filter candidate regions based on the similarity threshold, take the category with the highest similarity among known categories as the prediction result, and mark the region with a similarity higher than the threshold among unknown categories as a universal label;

[0024] Non-maximum suppression is performed on the candidate region, redundant prediction boxes are removed, and the final bounding box and category label are output.

[0025] Optionally, performing self-supervised training on the located unknown target to generate pseudo labels and performing fine-tuning training on the unknown category based on the pseudo labels to form a new category includes:

[0026] Extracting image features of the region where the unknown target is located, and generating corresponding newly added text description embedding features through a text encoder;

[0027] Calculating the cosine similarity between the newly added text description embedding feature and the pseudo text embedding feature;

[0028] If the cosine similarity exceeds a set threshold, a pseudo label of the unknown target is generated based on the cosine similarity, and the pseudo label is used to perform fine-tuning training on the unknown category to form a new category.

[0029] Optionally, the update formula of the shared matrix is:

[0030]

[0031] in, is the weight coefficient and 0< ≤1, is the updated shared matrix, is the historical sharing matrix, Assign independent low-rank matrices to the newly added categories.

[0032] A category-incremental target detection device, comprising:

[0033] An image feature unit, configured to extract multi-scale image features of an input image through an image feature extraction network;

[0034] The known feature unit is used to encode text labels of known categories, generate text embedding features of known categories, and calculate the average embedding features;

[0035] an unknown feature unit, configured to generate a pseudo text embedding feature of an unknown category based on the average text embedding feature and the general target embedding feature;

[0036] a feature pyramid unit, configured to fuse the multi-scale image features with the text embedding features and the pseudo-text embedding features through a cross-modal fusion network to generate a cross-modal feature pyramid;

[0037] An object localization unit is configured to predict the bounding box and category of a known object by similarity calculation based on the cross-modal feature pyramid, and to localize an unknown object using a universal label as an initial category;

[0038] A fine-tuning training unit is used to perform self-supervised training on the located unknown target, generate pseudo labels, and perform fine-tuning training on the unknown category based on the pseudo labels to form a new category;

[0039] The incremental update unit is used to allocate an independent low-rank matrix to each of the newly added categories, and realize model expansion in combination with the incremental update of the shared matrix.

[0040] A category-incremental target detection device includes a memory and a processor;

[0041] The memory is used to store programs;

[0042] The processor is used to execute the program to implement each step of the category-incremental target detection method as described in any one of the above items.

[0043] A readable storage medium stores a computer program thereon, which, when executed by a processor, implements the various steps of the category-incremental target detection method as described in any one of the above items.

[0044] A computer program product includes a computer program, which, when executed by a processor, performs the steps of any of the above-described class-incremental target detection methods.

[0045] As can be seen from the above technical solutions, the embodiments of this application provide a category-incremental object detection method and related equipment. This method uses an image feature extraction network to obtain multi-scale features of the input image, encodes text labels of known categories to generate text embedding features, and calculates the average features to generate pseudo-text embedding features for unknown categories. A cross-modal fusion network is used to fuse the multi-scale image features with the text embedding features to form a cross-modal feature pyramid. Based on this, the bounding boxes and categories of known objects are predicted, and unknown objects are located using a common label as the initial category. Self-supervised training is used to generate pseudo-labels for unknown objects, which are then fine-tuned to form new categories. Finally, independent low-rank matrices are assigned to the new categories, and model expansion is achieved through incremental updates of the shared matrix.

[0046] This solution effectively addresses the defects in the background technology: to address the problem that traditional methods have difficulty in identifying unknown objects, by generating pseudo-text embedding features of unknown categories and cross-modally fusing them with multi-scale image features, the model can learn the characteristic patterns of unknown targets and achieve accurate positioning and identification of unknown targets, breaking through the limitations of open-world scenario applications; to address the problem that traditional methods require overall retraining of all categories to expand detection categories, pseudo-labels are generated through self-supervised training to fine-tune unknown categories. Combined with incremental updates of low-rank matrices and shared matrices, there is no need to retrain all category data, reducing computing resource consumption and training time. At the same time, it avoids the interference of overall weight updates on the original category weights, improving the stability and reliability of the target detection system during category expansion, and promoting the in-depth application of target detection technology in multiple fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0048] Figure 1 This is a flow chart of a category-incremental target detection method disclosed in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of a class-incremental target detection device disclosed in an embodiment of the present application;

[0050] Figure 3 This is a hardware structure block diagram of a type of incremental target detection device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0051] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0052] The present application can be used in a variety of general or special computing device environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multi-processor devices, and distributed computing environments including any of the above devices or devices.

[0053] Next, we will introduce the application scheme. This application proposes the following technical scheme, please see below for details.

[0054] Figure 1 This is a flowchart of a category-incremental target detection method disclosed in an embodiment of the present application.

[0055] like Figure 1 As shown, the method may include:

[0056] Step S1: extract multi-scale image features of an input image through an image feature extraction network.

[0057] Specifically, we must first clarify the requirements. Incremental open target detection requires the model to not only detect known targets, but also locate unknown targets and learn their categories in continuous learning, so as to dynamically adapt to the newly added unknown target object categories and achieve adaptive category expansion in real scenarios. That is, at the current stage t, it is defined that there are n categories of known targets, and the set is , each category consists of samples S(I,T), where I represents the sample image and T represents the image category text label. Define the unknown target set as ,and After a round of incremental learning is completed, that is, in stage t+1, the set of known target categories will be expanded to .

[0058] The process of extracting multi-scale image features from an input image through an image feature extraction network. For example, in the DarkNet backbone network, this network extracts features at different levels of the image by stacking multiple convolutional layers. In shallow networks, the network primarily captures basic image details such as edges, texture, and color. As the network layers deepen, higher-level semantic features are gradually extracted, such as the shape, structure, and partial semantic information of objects. In this way, image features containing multi-scale information can be extracted from the input image, preserving the image's detailed information while acquiring high-level features with abstract semantics. This forms a multi-scale image feature set, providing rich visual information support for subsequent cross-modal fusion with text features.

[0059] Step S2: Encode the text labels of known categories, generate text embedding features of known categories, and calculate the average embedding features.

[0060] Specifically, for text labels of known categories (such as specific category names such as "car" and "pedestrian"), the pre-trained CLIP text encoder is used for processing. The CLIP text encoder can convert text labels into vector representations in a high-dimensional space, namely text embedding features. These vectors contain the semantic information of the text labels, so that the semantic associations and differences between texts of different categories can be reflected through the distance or similarity in the vector space. After obtaining the text embedding features of all known categories, these features are normalized (that is, the vector length is adjusted so that its modulus is 1 to eliminate the impact of the vector length on subsequent calculations), and then their average is calculated to obtain an average embedding feature that can represent the overall semantic features of all current known categories. This average embedding feature integrates the common semantic information of known categories and provides a key semantic reference benchmark for generating pseudo-text embedding features of unknown categories.

[0061] Step S3: Generate a pseudo text embedding feature of an unknown category based on the average text embedding feature and the general target embedding feature.

[0062] Specifically, a general target embedding feature is first defined. This feature represents the general semantic space of all possible targets in the open world and covers the potential common features of various targets. Then, using the average text embedding feature of the known category calculated previously, a trainable parameter is introduced, which is automatically learned and optimized during the model training process to calculate the pseudo-text embedding feature of the unknown category. The specific method is: subtract an offset related to the average text embedding feature of the known category from the general target embedding feature. The size of the offset is adjusted by the trainable parameter, and the direction is determined by the direction of the average text embedding feature of the known category. In this way, the generated pseudo-text embedding feature of the unknown category can be distinguished from the semantic space of the known category, representing a potential unknown semantic space different from the known category, providing a semantic-level feature representation for the subsequent positioning and recognition of unknown targets, so that the model can capture the characteristic patterns of unknown targets in the cross-modal fusion process.

[0063] The formula for generating the pseudo-text embedding feature of the unknown category is:

[0064]

[0065] in, Embed features for general purpose, is the average text embedding feature, and μ is a trainable parameter.

[0066] Step S4: The multi-scale image features are fused with the text embedding features and the pseudo-text embedding features through a cross-modal fusion network to generate a cross-modal feature pyramid.

[0067] Specifically, the RepVL-PAN network structure is used as a cross-modal fusion module to fuse the text embedding features of known categories, the pseudo-text embedding features of unknown categories, and the multi-scale image features. The network designs multiple layers of cross-modal interaction layers to perform semantic alignment and information interaction with the text embedding features on image features of different scales layer by layer: for each layer of multi-scale image features, the feature dimensions are first adjusted through convolution operations to match the dimensions of the text embedding features, and then the image features and text embedding features are fused through dot product operations or splicing operations to generate cross-modal features containing joint visual-semantic information. Through this bottom-up and top-down feature interaction process, a cross-modal feature pyramid is eventually formed, in which the features of each layer fuse the image information of the corresponding scale with the semantic information of known or unknown categories, providing feature representations with both spatial details and semantic understanding for subsequent target detection tasks.

[0068] Step S5: Based on the cross-modal feature pyramid, the bounding box and category of the known target are predicted by similarity calculation, and the unknown target is located, using the universal label as the initial category.

[0069] Specifically, the cross-modal feature pyramid is input into the model detection head, and the cosine similarity between the text embedding features of known categories, the pseudo-text embedding features of unknown categories, and the features of each spatial location in the feature pyramid is calculated. For known categories, a higher similarity value indicates a greater probability of an object of the corresponding category existing at that location. High-similarity regions are screened by setting a threshold, and bounding box coordinates of the known target are obtained through regression using a preset anchor box. The target category is then determined based on the text embedding feature corresponding to the maximum similarity. For unknown targets, the similarity between the pseudo-text embedding features and the feature pyramid is calculated to locate areas where the unknown target may exist. Since the category of the unknown target is not yet clear at this point, a general wildcard label such as "object" is used as its initial category. Finally, the non-maximum suppression (NMS) algorithm is used to filter out redundant prediction boxes and retain the optimal detection result, achieving accurate classification and localization of known targets and preliminary localization of unknown targets.

[0070] Step S6: Perform self-supervised training on the located unknown target, generate pseudo labels, and perform fine-tuning training on the unknown category based on the pseudo labels to form a new category.

[0071] Specifically, for the unknown target image region initially labeled "object," a backbone network (such as DarkNet) is used to extract its image features again, and a text description embedding of the target is generated using the CLIP text encoder (for example, an embedding vector describing "unknown object - with a specific shape / color" is generated based on the image content). The cosine similarity between this text description embedding and the pseudo-text embedding features of the unknown category is then calculated. High-confidence samples are screened based on the similarity threshold, and the corresponding pseudo-text embedding features are mapped to specific category labels (i.e., pseudo-labels are generated). These pseudo-labels are used to fine-tune the model and adjust the parameters of the feature modules related to the unknown category, enabling the model to learn the unique semantic features of the unknown target, thereby subdividing the initial general label "object" into specific new categories (such as "new traffic signs" and "obstacles of special shapes"), completing the transformation from unknown targets to newly added known categories.

[0072] Step S7: Allocate an independent low-rank matrix to each of the newly added categories, and implement model expansion in combination with incremental updates of the shared matrix.

[0073] Specifically, to prevent interference with existing categories during training (i.e., catastrophic forgetting), the model assigns each newly added category an independent low-rank matrix A. This matrix is responsible for capturing the specific features of the corresponding category (such as textures, structures, and other details unique to that category). During training, only the matrix A corresponding to the newly added category is updated, while the feature parameters learned for the original category remain unchanged. At the same time, a shared matrix B is introduced to capture common features shared by all categories (such as the basic geometric shape and spatial positional relationships of the object). When a new category is added, the shared matrix B is updated by weighted merging of the historical version with the new version learned for the current task.

[0074] The update formula of the shared matrix is:

[0075]

[0076] in, is the weight coefficient and 0< ≤1, is the updated shared matrix, is the historical sharing matrix, Assign independent low-rank matrices to the newly added categories.

[0077] Through this incremental update mechanism that combines category-specific matrices with shared matrices, the model can not only retain the detection capabilities of the original categories, but also efficiently learn the features of newly added categories, achieving dynamic expansion of detection categories and continuous optimization of model performance.

[0078] As can be seen from the above technical solutions, the embodiments of this application provide a category-incremental object detection method and related equipment. This method uses an image feature extraction network to obtain multi-scale features of the input image, encodes text labels of known categories to generate text embedding features, and calculates the average features to generate pseudo-text embedding features for unknown categories. A cross-modal fusion network is used to fuse the multi-scale image features with the text embedding features to form a cross-modal feature pyramid. Based on this, the bounding boxes and categories of known objects are predicted, and unknown objects are located using a common label as the initial category. Self-supervised training is used to generate pseudo-labels for unknown objects, which are then fine-tuned to form new categories. Finally, independent low-rank matrices are assigned to the new categories, and model expansion is achieved through incremental updates of the shared matrix.

[0079] This solution effectively addresses the defects in the background technology: to address the problem that traditional methods have difficulty in identifying unknown objects, by generating pseudo-text embedding features of unknown categories and cross-modally fusing them with multi-scale image features, the model can learn the characteristic patterns of unknown targets and achieve accurate positioning and identification of unknown targets, breaking through the limitations of open-world scenario applications; to address the problem that traditional methods require overall retraining of all categories to expand detection categories, pseudo-labels are generated through self-supervised training to fine-tune unknown categories. Combined with incremental updates of low-rank matrices and shared matrices, there is no need to retrain all category data, reducing computing resource consumption and training time. At the same time, it avoids the interference of overall weight updates on the original category weights, improving the stability and reliability of the target detection system during category expansion, and promoting the in-depth application of target detection technology in multiple fields.

[0080] In some embodiments of the present application, the process of generating a cross-modal feature pyramid by fusing the multi-scale image features with the text embedding features and the pseudo-text embedding features in step S4 by the cross-modal fusion network is introduced, which may specifically include:

[0081] Step S41: performing cross-modal interaction between the multi-scale image feature, the text embedding feature, and the pseudo-text embedding feature;

[0082] Step S42: Align the image features with the spatial semantic information of the text embedding through the attention mechanism to generate multi-level fusion features;

[0083] Step S43: Concatenate the multi-level fusion features according to scale to construct a cross-modal feature pyramid.

[0084] Specifically, the RepVL-PAN network structure serves as the core module for cross-modal fusion. It integrates multi-scale image features (including visual features at different levels) with text embeddings of known categories and pseudo-text embeddings of unknown categories. For each level of image features, a convolutional layer first adjusts the feature dimensions to align with the text embedding dimensions. Then, through dot products or feature concatenation, a preliminary correlation is established between visual and semantic features, achieving a preliminary fusion of cross-modal information and laying the foundation for subsequent semantic alignment.

[0085] Based on cross-modal interaction, an attention mechanism (such as self-attention or cross-attention) is introduced to fine-tune the spatial semantic relationship between image features and text embedding features. The attention mechanism calculates the semantic similarity between each spatial position in the image feature and the text embedding feature to generate an attention weight matrix. This matrix is used to weight and adjust the spatial response of the image feature, so that the model focuses on the image areas related to the text semantics (for example, in the image features corresponding to the text label "car", the attention mechanism will enhance the feature response of key parts such as the wheels and body). In this way, the semantic alignment of image features and text embeddings in spatial position is achieved, and a multi-level fusion feature containing joint visual-semantic information is generated. Each layer of fusion features corresponds to a different scale of the input image and combines spatial detail and semantic understanding capabilities.

[0086] The fused features of each level, aligned by the attention mechanism, are hierarchically spliced according to feature scale (e.g., high-to-low resolution or low-to-high resolution). For the top-down path, low-resolution, high-level semantic features are amplified through upsampling and laterally connected with high-resolution, low-level detail features, enhancing the guidance of high-level semantics on low-level details. For the bottom-up path, high-resolution, low-level features are scaled down through downsampling and fused with low-resolution, high-level features to enhance the semantic richness of low-level features. This bidirectional feature flow ultimately constructs a cross-modal feature pyramid, in which each layer of features fuses multi-scale visual information with semantic information of known or unknown categories, providing a hierarchical cross-modal feature representation for the subsequent object detection head to adapt to the detection needs of objects of varying sizes.

[0087] In some embodiments of the present application, step S5, based on the cross-modal feature pyramid, predicting the bounding box and category of the known target by similarity calculation, and locating the unknown target, and using the universal label as the initial category, is introduced. Specifically, it may include:

[0088] Step S51: For each spatial position of the cross-modal feature pyramid, the cosine similarity between the position and the known category text embedding feature and the unknown category pseudo-text embedding feature is calculated.

[0089] Specifically, the features at each level of the cross-modal feature pyramid are expanded into a two-dimensional spatial grid, with each grid point corresponding to a local area in the input image. For each grid point's feature vector, cosine similarity is calculated with all known text embedding features (e.g., vectors corresponding to categories like "car" and "pedestrian") and pseudo-text embedding features of unknown categories. Cosine similarity measures the directional similarity between vectors and ranges from [-1, 1]. Higher values indicate a higher degree of semantic match between the image region corresponding to the grid point and the text. Through this calculation, each spatial location receives a semantic relevance score with both known and unknown categories, providing a quantitative basis for target localization and classification.

[0090] Step S52: Filter candidate regions according to the similarity threshold, take the category with the highest similarity among known categories as the prediction result, and mark the region with a similarity higher than the threshold among unknown categories as a general label.

[0091] Specifically, set a similarity threshold for each spatial position:

[0092] Known category prediction: Compare the similarity between the position and all known category text embedding features. If the maximum similarity value is higher than the threshold, the position is predicted as the category with the highest similarity, and the corresponding bounding box is retained (generated based on the preset anchor box or feature map position mapping);

[0093] Unknown category positioning: If the similarity between the location and the unknown category pseudo-text embedding feature is higher than the threshold, it is determined to be an unknown target candidate area and temporarily marked with a general label such as "object", indicating that there is a target category in the area that has not been learned by the model.

[0094] Through threshold screening, invalid areas with low semantic matching are filtered out, and only high-confidence known target prediction results and unknown target candidate areas are retained to achieve preliminary classification and positioning of the target.

[0095] Step S53: Perform non-maximum suppression on the candidate region, remove redundant prediction boxes, and output the final bounding box and category label.

[0096] Specifically, the non-maximum suppression (NMS) algorithm is used to process multiple overlapping prediction boxes that may exist for the same target (such as candidate boxes generated by feature layers of different scales). The algorithm first sorts the candidate boxes according to their similarity scores, then starts with the high-scoring box and calculates its intersection-over-union ratio with subsequent boxes. If the intersection-over-union ratio exceeds a set threshold, the subsequent box is considered to point to the same target as the current box and is deleted, retaining only the box with the highest score. Through this process, redundant repeated predictions are removed, and ultimately a unique bounding box and corresponding category label (known category label or "object" general label) are output for each target, ensuring the accuracy and simplicity of the detection results.

[0097] In some embodiments of the present application, the process of performing self-supervised training on the located unknown target, generating pseudo labels, and fine-tuning the unknown category based on the pseudo labels to form a new category in step S6 is introduced, which may specifically include:

[0098] Step S61: extract the image features of the area where the unknown target is located, and generate corresponding new text description embedding features through a text encoder.

[0099] Specifically, for the unknown target region identified as the universal label "object" in step S5, the image of this region is again subjected to feature extraction using a backbone network (such as DarkNet) to obtain local image features. Simultaneously, the CLIP text encoder is used to encode the semantic description of the unknown target (e.g., a natural language description such as "irregular geometric shape with metallic luster" generated from the image content), converting the text description into a vector representation in a high-dimensional space. This is the newly added text description embedding feature. This embedding feature captures the semantic information of the unknown target and provides a semantic comparison basis for subsequent similarity calculations with the pseudo-text embedding feature.

[0100] Step S62: Calculate the cosine similarity between the newly added text description embedding feature and the pseudo text embedding feature.

[0101] Specifically, a cosine similarity calculation is performed between the newly added text description embedding features and the unknown category pseudo-text embedding features generated in step S3. Cosine similarity measures the directional consistency of two vectors in the semantic space. A higher value indicates a closer match between the semantics of the unknown target's text description and the latent semantics represented by the pseudo-text embedding features. This calculation quantifies the degree of match between the unknown target's actual semantics and the model's preset unknown semantic space, providing a screening basis for pseudo-label generation.

[0102] Step S63: If the cosine similarity exceeds a set threshold, a pseudo label of the unknown target is generated based on the cosine similarity, and the pseudo label is used to perform fine-tuning training on the unknown category to form a new category.

[0103] Specifically, a cosine similarity threshold is set. If the calculated similarity value exceeds this threshold, the semantic features of the unknown object are considered to be highly consistent with the unknown semantics represented by the pseudo-text embedding features, and the unknown object can be subdivided from the general label "object" into a specific new category. At this point, the model generates a pseudo-label for the unknown object (for example, "new mechanical part") based on the semantic cluster corresponding to the pseudo-text embedding feature with the highest similarity, and includes it in the set of known categories.

[0104] After generating pseudo-labels, the model is fine-tuned using samples of unknown objects containing the pseudo-labels. During fine-tuning, only the parameters of the feature modules related to the unknown categories are updated, while the parameters of the features of the known categories remain unchanged, thus avoiding interference with the detection capabilities of the original categories. Through iterative training, the model gradually learns the unique semantic features of the newly added categories, achieving the transition from unknown objects to newly known categories. Ultimately, this expands the detection system's category coverage and improves its adaptability in open-world scenarios.

[0105] The following describes a category-incremental target detection device provided in an embodiment of the present application. The category-incremental target detection device described below and the category-incremental target detection method described above can refer to each other.

[0106] See also Figure 2 , Figure 2 A schematic diagram of a class-incremental target detection device disclosed in an embodiment of the present application.

[0107] like Figure 2 As shown, the class incremental target detection device may include:

[0108] An image feature unit 110 is configured to extract multi-scale image features of an input image through an image feature extraction network;

[0109] A known feature unit 120 is used to encode text labels of known categories, generate text embedding features of known categories, and calculate average embedding features;

[0110] An unknown feature unit 130 is configured to generate a pseudo text embedding feature of an unknown category based on the average text embedding feature and the general target embedding feature;

[0111] A feature pyramid unit 140 is configured to fuse the multi-scale image features with the text embedding features and the pseudo-text embedding features through a cross-modal fusion network to generate a cross-modal feature pyramid;

[0112] An object localization unit 150 is configured to predict the bounding box and category of a known object by similarity calculation based on the cross-modal feature pyramid, and to localize an unknown object using a universal label as an initial category.

[0113] A fine-tuning training unit 160 is configured to perform self-supervised training on the located unknown target, generate pseudo labels, and perform fine-tuning training on the unknown category based on the pseudo labels to form a new category;

[0114] The incremental updating unit 170 is configured to allocate an independent low-rank matrix to each of the newly added categories, and implement model expansion in combination with the incremental update of the shared matrix.

[0115] As can be seen from the above technical solutions, the embodiments of this application provide a category-incremental object detection method and related equipment. This method uses an image feature extraction network to obtain multi-scale features of the input image, encodes text labels of known categories to generate text embedding features, and calculates the average features to generate pseudo-text embedding features for unknown categories. A cross-modal fusion network is used to fuse the multi-scale image features with the text embedding features to form a cross-modal feature pyramid. Based on this, the bounding boxes and categories of known objects are predicted, and unknown objects are located using a common label as the initial category. Self-supervised training is used to generate pseudo-labels for unknown objects, which are then fine-tuned to form new categories. Finally, independent low-rank matrices are assigned to the new categories, and model expansion is achieved through incremental updates of the shared matrix.

[0116] This solution effectively addresses the defects in the background technology: to address the problem that traditional methods have difficulty in identifying unknown objects, by generating pseudo-text embedding features of unknown categories and cross-modally fusing them with multi-scale image features, the model can learn the characteristic patterns of unknown targets and achieve accurate positioning and identification of unknown targets, breaking through the limitations of open-world scenario applications; to address the problem that traditional methods require overall retraining of all categories to expand detection categories, pseudo-labels are generated through self-supervised training to fine-tune unknown categories. Combined with incremental updates of low-rank matrices and shared matrices, there is no need to retrain all category data, reducing computing resource consumption and training time. At the same time, it avoids the interference of overall weight updates on the original category weights, improving the stability and reliability of the target detection system during category expansion, and promoting the in-depth application of target detection technology in multiple fields.

[0117] Optionally, the formula for generating the pseudo-text embedding feature of the unknown category is:

[0118]

[0119] in, Embed features for general purpose, is the average text embedding feature, and μ is a trainable parameter.

[0120] Optionally, the process of fusing the multi-scale image features with the text embedding features and the pseudo-text embedding features to generate a cross-modal feature pyramid includes:

[0121] Performing cross-modal interaction between the multi-scale image features, the text embedding features, and the pseudo-text embedding features;

[0122] Align image features with the spatial semantic information of text embeddings through the attention mechanism to generate multi-level fusion features;

[0123] The multi-level fusion features are spliced according to scale to construct a cross-modal feature pyramid.

[0124] Optionally, based on the cross-modal feature pyramid, bounding boxes and categories of known targets are predicted by similarity calculation, and unknown targets are located, using universal labels as initial categories, including:

[0125] For each spatial position of the cross-modal feature pyramid, the cosine similarity between it and the embedding features of the known category text and the embedding features of the unknown category pseudo-text is calculated respectively;

[0126] Filter candidate regions based on the similarity threshold, take the category with the highest similarity among known categories as the prediction result, and mark the region with a similarity higher than the threshold among unknown categories as a universal label;

[0127] Non-maximum suppression is performed on the candidate region, redundant prediction boxes are removed, and the final bounding box and category label are output.

[0128] Optionally, performing self-supervised training on the located unknown target to generate pseudo labels and performing fine-tuning training on the unknown category based on the pseudo labels to form a new category includes:

[0129] Extracting image features of the region where the unknown target is located, and generating corresponding newly added text description embedding features through a text encoder;

[0130] Calculating the cosine similarity between the newly added text description embedding feature and the pseudo text embedding feature;

[0131] If the cosine similarity exceeds a set threshold, a pseudo label of the unknown target is generated based on the cosine similarity, and the pseudo label is used to perform fine-tuning training on the unknown category to form a new category.

[0132] Optionally, the update formula of the shared matrix is:

[0133]

[0134] in, is the weight coefficient and 0< ≤1, is the updated shared matrix, is the historical sharing matrix, Assign independent low-rank matrices to the newly added categories.

[0135] The class-incremental target detection apparatus provided in the embodiments of the present application can be applied to class-incremental target detection equipment. Figure 3 The hardware structure diagram of the incremental target detection device is shown in FIG. Figure 3 ,The hardware structure of the category incremental target detection device may include: at least one processor 1, at least one communication interface 2, at least one memory 3 and at least one communication bus 4;

[0136] In the embodiment of the present application, the number of the processor 1, the communication interface 2, the memory 3, and the communication bus 4 is at least one, and the processor 1, the communication interface 2, and the memory 3 communicate with each other through the communication bus 4;

[0137] The processor 1 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention;

[0138] The memory 3 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory;

[0139] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0140] Extract multi-scale image features of the input image through an image feature extraction network;

[0141] Encode the text labels of known categories, generate text embedding features of known categories, and calculate the average embedding features;

[0142] generating a pseudo text embedding feature of an unknown category based on the average text embedding feature and the general target embedding feature;

[0143] fusing the multi-scale image features with the text embedding features and the pseudo-text embedding features through a cross-modal fusion network to generate a cross-modal feature pyramid;

[0144] Based on the cross-modal feature pyramid, the bounding box and category of the known target are predicted by similarity calculation, and the unknown target is located, using the universal label as the initial category;

[0145] Performing self-supervised training on the located unknown target to generate pseudo labels and performing fine-tuning training on the unknown category based on the pseudo labels to form a new category;

[0146] An independent low-rank matrix is allocated to each of the newly added categories, and model expansion is achieved by combining incremental updates of the shared matrix.

[0147] Optionally, the refined functions and extended functions of the program may refer to the above description.

[0148] The present application also provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0149] Extract multi-scale image features of the input image through an image feature extraction network;

[0150] Encode the text labels of known categories, generate text embedding features of known categories, and calculate the average embedding features;

[0151] generating a pseudo text embedding feature of an unknown category based on the average text embedding feature and the general target embedding feature;

[0152] fusing the multi-scale image features with the text embedding features and the pseudo-text embedding features through a cross-modal fusion network to generate a cross-modal feature pyramid;

[0153] Based on the cross-modal feature pyramid, the bounding box and category of the known target are predicted by similarity calculation, and the unknown target is located, using the universal label as the initial category;

[0154] Performing self-supervised training on the located unknown target to generate pseudo labels and performing fine-tuning training on the unknown category based on the pseudo labels to form a new category;

[0155] An independent low-rank matrix is allocated to each of the newly added categories, and model expansion is achieved by combining incremental updates of the shared matrix.

[0156] Optionally, the refined functions and extended functions of the program may refer to the above description.

[0157] The present application also provides a computer program product, including a computer program. When the computer program is executed by a processor, the execution method is as follows:

[0158] Extract multi-scale image features of the input image through an image feature extraction network;

[0159] Encode the text labels of known categories, generate text embedding features of known categories, and calculate the average embedding features;

[0160] generating a pseudo text embedding feature of an unknown category based on the average text embedding feature and the general target embedding feature;

[0161] fusing the multi-scale image features with the text embedding features and the pseudo-text embedding features through a cross-modal fusion network to generate a cross-modal feature pyramid;

[0162] Based on the cross-modal feature pyramid, the bounding box and category of the known target are predicted by similarity calculation, and the unknown target is located, using the universal label as the initial category;

[0163] Performing self-supervised training on the located unknown target to generate pseudo labels and performing fine-tuning training on the unknown category based on the pseudo labels to form a new category;

[0164] An independent low-rank matrix is allocated to each of the newly added categories, and model expansion is achieved by combining incremental updates of the shared matrix.

[0165] Optionally, the refined functions and extended functions of the program may refer to the above description.

[0166] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0167] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0168] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A category-incremental target detection method, characterized in that: include: Extract multi-scale image features of the input image through an image feature extraction network; Encode the text labels of known categories, generate text embedding features of known categories, and calculate the average embedding features; generating a pseudo text embedding feature of an unknown category based on the average text embedding feature and the general target embedding feature; fusing the multi-scale image features with the text embedding features and the pseudo-text embedding features through a cross-modal fusion network to generate a cross-modal feature pyramid; Based on the cross-modal feature pyramid, the bounding box and category of the known target are predicted by similarity calculation, and the unknown target is located, using the universal label as the initial category; Performing self-supervised training on the located unknown target to generate pseudo labels and performing fine-tuning training on the unknown category based on the pseudo labels to form a new category; An independent low-rank matrix is allocated to each of the newly added categories, and model expansion is achieved by combining incremental updates of the shared matrix.

2. The method according to claim 1, characterized in that The formula for generating the pseudo-text embedding feature of the unknown category is: in, Embed features for general purpose, is the average text embedding feature, and μ is a trainable parameter.

3. The method according to claim 1, characterized in that The process of fusing the multi-scale image features with the text embedding features and the pseudo-text embedding features to generate a cross-modal feature pyramid includes: Performing cross-modal interaction between the multi-scale image features, the text embedding features, and the pseudo-text embedding features; Align image features with the spatial semantic information of text embeddings through the attention mechanism to generate multi-level fusion features; The multi-level fusion features are spliced according to scale to construct a cross-modal feature pyramid.

4. The method according to claim 1, wherein Based on the cross-modal feature pyramid, the bounding boxes and categories of known targets are predicted by similarity calculation, and unknown targets are located, using universal labels as initial categories, including: For each spatial position of the cross-modal feature pyramid, the cosine similarity between it and the embedding features of the known category text and the embedding features of the unknown category pseudo-text is calculated respectively; Filter candidate regions based on the similarity threshold, take the category with the highest similarity among known categories as the prediction result, and mark the region with a similarity higher than the threshold among unknown categories as a universal label; Non-maximum suppression is performed on the candidate region, redundant prediction boxes are removed, and the final bounding box and category label are output.

5. The method according to claim 1, wherein Performing self-supervised training on the located unknown target, generating pseudo labels, and fine-tuning the unknown category based on the pseudo labels to form a new category, including: Extracting image features of the region where the unknown target is located, and generating corresponding newly added text description embedding features through a text encoder; Calculating the cosine similarity between the newly added text description embedding feature and the pseudo text embedding feature; If the cosine similarity exceeds a set threshold, a pseudo label of the unknown target is generated based on the cosine similarity, and the pseudo label is used to perform fine-tuning training on the unknown category to form a new category.

6. The method according to claim 1, characterized in that The update formula of the shared matrix is: in, is the weight coefficient and 0< ≤1, is the updated shared matrix, is the historical sharing matrix, Assign independent low-rank matrices to the newly added categories.

7. A category-incremental target detection device, characterized in that: include: An image feature unit, configured to extract multi-scale image features of an input image through an image feature extraction network; The known feature unit is used to encode text labels of known categories, generate text embedding features of known categories, and calculate the average embedding features; an unknown feature unit, configured to generate a pseudo text embedding feature of an unknown category based on the average text embedding feature and the general target embedding feature; a feature pyramid unit, configured to fuse the multi-scale image features with the text embedding features and the pseudo-text embedding features through a cross-modal fusion network to generate a cross-modal feature pyramid; An object localization unit is configured to predict the bounding box and category of a known object by similarity calculation based on the cross-modal feature pyramid, and to localize an unknown object using a universal label as an initial category; A fine-tuning training unit is used to perform self-supervised training on the located unknown target, generate pseudo labels, and perform fine-tuning training on the unknown category based on the pseudo labels to form a new category; The incremental update unit is used to allocate an independent low-rank matrix to each of the newly added categories, and realize model expansion in combination with the incremental update of the shared matrix.

8. A category-incremental target detection device, characterized in that: including memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement the steps of the category-incremental target detection method according to any one of claims 1 to 6.

9. A readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the category-incremental target detection method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the category-incremental target detection method according to any one of claims 1 to 6 are executed.

Images