An incremental learning method for open world object detection based on large language model

Abstract

The application provides an incremental learning method for open world target detection based on a large language model, comprising: obtaining a first open world target detection model trained by a first RGB image sample in which a target of a first category is labeled; establishing a training set comprising a plurality of second RGB image samples, each of which is labeled with a target of the first category and a target of a second category; processing the first category and the second category by using a large language model to generate attribute features in a text form of each category; processing the second RGB image sample and the attribute features in a text form of each category by using the first open world target detection model to obtain a target frame prediction value, a target category prediction value, and an unknown category target prediction value, thereby determining a total loss value; and updating parameters of the first open world target detection model based on the total loss value, thereby obtaining a second open world target detection model. The application enhances the generalization ability and adaptability of the model in a new scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of open world object detection, and in particular to an incremental learning method for open world object detection based on a large language model. BACKGROUND

[0002] At present, open world object detection (OWOD) has the following technical problems:

[0003] In terms of unknown rejection, the accuracy in complex scenes and fine-grained class recognition still needs to be improved;

[0004] In terms of incremental learning, the incremental learning process usually requires a long training time and computing resources, and the efficiency of incremental learning still needs to be improved. The incremental learning method based on playback needs to store a large amount of old class data, which will bring additional storage and computing costs. The incremental learning method based on network structure expansion will increase the complexity of the model, resulting in an increase in the number of model parameters, and an increase in the training and inference costs. SUMMARY

[0005] Therefore, the present application provides an incremental learning method for open world object detection based on a large language model to solve the above technical problems.

[0006] In a first aspect, an incremental learning method for open world object detection based on a large language model is provided, comprising:

[0007] obtaining a first open world object detection model, wherein the first open world object detection model is trained by first RGB image samples labeled with targets of a first class;

[0008] establishing a training set, wherein the training set comprises a plurality of second RGB image samples, and each second RGB image sample is labeled with targets of the first class and a second class;

[0009] processing the first class and the second class by using a large language model to generate attribute features in text form of each class;

[0010] processing the second RGB image samples and the attribute features in text form of each class by using the first open world object detection model to obtain target box prediction values, target class prediction values, and unknown class target prediction values;

[0011] determine a first loss value based on the target bounding box prediction value and the target bounding box true value; determine a second loss value based on the target category prediction value and the target category label value; determine a third loss value based on the unknown category target prediction value and the preset threshold value; and determine a total loss value based on the first loss value, the second loss value, and the third loss value.

[0012] update the parameters of the first open-world object detection model based on the total loss value, thereby obtaining a second open-world object detection model.

[0013] In a possible implementation, the text-form attribute features of each category include material, property, context, environment, size, shape, color, texture, appearance, and performance.

[0014] In a possible implementation, the first open-world object detection model includes a text backbone network, a visual feature extraction module, a similarity matrix calculation module, a category prediction module, a Transformer encoder-decoder, a bounding box regression branch, a known category classification branch, and an objectness detection head.

[0015] processing the second RGB image sample and the text-form attribute features of each category by using the first open-world object detection model to obtain the target bounding box prediction value, the target category prediction value, and the unknown category target prediction value; the processing includes:

[0016] processing all categories of text-form attribute features by using the text backbone network to obtain a category semantic mapping matrix representing text features;

[0017] processing the second RGB image sample by using the visual feature extraction module to obtain visual features;

[0018] calculating a similarity matrix between the text features and the visual features by using the similarity matrix calculation module;

[0019] processing the learnable category-feature mapping matrix and the similarity matrix by using the category prediction module to obtain a first category prediction result of the target of the second RGB image sample;

[0020] processing the visual features by using the Transformer encoder-decoder to obtain context features of each query;

[0021] processing the context features of each query by using the bounding box regression branch to obtain the target bounding box prediction value;

[0022] processing the context features of each query by using the classification branch to obtain a second category prediction result of each target,

[0023] The sum of the first category prediction result and the second category prediction result of each target is calculated by using an adder to obtain the target category prediction result of the second RGB image sample;

[0024] The context feature of each query is processed by using the object detection head to obtain the unknown category target prediction value.

[0025] In a possible implementation, the visual feature extraction module comprises a visual backbone network, a splicing unit and a RELU activation function.

[0026] The second RGB image sample is processed by using the visual feature extraction module to obtain the visual feature, which comprises:

[0027] The second RGB image sample is processed by using the visual backbone network to obtain visual features of multiple scales.

[0028] The visual features of multiple scales are spliced by using the splicing unit to obtain the fused visual feature V final .

[0029] The fused visual feature is processed by using the RELU activation function for dimension reduction to obtain the visual feature V reduced with the same dimension as the text feature.

[0030] V reduced = ReLU(W reduced ·V final )

[0031] Wherein, W reduced is a weight matrix, ReLU(·) is an activation function, and the size of V reduced is b×d′, b is the batch size, and d′ is the dimension of the text feature.

[0032] In a possible implementation, the similarity matrix between the text feature and the visual feature is calculated by using a similarity matrix calculation module, which comprises:

[0033] The visual feature V exp extended to three dimensions is calculated.

[0034]

[0035] Wherein, l n is a full one vector with a length of n, and n is the total number of the first category and the second category.

[0036] The text feature E exp extended to three dimensions is calculated.

[0037]

[0038] Wherein, lb is a full one vector with length b; E is a category semantic mapping matrix with size n x d', and is a tensor outer product operation;

[0039] A similarity matrix S is calculated, and an element S i,j is:

[0040]

[0041] wherein, is the i-th row of V exp ; and is the j-th column of E exp .

[0042] In one possible implementation, the category prediction module is used to process the learnable category-feature mapping matrix and the similarity matrix to obtain a first category prediction result of the target of the second RGB image sample; and the method comprises:

[0043] The first category prediction result logits is:

[0044] logits = S x W τ

[0045] wherein, W is a learnable category-feature mapping matrix.

[0046] In one possible implementation, based on the total loss value, the parameters of the first open world target detection model are updated, and the method comprises:

[0047] The parameters of the visual backbone network are determined as first parameters, and the parameters of the learnable category-feature mapping matrix W, the Transformer encoder-decoder, the bounding box regression branch, the known category classification branch, and the objectness detection head are determined as second parameters;

[0048] The first parameters are frozen, the second parameters are updated based on the total loss value, and the second parameters are converged until the second parameters are converged;

[0049] The first parameters and the second parameters are updated based on the total loss value, and the first parameters and the second parameters are converged until the first parameters and the second parameters are converged.

[0050] In a second aspect, the embodiments of the present application provide an incremental learning device for open world target detection based on a large language model, which comprises:

[0051] An acquisition unit is configured to acquire a first open world target detection model, wherein the first open world target detection model is trained by a first RGB image sample with a target of a first category labeled;

[0052] The establishing unit is configured to establish a training set, the training set comprising a plurality of second RGB image samples, each second RGB image sample being labeled with a first category and a second category of target;

[0053] The first processing unit is configured to process the first category and the second category by using a large language model to generate attribute features in a text form of each category;

[0054] The second processing unit is configured to process the second RGB image sample and the attribute features in the text form of each category by using the first open-world target detection model to obtain a target box prediction value, a target category prediction value, and an unknown category target prediction value;

[0055] The determining unit is configured to determine a first loss value based on the target box prediction value and a target box true value, determine a second loss value based on the target category prediction value and a target category labeled value, determine a third loss value based on the unknown category target prediction value and a preset threshold value, and determine a total loss value based on the first loss value, the second loss value, and the third loss value;

[0056] The incremental learning unit is configured to update parameters of the first open-world target detection model based on the total loss value, thereby obtaining a second open-world target detection model.

[0057] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method of the embodiments of the present application when executing the computer program.

[0058] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executable on a processor to implement the method of the embodiments of the present application.

[0059] The present application improves the incremental learning performance, reduces the catastrophic forgetting, and enhances the generalization ability and adaptability of the model in a new scenario. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the specific embodiments or prior art in the present application, the drawings needed in the specific embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0061] Figure 1 The flowchart of the incremental learning method of open-world target detection based on a large language model provided by the embodiments of the present application;

[0062] Figure 2 The flowchart of the implementation process of the specific application scenario provided by the embodiment of the present application is shown in the following figure:

[0063] Figure 3 The functional structure diagram of the incremental learning device for open world object detection based on a large language model provided by the embodiment of the present application is shown in the following figure:

[0064] Figure 4 The functional structure diagram of the electronic device provided by the embodiment of the present application is shown in the following figure. DETAILED DESCRIPTION

[0065] To make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0066] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.

[0067] First, the design idea of the embodiments of the present application is briefly introduced.

[0068] Open world object detection is an advanced object detection method designed to address the challenge of class inconsistency between training and test sets in the real world. Unlike traditional object detection tasks, OWOD not only requires the model to recognize known classes seen in the training phase, but also requires the model to detect these “unknown classes” and label them when encountering unknown objects, i.e. having the ability of “unknown rejection”. In addition, the model should have the ability of “incremental learning”, i.e. being able to learn new classes over time without losing the detection ability of known classes.

[0069] The following are representative methods of open world object detection:

[0070] CAT model: decouples the localization and classification of the object detection process and improves the decoder process. This model uses a multi-scale transformer encoder combined with a learnable position query to detect object instances, and adopts an adaptive pseudo-label mechanism to fully utilize the model's prior knowledge to generate pseudo-label candidate boxes and confidence. The performance of the model in unknown rejection and incremental learning is verified on benchmark datasets such as MS-COCO and PASCAL VOC.

[0071] UniDetector model: separates the proposal and ROI classification processes, introduces large-scale image text, and combines model parameters and networks trained on them. It uses a class-independent positioning network to calculate the positioning confidence and classification confidence separately. It uses a heterogeneous label space strategy to explore the performance of three model structures in open-world object detection tasks.

[0072] Vector Orthogonal Theory-based Method: The object and feature are represented as the amplitude and direction of polar coordinates. The regularization term is used to penalize the correlation between the two predictions. The affine transformation is used to calibrate the offset of important features in the incremental learning phase. The triple orthogonal enhances the objectivity of the model in identifying unknown objects, making it no longer overly dependent on supervised data sets.

[0073] Sal ienDet: Introduces a saliency map of spectral residuals into the object detector, which enhances image features to generate object proposals. To better train the model, the dataset is labeled with known class labels using the intersection-over-union metric, and at each training iteration, it is randomly proposed. This ensures that the training process is not confused by known objects and encourages more proposal searches.

[0074] Incremental learning is a key component of OWOD, which enables the model to learn new classes without retraining. The core challenge of incremental learning is to avoid "catastrophic forgetting", that is, not to lose the ability to recognize old classes when learning new classes. According to the technical idea of overcoming forgetting, incremental learning methods can be divided into the following four categories:

[0075] Incremental learning based on parameter regularization: important parameters are selected by importance estimation, and a penalty loss for important parameters is introduced in the loss function to limit the change of the parameter. For example, Elastic Weight Consolidation (EWC) uses the Fisher information matrix to measure the importance of each parameter in solving previous tasks, and modifies the gradient descent direction by explicit calculation to ensure that the model does not significantly decrease the performance of the old task while learning new tasks. Other methods such as parameter space rotation, disperse the weights that are originally close in the parameter space to reduce the interference between different tasks and reduce the probability of forgetting.

[0076] Incremental learning based on knowledge distillation: by establishing a distillation relationship between the output results or intermediate features of new and old models, the model's updating speed towards new classes is limited. For example, LwF uses KL divergence to measure the difference between the output distribution of the model on the new task and the previous task. By minimizing the KL divergence, the model can retain as much knowledge as possible from the previous task while learning new tasks. Other methods such as PODnet and sample relationship distillation use different strategies to constrain the model learning process to avoid catastrophic forgetting.

[0077] Replay-based incremental learning: Avoid forgetting by storing or generating representative data or features of the original model to participate in model update training. For example, ICARL is the first incremental learning method using data replay. To solve the storage space and data security problems, some methods choose to store old model class features or use data generation methods. For example, a generative adversarial network (GAN) is used to generate old class data, while the LR method saves the activation values of the intermediate layers of the model to participate in the update training, and adopts a hierarchical learning rate strategy to alleviate the problem of gradual invalidation of old class features.

[0078] Network structure-based incremental learning: Incremental learning is intuitively achieved by expanding the model network structure, directly outputting by adding new classes or task detection heads. For example, a gating mechanism is used to dynamically create and destroy paths between network layers to adapt to the needs of new tasks. Using an incremental learning structure composed of dynamic structure reorganization and master branch distillation prototype balancing, a side branch is added to the current model through a structure expansion method to optimize the training of new classes, and then the side branch information is losslessly integrated into the main branch through a structure reparameterization method.

[0079] To solve the problem that the existing technology in the field of open world object detection faces when detecting unknown objects, that is, the effective distinction between background and unknown objects, and the lack of supervision signal leads to limited model generalization ability. The present application proposes an OWOD framework based on the GIL paradigm, which integrates the attribute class prior knowledge generated by a large language model (LLM) into a visual detection model to realize open world object detection.

[0080] The present application introduces semantic priors generated by LLM to enhance the recognition ability of unknown classes through multi-modal feature similarity calculation. Specifically, based on the similarity score, the region with high objectness but deviating from the known class distribution is determined as the unknown object. At the same time, a learnable class-feature matrix is introduced, and the preliminary prediction of known classes is obtained by multiplying it with the similarity score. In the output stage, this prediction result is used to enhance the objectness detection head and is fused with the variability decoder output to generate the final class determination, realizing the unified recognition of known and unknown classes.

[0081] Weight freezing, gradual unfreezing of backbone network, and example replay strategy are adopted. In the first stage, the backbone network is frozen to maintain the previously learned knowledge and prevent overfitting to new features. In this stage, the model uses the detection head of new classes and old classes and the semantic fusion module to learn new class features. In the second stage, the example replay strategy is adopted to review the instances encountered before, and representative images of old classes are retained.

[0082] The present application can achieve:

[0083] Cross-modal fusion and optimization: The target attributes in the language model are fused with the visual features through semantic mapping matrices and cross-modal similarity measures, optimizing the detection performance of unknown targets.

[0084] Incremental learning: Combining weight freezing, gradual unfreezing of backbone networks, and example replay, it prevents forgetting old classes when introducing new classes, enhancing the stability of the model.

[0085] Unknown class target detection: Using the attribute information generated by LLM, the model's ability to recognize unknown targets is enhanced, improving the recall rate of unknown classes.

[0086] Known class target detection: Through the cross-modal fusion mechanism, the detection accuracy of known classes is improved, maintaining the detection ability of previously learned classes.

[0087] The advantages of the present application are:

[0088] 1. Improved incremental learning performance, reducing catastrophic forgetting

[0089] The present application adopts an incremental learning strategy that combines weight freezing, gradual unfreezing of backbone networks, and feature replay. In the early stage of each incremental learning phase, the backbone network is frozen to maintain previously learned knowledge and prevent overfitting to new features; in the later stage of training, the backbone network is gradually unfrozen and fine-tuned to enhance the model's adaptability to new classes. At the same time, the example replay strategy reviews previously encountered instances, preserving representative images of old classes, further reducing the problem of catastrophic forgetting. Experimental results show that in Task 3 and Task 4 of the M-OWODB dataset, the detection performance of previously known classes is improved by 1.4%, proving the effectiveness of the proposed incremental learning strategy.

[0090] 2. Enhanced model generalization and adaptability in new scenarios

[0091] By integrating the attribute information generated by LLM, more rich semantic information is provided, enhancing the model's generalization and recognition ability for complex targets. Experimental results show that the present application has stronger ability in identifying small size unknown targets, for example, in the experiments of the M-OWODB dataset, small size unknown targets are successfully detected, and the confidence is higher than that of the PROB method. In addition, the present application also performs better in detecting known classes that have been learned, while the PROB model fails to detect or misclassifies.

[0092] 3. Improved accuracy of unknown target detection

[0093] The application effectively distinguishes the background and unknown target by integrating the class-independent semantic target attributes generated by the LLM into target detection, capturing the implicit dependency between the image and various attributes. Experimental results show that on the two benchmark datasets M-OWODB and S-OWODB, the application is superior to the comparative method PROB in the unknown class recall rate (U-Recall) index, especially in task 1 and task 3 of the M-OWODB dataset, which increases by 1.2% and 1.4% respectively, and in task 1, task 2 and task 3 of the S-OWODB dataset, which increases by 1.4%, 0.3% and 2.8% respectively. This shows that the application can more effectively identify unknown targets and reduce the rate of missed detection.

[0094] 4. Enhanced robustness of known target detection

[0095] By cross-modal fusion mechanism, the similarity between the semantic mapping matrix and the visual feature is calculated, and the supervised learning of known categories is used to promote unsupervised learning of unknown categories, enhancing the generalization ability of the target detection head. Experimental results show that on the M-OWODB and S-OWODB datasets, the application is comparable or superior to previous methods in the average precision (mAP) of known categories, especially in task 3 of the M-OWODB dataset, the mAP of new categories increases by 5.1%, and the mAP of previously learned categories increases by 0.8%. This shows that the application not only can effectively detect unknown targets, but also can maintain the detection accuracy of known categories, enhancing the robustness of the model.

[0096] The key points and protection points of the application are:

[0097] 1. Construction of generative incremental learning paradigm (GIL):

[0098] Key points: Integrate the class-independent semantic target attributes generated by the large language model (LLM) into the visual detection model, measure the cross-modal correspondence by encoding semantic information and visual features, and capture the implicit dependency between the image and various attributes.

[0099] Protection points: Use the large language model to generate attribute feature sets containing multiple category attributes, and build a semantic mapping matrix through a semantic encoder, and perform cross-modal similarity calculation with visual features to enhance the detection ability of the model for unknown categories.

[0100] 2. Cross-modal fusion and optimization mechanism:

[0101] Key points: Integrate visual features with category-level semantic features, capture the complex relationship between attribute information and visual features by calculating similarity scores, and use multi-modal fusion features for known category prediction and unknown category detection.

[0102] Protection points: By globally average pooling the multi-scale visual features extracted from the backbone and connecting them, then applying a linear transformation to reduce their dimensionality, ensuring alignment with the text features in the shared feature space, and quantifying the relationship between visual and text features through cosine similarity, a similarity matrix is obtained, and then through matrix multiplication, a classification score is generated to optimize known class classification and enhance the decision of unknown classes through a target adjustment factor.

[0103] 3. Incremental learning strategy:

[0104] Key points: In the first stage, the backbone network is frozen to maintain previously learned knowledge and prevent overfitting to new features. In this stage, the model uses detection heads for new and old classes and a semantic fusion module to learn new class features. Then, the backbone network is gradually unfrozen for fine-tuning to enhance the model's adaptability to new classes. In the second stage, a feature replay strategy is used to review previously encountered instances, preserving representative images of old classes.

[0105] Protection points: During training, the weights of the backbone network are first frozen, and detection heads for new and old classes and a semantic fusion module are used to learn new class features, then the backbone network is gradually unfrozen for fine-tuning; at the same time, a feature replay strategy is used to review previously encountered instances, preserving representative images of old classes to alleviate the problem of catastrophic forgetting.

[0106] After introducing the application scenarios and design ideas of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.

[0107] As shown in Figure 1 , the present application provides an incremental learning method for open world object detection based on a large language model, comprising the following steps:

[0108] Step 101: Obtain a first open world object detection model, wherein the first open world object detection model is trained by a first RGB image sample labeled with a target of a first class;

[0109] Wherein, the first open world object detection model can only recognize the target of the first class, and the target of the second class is recognized as an object.

[0110] Step 102: Establish a training set, the training set comprising a plurality of second RGB image samples, each second RGB image sample labeled with a target of a first class and a target of a second class;

[0111] Step 103: Process the first class and the second class using a large language model to generate attribute features in text form for each class;

[0112] Step 104: processing the second RGB image sample and the text-form attribute features of each category by using the first open-world object detection model to obtain target frame prediction values, target category prediction values, and unknown category target prediction values;

[0113] Step 105: determining a first loss value based on the target frame prediction values and the target frame true values; determining a second loss value based on the target category prediction values and the target category label values; determining a third loss value based on the unknown category target prediction values and a preset threshold value; and determining a total loss value based on the first loss value, the second loss value, and the third loss value.

[0114] Step 106: updating the parameters of the first open-world object detection model based on the total loss value, thereby obtaining a second open-world object detection model.

[0115] In some embodiments, the text-form attribute features of each category include material, property, context, environment, size, shape, color, texture, appearance, and performance.

[0116] The names and descriptions of the first category and the second category in the current data set are provided to a large language model (LLM) in a predefined format. The LLM generates an attribute feature set containing multiple category attributes. Each category obtains a specific attribute description, such as a color attribute of red or yellow, or a shape attribute of square or triangle.

[0117] In some embodiments, the first open-world object detection model includes a text backbone network, a visual feature extraction module, a similarity matrix calculation module, a category prediction module, a Transformer encoder-decoder, a bounding box regression branch, a known category classification branch, and an objectness detection head.

[0118] The visual backbone network is designed to support parameter freezing and unfreezing strategies, thereby flexibly controlling knowledge transfer and preserving historical information at different training stages. After the image passes through the backbone network, intermediate representations will be extracted from multiple different depth layers. These feature representations have different receptive fields and semantic levels. After multi-scale fusion, they are spliced into a unified multi-scale visual feature map to more fully capture multi-level information from local texture to global semantics.

[0119] To enhance the robustness of the model under target scale changes and non-rigid shapes, the encoder uses a deformable self-attention mechanism to dynamically select key positions for attention, thereby reducing computational overhead while more effectively capturing long-range dependencies and modeling context information. This mechanism enables the model to select the most relevant feature positions in sparse space for target recognition, significantly improving the spatial modeling capability of the target.

[0120] After the feature encoding is completed, a set of learnable queries is introduced as the input of the Transformer decoder. These queries will be used to interact with the visual features during inference to predict the presence and location of target instances. To enhance the distribution modeling capability of the query representation, a covariance estimation module is introduced, which uses an exponential moving average strategy to statistically model all current query embeddings, dynamically estimating the mean and covariance matrix of their distribution. Based on this estimation, the Mahalanobis distance between each query embedding and the global distribution is further calculated to measure its representativeness and uncertainty, providing prior support for subsequent object discrimination and unknown class detection.

[0121] Finally, after passing through the Transformer decoder and combining the context features of each query, the model makes predictions through three branches respectively: the bounding box regression branch outputs the initial position information of the candidate target; the known class classification branch is used to determine which known class the current target belongs to; and the object detection head is used to determine whether the current query corresponds to a real target in the image and further supports the identification of unknown targets. Through the above mechanism, the model realizes the unified modeling and discrimination of known and unknown targets, effectively improving the adaptability and generalization ability in the open-world object detection task.

[0122] During this process, the model introduces a learnable class-feature mapping matrix to model the internal structure between class semantics. By multiplying this matrix with the multi-modal similarity scores, the preliminary prediction distribution logits for known classes are obtained, capturing the semantic similarity and discriminative features. To enhance the model's ability to identify unknown classes, the preliminary prediction results are further introduced into the object detection head to improve its sensitivity and robustness in identifying unknown targets.

[0123] In the final output stage, the model fuses the above preliminary prediction distribution with the class scores output by the deformable decoder, achieving a more robust and semantically consistent classification output. This fusion strategy effectively alleviates the confusion between known and unknown class features, while improving the generalization performance in out-of-distribution sample detection. This method demonstrates the enhancing effect of visual-linguistic fusion on multi-class identification tasks in open-world scenarios.

[0124] In some embodiments, the first open-world object detection model is used to process the second RGB image sample and the text-form attribute features of each class to obtain target box prediction values, target class prediction values, and unknown class target prediction values; including:

[0125] The text backbone network is used to process the text-form attribute features of all classes to obtain a class semantic mapping matrix representing the text features;

[0126] The second RGB image sample is processed by using a visual feature extraction module to obtain visual features;

[0127] A similarity matrix calculation module is used to calculate a similarity matrix between the text features and the visual features;

[0128] A category prediction module is used to process the learnable category-feature mapping matrix and the similarity matrix to obtain a first category prediction result of the target of the second RGB image sample;

[0129] A Transformer encoder-decoder is used to process the visual features to obtain a context feature of each query;

[0130] A bounding box regression branch is used to process the context feature of each query to obtain a target box prediction value;

[0131] A classification branch is used to process the context feature of each query to obtain a second category prediction result of each target,

[0132] An adder is used to calculate the sum of the first category prediction result and the second category prediction result of each target to obtain a target category prediction result of the second RGB image sample;

[0133] An objectness detection head is used to process the context feature of each query to obtain an unknown category target prediction value.

[0134] In some embodiments, the visual feature extraction module includes a visual backbone network, a concatenation unit, and a RELU activation function;

[0135] The second RGB image sample is processed by using a visual feature extraction module to obtain visual features; including:

[0136] The second RGB image sample is processed by using a visual backbone network to obtain visual features of multiple scales;

[0137] The visual features of multiple scales are concatenated by using a concatenation unit to obtain fused visual features V final ;

[0138] The fused visual features are processed by using a RELU activation function for dimension reduction to obtain visual features V reduced with the same dimension as the text features:

[0139] V reduced =ReLU(W reduced ·V final )

[0140] wherein, W reduced is a weight matrix, ReLU(·) is an activation function, and V reducedThe size of the text feature is bxd', b is a batch size, and d' is a dimension of the text feature.

[0141] In some embodiments, a similarity matrix between the text feature and the visual feature is calculated by using a similarity matrix calculation module; including:

[0142] The visual feature V is extended to three dimensions exp :

[0143]

[0144] Wherein, l n is a full 1 vector with a length of n, n is the total number of the first category and the second category;

[0145] The text feature E is extended to three dimensions exp :

[0146]

[0147] Wherein, l b is a full 1 vector with a length of b; E is a category semantic mapping matrix, with a size of nxd', is a tensor outer product operation;

[0148] The similarity matrix S is calculated, and the element S i,j of the i-th row and the j-th column of the similarity matrix S is:

[0149]

[0150] Wherein, is the i-th row of V exp ; is the j-th column of E exp .

[0151] In some embodiments, the first category prediction result of the target of the second RGB image sample is obtained by processing the learnable category-feature mapping matrix and the similarity matrix by using a category prediction module; including:

[0152] The first category prediction result logits is:

[0153] logits=S×W τ ∈R b×q×n×a

[0154] where W is a learnable class-feature mapping matrix. Where q is the number of queries, a is the number of attributes. Through transposition and expansion, logits are used to predict known classes. During training, due to the invariance of matrix E and the similarity algorithm, the model continuously optimizes W through backpropagation, thus finding the most suitable feature content for the current scene. This optimization ensures that, when integrating matching information and unknown class detection heads, supervised known class data can help train unsupervised unknown class data.

[0155] In some embodiments, based on the total loss value, the parameters of the first open world object detection model are updated, including:

[0156] The parameters of the visual backbone network are determined as first parameters, and the parameters of the learnable class-feature mapping matrix W, the Transformer encoder-decoder, the bounding box regression branch, the known class classification branch, and the objectness detection head are determined as second parameters;

[0157] The first parameters are frozen, and the second parameters are updated based on the total loss value until the second parameters converge;

[0158] The first parameters and the second parameters are updated based on the total loss value until the first parameters and the second parameters converge.

[0159] where the total loss value L is:

[0160] L = λ cls · L cls + λ box · L box + λ giou · L giou + λ obj · L obj

[0161] where λ cls , λ box , λ giou , λ obj represent the weight coefficients of the classification loss L cls , the bounding box L1 loss L box , the GIoU loss L giou , and the object likelihood loss L obj , respectively;

[0162] The classification loss uses an improved Focal Loss to address the class imbalance problem in object detection. Its calculation method is:

[0163]

[0164] where N boxes is the number of matched positive samples; p ijis the predicted class probability (after Sigmoid); y ij is the target class (One-Hot encoding); a t is the class weight (default a t = 0.25); g is the adjustment factor (default g = 2) to reduce the weight of easy-to-classify samples; BCE is the binary cross-entropy loss.

[0165] The bounding box regression loss contains L1 loss and GIoU loss, which respectively calculate the coordinate error and overlap degree of the predicted box and the real box;

[0166] The target likelihood loss is used to measure the predicted object existence probability, which is calculated as:

[0167]

[0168] wherein: is the model predicted target probability; min_obj is the minimum target probability threshold (to prevent gradient disappearance). This loss encourages the model to output high confidence for matching positive samples, while suppressing the prediction of background regions.

[0169] The specific implementation process of the present application will be described below in combination with a specific application scenario.

[0170] As Figure 2 shown, the model is gradually expanded in classes using the weight freezing, gradual unfreezing of the backbone network, and incremental learning strategy through example replay. In the initial stage, the model is trained on labeled data containing the first 20 known classes, while the remaining 60 classes are treated as unknown to enhance the open world perception ability. After the end of this stage, the top 25 and bottom 25 samples are selected from each class based on the sample representativeness score, and a sample revisit pool of the first 20 classes is constructed to provide support for subsequent catastrophic forgetting mitigation.

[0171] In each subsequent stage, 20 new classes are introduced each time to gradually expand the known class set of the model. To ensure stable integration of new and old knowledge, the backbone network freezing strategy is adopted at the beginning of each stage, that is, the parameters of the visual backbone network are kept unchanged, and only the high-level feature integration and classification module is updated, to avoid drastic disturbance to the original feature distribution caused by new class training. Subsequently, the backbone parameters are gradually unfrozen in the later training stage, so that the model can adapt to the diversity of new classes while maintaining the unity of overall feature expression, thereby realizing the dynamic balance of "stability-plasticity".

[0172] In addition, at the end of each stage, a sample playback mechanism is introduced to re-incorporate representative samples reserved in the initial stage into the training process, allowing the model to review old class instances while learning new classes, thereby effectively suppressing catastrophic forgetting. This mechanism helps to improve cross-stage recognition accuracy and model generalization ability by maintaining the continuity of old class feature distribution.

[0173] After completing the training of each stage, the model identifies the current seen classes as known classes, and performs unknown class identification under the open world assumption for the still unseen classes. The above stage cycle is iterated until the learning of all 80 classes is completed. Finally, the model no longer outputs unknown class labels, forming a full-class closed-set recognition capability.

[0174] Based on the above embodiment, the embodiment of the present application provides an incremental learning device for open world target detection based on a large language model. Referring to Figure 3 The incremental learning device for open world target detection based on a large language model provided by the embodiment of the present application at least includes:

[0175] The acquisition unit 201 is configured to acquire a first open world target detection model, wherein the first open world target detection model is trained by first RGB image samples labeled with targets of a first class;

[0176] The establishment unit 202 is configured to establish a training set, wherein the training set includes a plurality of second RGB image samples, and each second RGB image sample is labeled with targets of the first class and a second class;

[0177] The first processing unit 203 is configured to process the first class and the second class by using a large language model to generate attribute features in text form of each class;

[0178] The second processing unit 204 is configured to process the second RGB image samples and the attribute features in text form of each class by using the first open world target detection model to obtain target box prediction values, target class prediction values, and unknown class target prediction values;

[0179] The determination unit 205 is configured to determine a first loss value based on the target box prediction values and target box true values, determine a second loss value based on the target class prediction values and target class label values, determine a third loss value based on the unknown class target prediction values and a preset threshold value, and determine a total loss value based on the first loss value, the second loss value, and the third loss value;

[0180] The incremental learning unit 206 is configured to update parameters of the first open world target detection model based on the total loss value, thereby obtaining a second open world target detection model.

[0181] It should be noted that the principle of solving the technical problem of the incremental learning device 200 for open world target detection based on a large language model provided in the embodiments of the present application is similar to the method provided in the embodiments of the present application, therefore, the implementation of the incremental learning device 200 for open world target detection based on a large language model provided in the embodiments of the present application can be referred to the implementation of the method provided in the embodiments of the present application, and the repeated parts will not be described here.

[0182] Based on the above embodiments, the embodiments of the present application further provide an electronic device, as shown in Figure 4 The electronic device 300 provided in the embodiments of the present application at least includes a processor 301, a memory 302 and a computer program stored in the memory 302 and executable on the processor 301, and the processor 301 implements the incremental learning method for open world target detection based on a large language model provided in the embodiments of the present application when executing the computer program.

[0183] The electronic device 300 provided in the embodiments of the present application can further include a bus 303 connected to different components (including the processor 301 and the memory 302). Among them, the bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, etc.

[0184] The memory 302 can include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022, and can further include a read-only memory (ROM) 3023.

[0185] The memory 302 can further include a program tool 3025 having a set of (at least one) program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules, and program data, each of these examples or some combination thereof can include the implementation of a network environment.

[0186] The electronic device 300 can also communicate with one or more external devices 304 such as a keyboard, a remote control, and / or any device (e.g., a handset, a computing pad, etc.) that enables a user to interact with the electronic device 300. Additionally, the electronic device 300 can communicate with one or more devices that enable Figure 4 The network adapter 306 communicates to the other modules of the electronic device 300 via the bus 303, as shown. It should be understood that although not shown, other hardware and / or software modules could be used in connection with the electronic device 300. These include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID subsystems, tape drives, and data archival storage subsystems, etc. Figure 4

[0187] It should be appreciated that the electronic device 300 shown in Figure 4 is only one example of an electronic device and should not be taken to limit the scope of functionality of various embodiments of the present application.

[0188] The embodiments of the present application further provide a computer readable storage medium, which stores computer instructions. The computer instructions are executed by a processor to implement the incremental learning method for open world object detection based on a large language model provided by the embodiments of the present application. Specifically, the executable program can be built-in or installed in the electronic device 300. Thus, the electronic device 300 can implement the incremental learning method for open world object detection based on a large language model provided by the embodiments of the present application by executing the built-in or installed executable program.

[0189] The incremental learning method for open world object detection based on a large language model provided by the embodiments of the present application can also be implemented as a program product. The program product includes program codes. When the program product can run on the electronic device 300, the program codes are used to make the electronic device 300 execute the incremental learning method for open world object detection based on a large language model provided by the embodiments of the present application.

[0190] ​The program product provided by the embodiments of the present application can adopt any combination of one or more readable media, wherein the readable media can be a readable signal medium or a readable storage medium, and the readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. Specifically, more specific examples (non-exhaustive list) of the readable storage medium include an electrical connection with one or more wires, a portable disc, a hard disk, a RAM, a ROM, an Erasable Programmable Read Only Memory (EPROM), an optical fiber, a portable Compact Disc Read-Only Memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0191] The program product provided by the embodiments of the present application can adopt a CD-ROM and include program codes, and can also run on a computing device. However, the program product provided by the embodiments of the present application is not limited thereto, and in the embodiments of the present application, the readable storage medium can be any tangible medium containing or storing programs, which can be used by or in combination with an instruction execution system, device or component.

[0192] It should be noted that although several units or sub-units of the apparatus are mentioned in the above detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into units embodied by multiple units.

[0193] In addition, although the operations of the method of the present application are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, combined into one step, and / or divided into multiple steps.

[0194] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit. Although the present application is described in detail with reference to the embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or equivalently replaced without departing from the spirit and scope of the technical solutions of the present application, and they should be covered in the scope of the claims of the present application.

Claims

1. An incremental learning method for open-world object detection based on a large language model, characterized in that, The method comprises the following steps: obtaining a first open-world target detection model, wherein the first open-world target detection model is trained by first RGB image samples labeled with targets of a first category; establishing a training set, wherein the training set comprises a plurality of second RGB image samples, and each second RGB image sample is labeled with targets of the first category and a second category; processing the first category and the second category by using a large language model to generate attribute features in text form of each category; processing the second RGB image samples and the attribute features in text form of each category by using the first open-world target detection model to obtain target box prediction values, target category prediction values, and unknown category target prediction values; determining a first loss value based on the target box prediction values and target box true values, determining a second loss value based on the target category prediction values and target category label values, and determining a third loss value based on the unknown category target prediction values and a preset threshold value; and determining a total loss value based on the first loss value, the second loss value, and the third loss value; updating parameters of the first open-world target detection model based on the total loss value to obtain a second open-world target detection model; the first open-world target detection model comprises a text backbone network, a visual feature extraction module, a similarity matrix calculation module, a category prediction module, a Transformer encoder-decoder, a bounding box regression branch, a known category classification branch, and an objectness detection head; processing the second RGB image samples and the attribute features in text form of each category by using the first open-world target detection model to obtain target box prediction values, target category prediction values, and unknown category target prediction values; the processing comprises the following steps: processing all attribute features in text form of each category by using the text backbone network to obtain a category semantic mapping matrix representing text features; processing the second RGB image samples by using the visual feature extraction module to obtain visual features; calculating a similarity matrix between the text features and the visual features by using the similarity matrix calculation module; processing the learnable category-feature mapping matrix and the similarity matrix by using the category prediction module to obtain a first category prediction result of the target of the second RGB image sample; processing the visual features by using the Transformer encoder-decoder to obtain context features of each query; processing the context features of each query by using the bounding box regression branch to obtain target box prediction values; processing the context features of each query by using the classification branch to obtain a second category prediction result of each target, calculating a sum of the first category prediction result and the second category prediction result of each target by using an adder to obtain a target category prediction result of the second RGB image sample; processing the context features of each query by using the objectness detection head to obtain unknown category target prediction values.

2. The method of claim 1, wherein, The attribute features in text form of each category comprise materials, properties, contexts, environments, sizes, shapes, colors, and textures.

3. The method of claim 1, wherein, The visual feature extraction module comprises a visual backbone network, a splicing unit, and a RELU activation function; The second RGB image sample is processed by using a visual feature extraction module to obtain visual features; the processing includes: The second RGB image sample is processed by using a visual backbone network to obtain visual features of multiple scales; The plurality of scale visual features are spliced by the splicing unit to obtain fused visual features ; The visual features are reduced in dimension by using a RELU activation function to obtain visual features with the same dimension as the text features : wherein, is a weight matrix, is an activation function, is of size is a batch size, is a dimension of the text features.

4. The method of claim 3, wherein, A similarity matrix calculation module is used to calculate a similarity matrix between the text features and the visual features; the processing includes: Computing visual features after extension to three dimensions : wherein is an all-one vector of length n, n being the total number of the first category and the second category; Computing text features after extending to three dimensions : wherein, is a vector of all ones of length b; is a category semantic mapping matrix of size , is a tensor outer product operation; Computing a similarity matrix The element in the i-th row and j-th column of the matrix is: wherein is the ith row of is the jth column of 5. The method of claim 4, wherein, A category prediction module is used to process the learnable category-feature mapping matrix and the similarity matrix to obtain a first category prediction result of the target of the second RGB image sample; the processing includes: The first category prediction result logits is: wherein, is a learnable class-feature mapping matrix.

6. The method of claim 5, wherein, Based on the total loss value, the parameters of the first open world target detection model are updated, including: determining parameters of the visual backbone network as first parameters, a learnable class-feature mapping matrix parameters of the transformer encoder-decoder, the bounding box regression branch, the known class classification branch, and the objectness detection head as second parameters The first parameter is frozen, and the second parameter is updated based on the total loss value until the second parameter converges; The first parameter and the second parameter are updated based on the total loss value until the first parameter and the second parameter converge.

7. An incremental learning device for open-world object detection based on a large language model, characterized in that, The processing includes: An acquisition unit is configured to acquire a first open world target detection model, wherein the first open world target detection model is trained by a first RGB image sample labeled with a target of a first category; An establishment unit is configured to establish a training set, wherein the training set includes a plurality of second RGB image samples, and each second RGB image sample is labeled with a target of a first category and a target of a second category; A first processing unit is configured to process the first category and the second category by using a large language model to generate attribute features in a text form of each category; A second processing unit is configured to process the second RGB image sample and the attribute features in a text form of each category by using the first open world target detection model to obtain a target box prediction value, a target category prediction value, and an unknown category target prediction value; A determination unit is configured to determine a first loss value based on the target box prediction value and a target box true value, determine a second loss value based on the target category prediction value and a target category label value, determine a third loss value based on the unknown category target prediction value and a preset threshold value, and determine a total loss value based on the first loss value, the second loss value, and the third loss value; An incremental learning unit is configured to update parameters of the first open world target detection model based on the total loss value, thereby obtaining a second open world target detection model The first open world target detection model includes a text backbone network, a visual feature extraction module, a similarity matrix calculation module, a category prediction module, a Transformer encoder-decoder, a bounding box regression branch, a known category classification branch, and an objectness detection head; The second processing unit is specifically configured to: Process the attribute features in a text form of all categories by using the text backbone network to obtain a category semantic mapping matrix representing the text features; Process the second RGB image sample by using the visual feature extraction module to obtain visual features; Calculate a similarity matrix between the text features and the visual features by using the similarity matrix calculation module; Process the learnable category-feature mapping matrix and the similarity matrix by using the category prediction module to obtain a first category prediction result of the target of the second RGB image sample; Process the visual features by using the Transformer encoder-decoder to obtain a context feature of each query; The context feature of each query is processed by using the bounding box regression branch to obtain a target box prediction value; The context feature of each query is processed by using the classification branch to obtain a second category prediction result of each target, The sum of the first category prediction result and the second category prediction result of each target is calculated by using the adder to obtain a target category prediction result of the second RGB image sample; The context feature of each query is processed by using the objectness detection head to obtain an unknown category target prediction value.

8. An electronic device, comprising: The computer readable storage medium stores computer instructions, and the computer instructions are executed by the processor to implement the method in any one of claims 1-6. The computer readable storage medium stores computer instructions, and the computer instructions are executed by the processor to implement the method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, ​

Images