A target detection method and device based on incremental learning perception model

Abstract

The present application provides a target detection method and device based on an incremental learning perception model, wherein the method includes: obtaining an RGB image at the current moment captured by a camera on the vehicle; processing the RGB image at the current moment using a region proposal network to obtain multiple proposal frames; processing the RGB image at the current moment using a Transformer encoding layer to obtain a deep semantic feature map; fusing each proposal frame, the deep semantic feature map, and the prototype vector of each target type to obtain a fused feature of each proposal frame; wherein the prototype vector of each target type is obtained by incremental learning based on the target type in the current scene of the vehicle; processing the fused features of each proposal frame using a region projection layer to obtain a target positioning result; and processing the fused features of each proposal frame using a classification layer to obtain a target classification result. The present application improves the target detection accuracy of RGB images in unknown scenes.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a target detection method and device based on an incremental learning perception model. Background Art

[0002] When handling updated target information in dynamic environments, autonomous vehicles often rely on visual perception models that rely on large amounts of annotated data. However, practical applications present several challenges: In certain sensitive areas, limited prior information makes training challenging, leading to uncertainty in perception model training; retraining or fine-tuning the entire model is time-consuming and inefficient; and, when training on new tasks, performance on old tasks often significantly degrades, resulting in a common flaw: catastrophic forgetting. These issues result in low detection accuracy and limited recognition capabilities for visual perception models in complex, dynamic scenes, making them unable to meet practical needs. Summary of the Invention

[0003] In view of this, the present application provides a target detection method and device based on an incremental learning perception model to solve the above technical problems.

[0004] In a first aspect, embodiments of the present application provide a target detection method based on an incrementally learned perception model, which is applied to a vehicle, including:

[0005] Get the RGB image of the current moment captured by the camera on the vehicle;

[0006] Use the pre-trained region proposal network to process the current RGB image and obtain multiple proposal boxes;

[0007] Use the pre-trained Transformer encoding layer to process the current RGB image to obtain a deep semantic feature map;

[0008] The proposal boxes, deep semantic feature maps, and prototype vectors of each target type are fused to obtain the fused features of each proposal box; wherein the prototype vectors of each target type are incrementally learned based on the target types in the current scene of the vehicle;

[0009] The regional projection layer is used to process the fusion features of each proposal frame to obtain the target positioning result;

[0010] The classification layer is used to process the fusion features of each proposal box to obtain the target classification result.

[0011] In one possible implementation, the method further includes:

[0012] Receive the prototype file that records the prototype vectors of each target type sent by the operator, and update the original prototype vectors of each target type stored on the vehicle side.

[0013] In one possible implementation, the method further includes:

[0014] The target detection result of the current RGB image collected by the vehicle end is sent to the operation end. The target detection result includes: target positioning result and target classification result.

[0015] In one possible implementation, the prototype file is generated by the operating terminal, and the generation steps include:

[0016] Get k RGB images containing the new target type from all RGB images at the current moment and before;

[0017] Use the SAM image segmentation algorithm to process k RGB images containing new target types and obtain target masks of k new target types;

[0018] Use the prototype network to process the RGB images of k new target types and their corresponding target masks to obtain the prototype vectors of the new target types;

[0019] Add the new target type and its prototype vector to the prototype file and send the updated prototype file to the ego vehicle.

[0020] In one possible implementation, each proposal box, deep semantic feature map and prototype vector of each object type are fused to obtain the fused features of each proposal box, including:

[0021] Align the deep semantic feature map with each proposal box to obtain the proposed features of each proposal box;

[0022] Perform a dot product operation on the proposed features of each proposal box and the prototype vector of each target type to obtain the fused features of each proposal box.

[0023] In one possible implementation, the method further includes a step of jointly training the region proposal network, the Transformer encoding layer, and the prototype network.

[0024] In one possible implementation, the steps for jointly training the region proposal network, the Transformer encoding layer, and the prototype network include:

[0025] Obtain a training set, wherein the training set includes multiple RGB image samples labeled with real target frames and target categories;

[0026] Use the region proposal network to process the RGB image samples and obtain multiple proposal box samples;

[0027] Use the pre-trained Transformer encoding layer to process the RGB image samples to obtain deep semantic feature map samples;

[0028] Use the SAM image segmentation network to process the RGB image samples and obtain multiple target masks;

[0029] Using a prototype network to process multiple RGB images and their corresponding object masks to obtain prototype vectors of basic object types; wherein the basic object types are object categories in the RGB image samples;

[0030] Fuse each proposal box sample, deep semantic feature map sample and prototype vector of the basic target type to obtain the fusion feature of each proposal box sample;

[0031] The region projection layer is used to process the fusion features of each proposal box sample to obtain multiple predicted target boxes;

[0032] Using multiple predicted target frames and true target frames, calculate the first loss value;

[0033] Use the classification layer to process the fusion features of each proposal box sample to obtain the predicted classification result of the target;

[0034] Calculate the second loss value using the predicted classification result of the target and the actual target category;

[0035] The first loss value and the second loss value are used to update the parameters of the region proposal network, the Transformer encoding layer, and the prototype network.

[0036] In a second aspect, an embodiment of the present application provides a target detection device based on an incremental learning perception model, which is applied to a vehicle, including:

[0037] An acquisition unit, used to acquire the RGB image at the current moment captured by the camera on the vehicle;

[0038] The first processing unit is used to process the RGB image at the current moment using a pre-trained region proposal network to obtain multiple proposal boxes;

[0039] The second processing unit is used to process the RGB image at the current moment using the pre-trained Transformer encoding layer to obtain a deep semantic feature map;

[0040] A fusion unit is used to fuse each proposal frame, the deep semantic feature map, and the prototype vector of each target type to obtain a fused feature of each proposal frame; wherein the prototype vector of each target type is obtained by incremental learning based on the target type in the current scene of the vehicle;

[0041] The positioning unit is used to process the fusion features of each proposal box using the region projection layer to obtain the target positioning result;

[0042] The classification unit is used to process the fusion features of each proposal box using the classification layer to obtain the target classification result.

[0043] In a third aspect, an embodiment of the present application provides an electronic device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the target detection method based on the incremental learning perception model of the embodiment of the present application is implemented.

[0044] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the target detection method based on the incremental learning perception model of the embodiment of the present application is implemented.

[0045] This application improves the object detection accuracy of RGB images collected by the vehicle in unknown scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 A flowchart of a target detection method based on an incremental learning perception model provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of an incremental perception system provided in an embodiment of the present application;

[0049] Figure 3 A functional structure diagram of a target detection device based on an incremental learning perception model provided in an embodiment of the present application;

[0050] Figure 4 This is a functional structure diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Generally, the components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0052] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for protection, but merely represents selected embodiments of the present application. All other embodiments obtained by persons of ordinary skill in the art based on the embodiments in the present application without creative work are within the scope of protection of the present application.

[0053] First, a brief introduction to the design concept of the embodiments of the present application is given.

[0054] Existing visual perception model learning methods are mostly targeted at scenarios with extensive prior information or training samples. Effective perception model learning strategies for small amounts of prior target information are lacking, making them inadequate for the demands of real-time perception and recognition in complex environments. Data scarcity and imbalance pose significant challenges to model training and optimization, necessitating the development of effective methods to enhance model detection performance. Therefore, how to incrementally learn models based on minimal prior target information, while maintaining performance and avoiding catastrophic forgetting, remains a pressing technical challenge.

[0055] To this end, this application proposes an object detection method based on an incremental learning perception model. First, real-time data is acquired, including the image to be detected and a small number of prompt images, where the small number of prompt images refers to keyframe images containing the new target category. The image to be detected is considered the query image, and the small number of prompt images are considered support images. Then, an initial proposal box is generated for the query image using a region proposal network (RPN). Next, a Transformer encoding layer is used to extract feature maps from the query and support images. Prototype feature vectors for the new category are generated by calculating the average features of the support images. These prototype feature vectors represent the central features of each category. The query image feature map, the prototype feature vector, and the proposal box are then fused, ensuring that the fused features contain more semantic and spatial information. Subsequently, the region projection layer uses the fused features to convert the initial proposal into precise bounding box coordinates, ensuring accurate target localization. Finally, in the classification layer, the target is classified based on the fused features, generating a class label and confidence score for the target. Through these steps, the incremental detection algorithm can achieve efficient and accurate target detection and classification in complex environments.

[0056] The advantages of this application are:

[0057] 1. The SAM image segmentation algorithm, implemented through human-computer interaction, enables real-time acquisition of mask and text annotation information for new objects in an image. It performs exceptionally well in the face of data scarcity and category updates, providing accurate annotation data for the model's learning process. During testing, image segmentation accuracy and speed were measured through on-site testing. The results showed that image segmentation accuracy exceeded 95% for both large and small objects, and the segmentation speed was less than 1 second, fully meeting the requirements for real-time detection performance.

[0058] 2. The prototype feature extraction algorithm uses metric learning to generate representative prototype features by learning the feature distribution of a small number of prior samples. This algorithm not only performs well in the face of data scarcity and category updates, but also captures commonalities between tasks through a multi-task training process using meta-learning, further enhancing the model's generalization capabilities. During testing, the speed of prototype feature file generation was tested. The actual test time for this content was less than 1 second when tested on small sample image data, fully meeting the requirements for real-time detection performance.

[0059] 3. This algorithm uses an incremental learning algorithm for the perception model with a small amount of prior target information. Building on a pre-trained model, it employs a meta-learning-based prototype network approach to achieve incremental learning of the model with few samples. By introducing the prototype module into the base model, detection performance is rapidly improved without further fine-tuning the model, while maintaining a high recall rate and avoiding the catastrophic forgetting problem associated with full fine-tuning. This ensures the system's ability to continuously learn and adapt to new environments, enabling it to quickly respond to the emergence of new targets and data.

[0060] After introducing the application scenarios and design concepts of the embodiments of the present application, the technical solutions provided by the embodiments of the present application are described below.

[0061] like Figure 1 As shown, the present application implements a target detection method based on an incremental learning perception model, which is applied to the vehicle side, including:

[0062] Step 101: Obtain the RGB image at the current moment captured by the camera on the vehicle;

[0063] Step 102: Use the pre-trained region proposal network to process the current RGB image to obtain multiple proposal boxes;

[0064] For example, the Region Proposal Network (RPN) can generate initial proposal boxes for targets in RGB images. Specifically, the RGB image is regarded as a query image, and the RPN is responsible for generating multiple candidate regions in the image, which may contain target objects. The RPN generates a series of anchor boxes on the image through a sliding window mechanism. These anchor boxes have different scales and aspect ratios to accommodate various possible target sizes and shapes. For each anchor box, the RPN calculates the probability that it contains the target and adjusts it to more accurately locate the target.

[0065] Step 103: Use the pre-trained Transformer encoding layer to process the RGB image at the current moment to obtain a deep semantic feature map;

[0066] Exemplarily, the Transformer Encoder Layer is used to extract feature maps of RGB images. The Transformer Encoder Layer can capture the deep semantic features of the image and extract feature maps of the query image. The feature maps play an important role in the subsequent fusion processing. At the same time, the same Transformer Encoder Layer is used to extract feature maps of the support images, and the feature vectors are obtained through the feature maps, which are used to calculate the prototype feature vectors of the new categories. Here, the backbone network of the Transformer-based pre-trained Dinov2 ViT model is used as the Transformer Encoder Layer. Specifically, the Transformer Encoder Layer divides the RGB image into blocks and calculates the relationship between each image block through a multi-head self-attention mechanism to generate high-dimensional feature representations. These feature representations not only contain the global information of the image, but also retain fine-grained local features, thereby improving the expressive power of the model.

[0067] Step 104: Fusing each proposal frame, the deep semantic feature map, and the prototype vector of each object type to obtain a fused feature of each proposal frame; wherein the prototype vector of each object type is obtained by incremental learning based on the object types in the current scene of the vehicle;

[0068] Step 105: Use the region projection layer to process the fusion features of each proposal box to obtain the target positioning result;

[0069] Exemplary, the Region Projection Layer is used to further correct the proposal box proposed by the region proposal network to generate accurate bounding box coordinates. For the target object in the query image, the detection box generated by the region proposal network will have a certain deviation, which affects the precise positioning of the target. By expanding the proposal box, and then using the rich semantic information and spatial information contained in the fusion features, the initial proposal is diffused to the target area at the pixel level in the form of a heat map conversion, and finally the target area is projected to the accurate bounding box through the learnable spatial integration layer. Specifically, the learnable spatial integration layer calculates the center position and size of the bounding box by weighted averaging the probability distribution of the target area. This method avoids the gradient explosion and vanishing problems that may exist in traditional regression methods, thereby achieving more accurate target positioning.

[0070] Step 106: Use the classification layer to process the fusion features of each proposal box to obtain the target classification result.

[0071] Exemplary, the classification layer is used to classify targets based on the fused features obtained by fusion. Since in this incremental detection system, it is necessary to continuously introduce prototype feature vectors of new categories to achieve the incremental effect, a multi-layer binary network is used in the classification layer to calculate the classification score of each category relative to other categories. These scores are converted into probability distributions through the softmax function, indicating the probability that the proposal box belongs to each category. Finally, the system classifies each proposal box according to these probability distributions. In this way, the system can accurately identify and classify targets of new categories under few-sample conditions, achieving efficient target detection and classification.

[0072] Exemplary target types include: pedestrians, bicycles, electric vehicles, trucks, trailers, cars, and animals.

[0073] In some embodiments, the method further comprises:

[0074] Receive the prototype file that records the prototype vectors of each target type sent by the operator, and update the original prototype vectors of each target type stored on the vehicle side.

[0075] In some embodiments, the method further comprises:

[0076] The target detection result of the current RGB image collected by the vehicle end is sent to the operation end. The target detection result includes: target positioning result and target classification result.

[0077] In some embodiments, the prototype file is generated by the operator, and the generation steps include:

[0078] Get k RGB images containing the new target type from all RGB images at the current moment and before;

[0079] Use the SAM image segmentation algorithm to process k RGB images containing new target types and obtain target masks of k new target types;

[0080] Use the prototype network to process the RGB images of k new target types and their corresponding target masks to obtain the prototype vectors of the new target types;

[0081] Add the new target type and its prototype vector to the prototype file and send the updated prototype file to the ego vehicle.

[0082] In this embodiment, a small amount of image data containing the new target is collected as a prompt image for the perception model. The SAM image segmentation algorithm is used to perform mask extraction on the new target in the image through human-computer interaction. The category text label is combined to form the annotation data of the new target object, providing input data for the subsequent extraction of prototype features.

[0083] SAM can segment any object based on command information and image data. Drawing on the prompt strategy from the field of natural language processing (NLP), it provides prompts for image segmentation tasks, enabling rapid segmentation of any object. Prompts can be foreground / background points, rough boxes, masks, or arbitrary text. The model takes the original image and prompt information as input, and aims to output a "valid" segmentation. Validity means that the model can output at least one mask of the object even when the prompt is ambiguous.

[0084] The SAM model consists of three core components: the Image Encoder, the Prompt Encoder, and the Mask Decoder. The image is encoded by the Image Encoder, and the prompt is encoded by the Prompt Encoder. The two embeddings are then passed through a lightweight Mask Decoder to generate fused features. The Encoder uses an existing model, while the Decoder uses the Transformer.

[0085] Component 1: Image Encoder

[0086] The image encoder maps the image to the feature space. Essentially, this encoder can be any network structure. Here, we use a fine-tuned ViT model, or a traditional convolutional structure.

[0087] Based on the vit_b version of the ViT model, the input original image is padded into a square according to the maximum side length, and then the size is resized to 3x1024x1024. The process through the ViT structure is as follows:

[0088] Patch Embedding: The input image passes through a convolution base, dividing the image into 16x16 patches with a stride of 16. This reduces the size of the feature map by a factor of 16, while also mapping the number of channels from 3 to 768.

[0089] Positional Embedding. After patch embedding, the output tokens need to be positionally encoded to preserve the spatial information of the image. Positional encoding can be understood as a map with the same number of rows as the input sequence. Each row represents a vector with the same dimension as the input sequence tokens. The positional encoding operation is summed, so the dimension remains unchanged.

[0090] Transformer Encoder. The feature map passes through 16 Transformer blocks, 12 of which use an attention mechanism based on Window Partition (dividing the feature map into 14x14 windows for local attention) to process local information. The remaining four blocks are global attention modules, which are interspersed between Window Partition modules to capture the global context of the image.

[0091] Partition operation. In a non-global attention block, in order to adapt to the 14x14 window size, the input feature map needs to be padded and split. The specific process is as follows:

[0092] ① Input feature map: The initial size of the input feature map is 1x64x64x768.

[0093] ② Determine the minimum divisible size: The window size is 14*14. To find the minimum feature map size that is divisible by 14, we need to find the smallest number greater than or equal to 64 and divisible by 14 for both width and height. These numbers are 70(64+6) and 70(64+6), respectively. Therefore, the minimum divisible feature map size is 1x70x70x768.

[0094] ③Padding: To expand the feature map size from 64x64 to 70x70, a 6x6 area is added to the lower right corner, because 70-64=6. This padding ensures that the window can be correctly divided at the edge of the feature map.

[0095] ④ Split feature map: Split the features after padding Figure 1 The feature map is split into 14x14 windows of size x70x70x768. Since 70 / 14=5, the feature map can be split into 5x5 windows of size 14x14, for a total of 5x5=25 windows. Each window has a size of 14x14x768.

[0096] Unpartition operation. In the non-global attention block, the features output by the attention layer are Figure 1 The conversion of x70x70x768 into a 1x64x64x768 feature map is actually done by taking out the 1x64x64x768 part in the upper left corner from the 1x70x70x768 feature map through the slicing operation x=x[:1,:64,:64,:].

[0097] Neck Convolution: Through two layers of convolution (Neck), the number of channels is reduced to 256, and the final ImageEmbedding is generated.

[0098] Component 2: Prompt Encoder

[0099] Prompt Encoder maps the input prompt information to the corresponding feature space. The channel of its feature is consistent with the channel of image embedding. The input data type of this component can be point, box, mask or text. The present invention uses box data as prompt input to generate the sparse_embedding required by the subsequent MaskEncoder.

[0100] The bounding box generally has two points, and the encoding process is as follows:

[0101] Shift the input marker box corner coordinates from the top right corner of the pixel to the center of the pixel.

[0102] The adjusted bounding box corner coordinates and the size of the input image are used as input, and a high-dimensional embedding representation corner_Embedding is obtained using a random Gaussian matrix and sine and cosine encoding.

[0103] After adding corner_Embedding to a learnable embedding vector, we get sparseembedding as the output of Prompt Encoder.

[0104] Component 3: Mask Decoder

[0105] The role of Mask Encoder is to use the output of Image Encoder and Prompt Encoder as information source, and predict the segmentation mask after deep fusion.

[0106] The key idea of ​​Mask Encoder lies in the Transformer's self-attention mechanism and cross-attention mechanism. Specifically, in a Transformer block, one self-attention and three cross-attention are used to repeatedly fuse and update the previously obtained input information image_embedding and sparse_embedding. At the same time, in order to prevent position information from being lost, position encoding is introduced multiple times to ensure that accurate segmentation results are finally obtained.

[0107] In this embodiment, a picture containing a new target is used as a prompt picture. The SAM image segmentation algorithm is used to extract the target mask by manually inputting box data through manual interaction. The mask is then combined with the category text label to form a small amount of annotation data of the new target object in the prompt image, providing a data basis for the subsequent extraction of the prototype of the new target.

[0108] The Prototype Network's workflow involves generating a prototype feature vector for each base category during the learning phase. This vector is calculated by taking the mean of the feature vectors of all samples in that category and represents the central feature of that category. During the detection phase, upon receiving a sample from a new category, the model first extracts its feature vector using the pretrained model. The prototype feature vector for the new category sample is then calculated in the same manner. This prototype feature vector is then combined with the prototype feature vectors of the base category for subsequent target prediction.

[0109] In the small sample classification task, suppose there are N labeled samples S = (x1, y1), ..., (x N ,y N ),in is the sample feature vector of D dimensions, y∈1,...,K is the corresponding label. K Represents the set of samples of the kth class. The prototype network calculates the M-dimensional prototype feature vector of each class The calculated function is Where φ is a learnable parameter. The prototype feature vector c k The calculation formula is as follows:

[0110]

[0111] The Prototype network method under the meta-learning framework solves the few-shot learning problem through metric learning, which can significantly improve the model's adaptability to new tasks.

[0112] In some embodiments, each proposal box, the deep semantic feature map, and the prototype vector of each target type are fused to obtain a fused feature of each proposal box, including:

[0113] Align the deep semantic feature map with each proposal box to obtain the proposed features of each proposal box;

[0114] Perform a dot product operation on the proposed features of each proposal box and the prototype vector of each target type to obtain the fused features of each proposal box.

[0115] In some embodiments, the method further includes: a step of jointly training the region proposal network, the Transformer encoding layer, and the prototype network; specifically including:

[0116] Obtain a training set, wherein the training set includes multiple RGB image samples labeled with real target frames and target categories;

[0117] Use the region proposal network to process the RGB image samples and obtain multiple proposal box samples;

[0118] Use the pre-trained Transformer encoding layer to process the RGB image samples to obtain deep semantic feature map samples;

[0119] Use the SAM image segmentation network to process the RGB image samples and obtain multiple target masks;

[0120] Using a prototype network to process multiple RGB images and their corresponding object masks to obtain prototype vectors of basic object types; wherein the basic object types are object categories in the RGB image samples;

[0121] Fuse each proposal box sample, deep semantic feature map sample and prototype vector of the basic target type to obtain the fusion feature of each proposal box sample;

[0122] The region projection layer is used to process the fusion features of each proposal box sample to obtain multiple predicted target boxes;

[0123] Using multiple predicted target frames and true target frames, calculate the first loss value;

[0124] Use the classification layer to process the fusion features of each proposal box sample to obtain the predicted classification result of the target;

[0125] Calculate the second loss value using the predicted classification result of the target and the actual target category;

[0126] The first loss value and the second loss value are used to update the parameters of the region proposal network, the Transformer encoding layer, and the prototype network.

[0127] The specific implementation process of this application is described below in conjunction with a specific application scenario.

[0128] like Figure 2 As shown, the incremental perception system includes the operation end, the server end and the vehicle end, among which,

[0129] The vehicle uses a camera to capture image data from the real world, encompassing complex scenarios and environmental conditions. This image data is processed by the vehicle's detection algorithm, identifying objects within the image and generating detection results. These detection results are then transmitted to the server via the network. The server then forwards these images to the operator for further processing. Simultaneously, the vehicle receives the prototype file from the server and restarts the algorithm to adapt to the new object category.

[0130] The operator receives vehicle-side images from the server and displays them on a QT interactive interface. Using a segmentation algorithm, the operator selects objects in the image and generates a mask. Using this mask and image information, the operator generates a new prototype file and sends it back to the server via the network. The server then forwards the file to the vehicle, which receives the new prototype file and restarts the detection algorithm to ensure the system can adapt to dynamic environmental changes in real time.

[0131] The server is responsible for data transfer and transmission throughout the entire process. It first receives the detection result images transmitted by the vehicle and forwards them to the operator. Simultaneously, the server receives the prototype files generated by the operator and forwards them to the vehicle. To ensure the new algorithm and prototype files are applied, the server sends a restart command to the vehicle after confirming the successful file transfer.

[0132] The specific workflow is as follows:

[0133] The onboard camera captures real-time images and performs preliminary processing using the vehicle-side detection algorithm to generate detection results. These detection results are transmitted via the network to the server, which receives and stores the image data and then forwards it to the operator. The operator displays the received image on the QT interactive interface and uses the segmentation algorithm to generate a mask for the target using the tooltip box selected on the image. A new prototype file is generated based on the mask and text category annotations. The operator sends the generated prototype file back to the server via the network, which receives and forwards it to the vehicle. After confirming the successful file transfer, the server sends a restart command to the vehicle. After receiving the new prototype file, the vehicle restarts the detection algorithm and uses the model with the updated prototype file to perform target detection to adapt to the new target category.

[0134] The above steps utilize the SAM segmentation algorithm to obtain a mask for the new target in the image as annotation information. Building on the pre-trained model, a fast, training-free fine-tuning method is used to develop an incremental perception system that adapts to the recognition of new targets. This improves the system's ability to recognize newly added targets and ensures effective target detection and recognition even with limited data. This system achieves efficient and accurate target detection and classification in complex real-world environments. The incremental detection algorithm combines real-time image acquisition, dynamic target detection, and model updates to ensure the system can rapidly adapt to dynamic changes in the real world, providing solid technical support for modern computer vision detection and recognition.

[0135] Based on the above embodiments, the present application provides a target detection device based on a perception model of incremental learning. Figure 3 As shown, the target detection device 200 based on the incremental learning perception model provided in the embodiment of the present application includes at least:

[0136] An acquisition unit 201 is used to acquire an RGB image at the current moment captured by a camera on the vehicle;

[0137] A first processing unit 202 is configured to process the current RGB image using a pre-trained region proposal network to obtain a plurality of proposal boxes;

[0138] The second processing unit 203 is used to process the RGB image at the current moment using the pre-trained Transformer encoding layer to obtain a deep semantic feature map;

[0139] A fusion unit 204 is configured to fuse each proposal frame, the deep semantic feature map, and the prototype vector of each object type to obtain a fused feature of each proposal frame; wherein the prototype vector of each object type is obtained by incremental learning based on the object types in the current scene of the vehicle;

[0140] The positioning unit 205 is used to process the fusion features of each proposal box using the region projection layer to obtain the target positioning result;

[0141] The classification unit 206 is used to process the fusion features of each proposal box using the classification layer to obtain a target classification result.

[0142] It should be noted that the principle of solving technical problems by the target detection device 200 based on the incremental learning perception model provided in the embodiment of the present application is similar to the method provided in the embodiment of the present application. Therefore, the implementation of the target detection device 200 based on the incremental learning perception model provided in the embodiment of the present application can refer to the implementation of the method provided in the embodiment of the present application, and the repeated parts will not be repeated.

[0143] Based on the above embodiments, the present application also provides an electronic device, referring to Figure 4 As shown, the electronic device 300 provided in the embodiment of the present application includes at least: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the target detection method based on the incremental learning perception model provided in the embodiment of the present application is implemented.

[0144] The electronic device 300 provided in the embodiment of the present application may further include a bus 303 connecting different components (including the processor 301 and the memory 302). The bus 303 represents one or more of several types of bus structures, including a memory bus, a peripheral bus, a local bus, and the like.

[0145] The memory 302 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 3021 and / or a cache memory 3022 , and may further include a read-only memory (ROM) 3023 .

[0146] The memory 302 may also include a program tool 3025 having a set (at least one) of program modules 3024, including but not limited to: an operating subsystem, one or more application programs, other program modules and program data, each of which or some combination may include an implementation of a network environment.

[0147] The electronic device 300 may also communicate with one or more external devices 304 (e.g., keyboards, remote controls, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 300 (e.g., mobile phones, computers, etc.), and / or any device that enables the electronic device 300 to communicate with one or more other electronic devices 300 (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 305. Furthermore, the electronic device 300 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 306. Figure 4 As shown, the network adapter 306 communicates with other modules of the electronic device 300 via the bus 303. Figure 4 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 300, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, disk arrays (Redundant Arrays of Independent Disks, RAID) subsystems, tape drives, and data backup storage subsystems.

[0148] It should be noted that Figure 4 The electronic device 300 shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0149] The present application also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the object detection method based on the incremental learning perception model provided in the present application. Specifically, the executable program can be built into or installed in the electronic device 300. In this way, the electronic device 300 can implement the object detection method based on the incremental learning perception model provided in the present application by executing the built-in or installed executable program.

[0150] The target detection method based on the incremental learning perception model provided in the embodiment of the present application can also be implemented as a program product, which includes a program code. When the program product can be run on the electronic device 300, the program code is used to enable the electronic device 300 to execute the target detection method based on the incremental learning perception model provided in the embodiment of the present application.

[0151] The program product provided in the embodiments of the present application may adopt any combination of one or more readable media, wherein the readable medium may be a readable signal medium or a readable storage medium, and the readable storage medium may be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination of the above. Specifically, more specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, RAM, ROM, Erasable Programmable Read Only Memory (EPROM), optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0152] The program product provided in the embodiments of the present application may be a CD-ROM and include program code, and may also be run on a computing device. However, the program product provided in the embodiments of the present application is not limited thereto. In the embodiments of the present application, the readable storage medium may be any tangible medium containing or storing a program, and the program may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0153] It should be noted that although several units or subunits of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, depending on the embodiment of the application, the features and functions of two or more units described above can be embodied in a single unit. Conversely, the features and functions of a single unit described above can be further divided and embodied by multiple units.

[0154] Furthermore, although the operations of the method of the present application are described in a particular order in the accompanying drawings, this does not require or imply that the operations must be performed in this particular order, or that all illustrated operations must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0155] Finally, it should be noted that the above embodiments are intended only to illustrate the technical solutions of this application and are not intended to limit the scope of the present invention. Although this application has been described in detail with reference to the embodiments, it should be understood by those skilled in the art that modifications or equivalent substitutions to the technical solutions of this application do not depart from the spirit and scope of the technical solutions of this application and should be encompassed by the claims of this application.

Claims

1. A target detection method based on an incremental learning perception model, applied to the vehicle, characterized in that: include: Get the RGB image of the current moment captured by the camera on the vehicle; Use the pre-trained region proposal network to process the current RGB image and obtain multiple proposal boxes; Use the pre-trained Transformer encoding layer to process the current RGB image to obtain a deep semantic feature map; The proposal boxes, deep semantic feature maps, and prototype vectors of each target type are fused to obtain the fused features of each proposal box; wherein the prototype vectors of each target type are incrementally learned based on the target types in the current scene of the vehicle; The regional projection layer is used to process the fusion features of each proposal frame to obtain the target positioning result; Use the classification layer to process the fusion features of each proposal frame to obtain the target classification result; The method further comprises: Receive the prototype file that records the prototype vectors of each target type sent by the operator, and update the original prototype vectors of each target type stored on the vehicle side; The method further comprises: Send the target detection result of the current RGB image collected by the vehicle end to the operation end, the target detection result includes: target positioning result and target classification result; The prototype file is generated by the operation end, and the generation steps include: Get k RGB images containing the new target type from all RGB images at the current moment and before; Use the SAM image segmentation algorithm to process k RGB images containing new target types and obtain target masks of k new target types; Use the prototype network to process the RGB images of k new target types and their corresponding target masks to obtain the prototype vectors of the new target types; Add the new target type and its prototype vector to the prototype file and send the updated prototype file to the ego vehicle.

2. The target detection method based on the incremental learning perception model according to claim 1, characterized in that Each proposal box, deep semantic feature map and prototype vector of each target type are fused to obtain the fusion features of each proposal box, including: Align the deep semantic feature map with each proposal box to obtain the proposed features of each proposal box; Perform a dot product operation on the proposed features of each proposal box and the prototype vector of each target type to obtain the fused features of each proposal box.

3. The target detection method based on the incremental learning perception model according to claim 2, characterized in that: The method further comprises: Steps for jointly training the region proposal network, Transformer encoding layer, and prototypical network.

4. The target detection method based on the incremental learning perception model according to claim 3 is characterized in that The steps for jointly training the region proposal network, Transformer encoding layer, and prototype network include: Obtain a training set, wherein the training set includes multiple RGB image samples labeled with real target frames and target categories; Use the region proposal network to process the RGB image samples and obtain multiple proposal box samples; Use the pre-trained Transformer encoding layer to process the RGB image samples to obtain deep semantic feature map samples; Use the SAM image segmentation network to process the RGB image samples and obtain multiple target masks; Using a prototype network to process multiple RGB images and their corresponding object masks to obtain prototype vectors of basic object types; wherein the basic object types are object categories in the RGB image samples; Fuse each proposal box sample, deep semantic feature map sample and prototype vector of the basic target type to obtain the fusion feature of each proposal box sample; The region projection layer is used to process the fusion features of each proposal box sample to obtain multiple predicted target boxes; Using multiple predicted target frames and true target frames, calculate the first loss value; Use the classification layer to process the fusion features of each proposal box sample to obtain the predicted classification result of the target; Calculate the second loss value using the predicted classification result of the target and the actual target category; The first loss value and the second loss value are used to update the parameters of the region proposal network, the Transformer encoding layer, and the prototype network.

5. A target detection device based on an incremental learning perception model, applied to a vehicle, characterized in that: include: An acquisition unit, used to acquire the RGB image at the current moment captured by the camera on the vehicle; The first processing unit is used to process the RGB image at the current moment using a pre-trained region proposal network to obtain multiple proposal boxes; The second processing unit is used to process the RGB image at the current moment using the pre-trained Transformer encoding layer to obtain a deep semantic feature map; A fusion unit is used to fuse each proposal frame, the deep semantic feature map, and the prototype vector of each target type to obtain a fused feature of each proposal frame; wherein the prototype vector of each target type is obtained by incremental learning based on the target type in the current scene of the vehicle; The positioning unit is used to process the fusion features of each proposal box using the region projection layer to obtain the target positioning result; The classification unit is used to process the fusion features of each proposal box using the classification layer to obtain the target classification result; The device further includes an updating unit, specifically configured to: Receive the prototype file that records the prototype vectors of each target type sent by the operator, and update the original prototype vectors of each target type stored on the vehicle side; The device further includes a sending unit, specifically configured to: Send the target detection result of the current RGB image collected by the vehicle end to the operation end, the target detection result includes: target positioning result and target classification result; The prototype file is generated by the operation end, and the generation steps include: Get k RGB images containing the new target type from all RGB images at the current moment and before; Use the SAM image segmentation algorithm to process k RGB images containing new target types and obtain target masks of k new target types; Use the prototype network to process the RGB images of k new target types and their corresponding target masks to obtain the prototype vectors of the new target types; Add the new target type and its prototype vector to the prototype file and send the updated prototype file to the ego vehicle.

6. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 4 when executing the computer program.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and when the computer instructions are executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images