Long-tail target detection and fine-grained large model pre-training method, storage medium and electronic equipment

Abstract

The invention discloses a long-tail target detection and fine-grained large model pre-training method based on dynamic heavy balance comparative learning, and the method comprises the following steps: firstly, on an unbalanced data set, strengthening the sampling of tail samples through a dynamic heavy balance strategy, and constructing a preliminary sampling data set; thirdly, training the preliminary sampling data set by adopting a global-local contrast learning mechanism; and finally, constructing a pre-training model for long-tail target detection and large-scale fine-grained image feature learning. According to the invention, through global-local contrast learning, a dynamic weight balance strategy and a dual reconstruction mechanism, the fine-grained feature representation capability of the model in the image is improved, and the performance of the model in long-tail object detection, especially the detection performance of tail samples, is significantly improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, covering two directions: long-tail target detection and fine-grained large models, and specifically relates to a pre-training method, storage medium and electronic device for long-tail target detection and fine-grained large models based on dynamic rebalanced contrastive learning. Background Art

[0002] With the rapid development of deep learning technology, significant progress has been made in the field of computer vision, especially in the paradigm of large-scale pre-training and fine-tuning optimization. The widespread use of pre-training methods enables models to capture domain-specific or task-related concepts, thereby significantly improving the performance of downstream tasks. In the field of object detection, traditional methods usually rely on large-scale datasets for pre-training so that the backbone network of the model can obtain good feature representation. However, these pre-training methods often ignore the problem of random initialization of model components caused by long-tail data distribution, which brings significant performance limitations on key detection components.

[0003] For long-tail data distribution scenarios, traditional supervised pre-training models rely on the distribution of labeled data, resulting in models that perform well on head categories but poorly on tail categories. Although self-supervised learning has shown the potential to enable models to learn richer and more effective feature representations without relying on labeled data, it still faces challenges. One of the most important problems is simplicity bias, which means that deep neural networks tend to rely on simple prediction patterns and ignore complex features that are critical for model generalization. This bias is particularly prominent in tail categories, because samples in tail categories are scarce and easily ignored by the model, making it difficult to effectively capture the complex features of these categories.

[0004] Similarly, simplicity bias is also a significant problem in the pre-training of fine-grained large models. Fine-grained tasks usually require the model to distinguish between categories with extremely high similarity, which means that the model needs to learn rich details and complex features. However, deep neural networks tend to focus on simple patterns that are easy to capture during pre-training, while ignoring these fine-grained features, resulting in poor performance of the model when processing complex fine-grained images. Therefore, how to effectively solve the simplicity bias and improve the model's ability to learn fine-grained features remains an important research challenge.

[0005] In this context, how to effectively deal with the problems of long-tail data distribution and fine-grained feature learning during pre-training and balance the feature learning of head and tail categories has become an important research direction in the field of computer vision. Therefore, developing a new pre-training strategy can not only ensure the generalization ability of the model in large-scale datasets, but also effectively solve the problems caused by the scarcity of tail category data and the simplicity bias in fine-grained feature learning. It is suitable for a variety of visual tasks such as object detection and fine-grained feature learning, which will be of practical significance. Summary of the invention

[0006] The purpose of the present invention is to provide a long-tail target detection and fine-grained large model pre-training method based on dynamic rebalanced contrastive learning.

[0007] The technical solution for achieving the purpose of the present invention is:

[0008] A method for long-tail object detection and pre-training of fine-grained large models based on dynamic rebalanced contrastive learning, comprising the following steps:

[0009] Step 1: On the unbalanced dataset, a dynamic rebalancing strategy is used to strengthen the sampling of categories with very few samples and construct a preliminary sampling dataset. During the sampling process, the sampling frequency of the head and tail classes is dynamically adjusted to ensure the effective use of the tail class samples in pre-training.

[0010] Step 2: Use global-local contrast learning mechanism and dual reconstruction mechanism to train the preliminary sampled data set to jointly solve the problem of long-tail distribution of data and capture more fine-grained features; in global contrast learning, two different image views are generated by random enhancement, and the encoder and momentum encoder are used to generate global feature representation; and local contrast learning is performed at the same time, and the detailed features of the object are captured through the object-level feature extraction module to ensure that the model can accurately represent the object information in complex scenes; at the same time, a dual reconstruction mechanism including appearance reconstruction and semantic reconstruction is introduced for reconstruction. Appearance reconstruction forces the model to capture the detailed features of the input image and maintain the consistency of visual information through pixel-level reconstruction tasks; semantic reconstruction reconstructs partially occluded images to maintain the semantic consistency between the original image and the occluded image, thereby improving the model's feature learning ability, especially the representation ability of tail class samples;

[0011] Step 3: Build a pre-trained model for long-tail object detection and large-scale fine-grained image feature learning.

[0012] A computer storage medium stores a computer program, which implements the steps of the method when executed by a processor.

[0013] An electronic device includes a memory and one or more processors, wherein the memory is used to store one or more programs; when the one or more programs are executed by the one or more processors, the described method is implemented.

[0014] Compared with the prior art, the present invention has the following significant advantages:

[0015] (1) Through the dynamic rebalancing strategy, the present invention strengthens the sampling of categories with sparse sample numbers in unbalanced data sets, ensuring that the tail categories receive more attention during the pre-training process, thereby significantly improving the detection accuracy of long-tail categories and effectively solving the performance problem caused by long-tail data distribution in the prior art.

[0016] (2) By adopting the global-local contrast learning mechanism and the dual reconstruction mechanism, the present invention can perform joint feature learning at both the global semantic and object detail levels, so that the model can not only learn a wide range of semantic information, but also deeply explore the detailed features at the object level, thereby performing better in large-scale fine-grained image feature learning tasks.

[0017] (3) Through the appearance reconstruction and semantic reconstruction in the dual reconstruction mechanism, the model captures the subtle visual features of the image while maintaining semantic consistency, thereby effectively alleviating the simplicity bias problem in deep learning. This design enables the model to have stronger robustness and generalization ability in long-tail object detection and fine-grained tasks, and can better cope with complex visual scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flow chart of the present invention.

[0019] Figure 2 It is a schematic diagram of the framework of the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0021] like Figure 1 and 2 As shown, the present invention provides a long-tail target detection and fine-grained large model pre-training method based on dynamic rebalanced contrastive learning, which specifically includes the following steps:

[0022] Step 1: On the unbalanced data set, a dynamic rebalancing strategy is used to strengthen the sampling of categories with very few samples and construct a preliminary sampling data set;

[0023] The dynamic rebalancing strategy is used to accurately control the category distribution at the image and instance levels and ensure that the tail categories are fully represented. The dynamic rebalancing strategy proposed in this paper combines a dynamic adjustment mechanism to enable the model to learn the overall distribution of the dataset in the early stage of pre-training and gradually shift the focus to the tail categories. Specifically, for each category c, its proportion score at the image and instance levels is calculated, which are recorded as and And calculate the comprehensive score f of the category through these scores c :

[0024]

[0025] in, T represents the current training round number, T max Represents the total number of training theories. As pre-training progresses, the parameter α d Gradually increasing, the focus of the model shifts from image-level balance to instance-level balance, and strengthens the attention to the tail category. c , define the category-level replication factor r c , whose formula is The hyperparameter t is set to 0.001 to ensure that the tail categories are sampled more frequently during training, thereby effectively alleviating the data imbalance problem and avoiding the risk of overfitting due to excessive repeated sampling.

[0026] Step 2: Use the global-local contrast learning mechanism and dual reconstruction mechanism to train the preliminary sampled data set to jointly solve the long-tail distribution problem of the data and capture more fine-grained features;

[0027] The global-local contrastive learning and dual reconstruction mechanism in step 2 are the core of model pre-training.

[0028] In global contrastive learning, for an input image I, different views x and x are generated. + , and use it as the input of the encoder and momentum encoder. Each view is randomly enhanced so that the model learns object-level representations that are invariant to translation and scale. The enhancements for x include Resize, RandomFlip, and + The enhancements also include RandomGrayscale, RandomGaussianBlur, RandomSolarize, and ColorJitter. + Generate global-level representations z and z through independent projectors + , and perform l2 normalization. The network is trained using the InfoNCE loss:

[0029]

[0030] The temperature hyperparameter τ is set to 0.2. represents the global contrast loss. The local contrast learning mechanism enables the model to learn a comprehensive representation of diverse object proposals by maximizing the similarity of object proposals across enhanced views. Specifically, a class-agnostic detector is used to generate a series of proposals The object-level representation is obtained by the target detection head, denoted as z bb The local contrastive learning loss for local-level representation is defined as:

[0031]

[0032] in, represents the local contrast loss, represents the positive sample object-level representation of the input x, Object-level representation for negative samples

[0033] On this basis, a dual reconstruction mechanism is introduced to solve the simplicity bias problem of the model and improve the detection effect of tail categories. It consists of two key parts: appearance reconstruction and semantic reconstruction, with the aim of enabling the model to capture detailed features and maintain semantic consistency at the same time.

[0034] Appearance reconstruction uses pixel-level reconstruction tasks to encourage the model to capture as many detailed features in the input image as possible to ensure accurate retention of visual information. Specifically, the encoder f maps the input image x to the feature space to obtain the feature representation z, and the generator g attempts to reconstruct the input image through the feature to obtain the reconstructed image. The loss function is defined using pixel-level mean square error to ensure that the model can reconstruct the input image with high fidelity:

[0035]

[0036] in, represents the appearance reconstruction loss, f(x) represents the potential features of the input x;

[0037] Semantic reconstruction ensures that the model can still understand the semantics of the original image when encountering visual perturbations by maintaining the semantic consistency between the original image and the reconstructed image after occlusion. Specifically, the reconstructed image g(f(x)) is masked to generate the occluded version, denoted as The encoder then regenerates the corresponding feature representation. Its loss is calculated by evaluating the consistency between the original image features and the occluded image features, as follows:

[0038]

[0039] in, represents semantic reconstruction loss;

[0040] The dual reconstruction component forces the model to capture visual details and maintain semantic consistency at the same time, thereby effectively solving the simplicity bias problem and improving the overall performance of long-tail object detection and large-scale fine-grained image feature learning.

[0041] Step 3: Build a pre-trained model for long-tail object detection and large-scale fine-grained image feature learning.

[0042] The training method described in step 2 is co-trained to build a pre-trained model for object detection suitable for long-tail distribution and a large model for fine-grained feature learning.

[0043] The loss of the joint training of global-local contrastive learning is expressed as:

[0044]

[0045] in, stands for global-local contrast loss;

[0046] The loss of dual reconstruction joint training is expressed as:

[0047]

[0048] in, represents the dual reconstruction loss;

[0049] The hyperparameter settings in formula (6) and formula (7) are as follows: c Set to 0.1, β c Set to 0.05, α r is set to 0.1. Finally, the loss of the contrastive learning method combined with dual reconstruction is expressed as:

[0050]

[0051] in, Represents the final loss of model training, Represents the detection loss, which is used to make the training more stable and better adapted to the pre-training task. In this way, the model's feature learning of tail data can be enhanced, effectively alleviating the simplicity bias problem, and prompting the model to learn more fine-grained feature representations, thereby building a pre-training model for long-tail object detection and large-scale fine-grained image feature learning.

Claims

1. The pre-training method of long-tail object detection and fine-grained large model includes the following steps: Step 1: On the unbalanced data set, a dynamic rebalancing strategy is used to strengthen the sampling of categories with very few samples and construct a preliminary sampling data set; Step 2: Use the global-local contrast learning mechanism and dual reconstruction mechanism to train the preliminary sampled data set to jointly solve the long-tail distribution problem of the data and capture more fine-grained features; Step 3: Build a pre-trained model for long-tail object detection and large-scale fine-grained image feature learning.

2. The method for long-tail object detection and pre-training of a fine-grained large model according to claim 1, characterized in that: Step 1 uses a dynamic rebalancing strategy to precisely control the category distribution at the image and instance level and ensure that the tail categories are fully represented. The proposed dynamic rebalancing strategy combines a dynamic adjustment mechanism to enable the model to learn the overall distribution of the dataset at the beginning of pre-training and gradually shift the focus to the tail categories. Specifically, for each category c, its proportion scores at the image and instance levels are calculated, respectively recorded as and And calculate the comprehensive score f of the category through these scores c : in, T represents the current training round number, T max Represents the total number of training theories; as pre-training progresses, parameter α d Gradually increases, so that the focus of the model shifts from image-level balance to instance-level balance, and strengthens the attention to the tail category; based on the score f c , define the category-level replication factor r c , whose formula is The hyperparameter t is set to 0.001 to ensure that the tail categories are sampled more frequently during training, thereby effectively alleviating the data imbalance problem and avoiding the risk of overfitting due to excessive repeated sampling.

3. The method for long-tail object detection and pre-training of fine-grained large models according to claim 1, characterized in that: The global-local contrast learning and dual reconstruction mechanism in step 2 are the core of model pre-training; Specifically, in global contrast learning, for an input image I, different views x and x are generated. + , and use it as the input of the encoder and momentum encoder; each view is randomly enhanced so that the model learns object-level representations that are invariant to translation and scale, where the enhancements for x include Resize, RandomFlip, and the enhancements for x + The enhancements also include RandomGrayscale, RandomGaussianBlur, RandomSolarize, and ColorJitter; subsequently, x and x + Generate global-level representations z and z through independent projectors + , and perform l2 normalization; train the network using InfoNCE loss: Among them, the temperature hyperparameter τ is set to 0.2, represents the global contrast loss; the local contrast learning mechanism enables the model to learn a comprehensive representation of diverse object proposals by maximizing the similarity of object proposals across enhanced views; specifically, a class-independent detector is used to generate a series of proposals The object-level representation is obtained by the target detection head, denoted as z bb ; The local contrastive learning loss of the local level representation is defined as: in, represents the local contrast loss, represents the positive sample object-level representation of the input x, Object-level representations representing negative samples; On this basis, a dual reconstruction mechanism is introduced to solve the simplicity bias problem of the model and improve the detection effect of tail categories. It consists of two key parts: appearance reconstruction and semantic reconstruction, the purpose of which is to enable the model to capture detailed features and maintain semantic consistency at the same time. Appearance reconstruction uses pixel-level reconstruction tasks to enable the model to capture detailed features in the input image and ensure accurate retention of visual information. Specifically, the encoder f maps the input image x to the feature space to obtain the feature representation z, and the generator g attempts to reconstruct the input image through the feature to obtain the reconstructed image. The loss function is defined using pixel-level mean square error to ensure that the model can reconstruct the input image with high fidelity: in, represents the appearance reconstruction loss, f(x) represents the potential features of the input x; Semantic reconstruction ensures that the model can still understand the semantics of the original image when encountering visual perturbations by maintaining the semantic consistency between the original image and the reconstructed image after occlusion. Specifically, a mask operation is performed on the reconstructed image g(f(x)) to generate an occluded version, denoted as The corresponding feature representation is then regenerated through the encoder; its loss is calculated by evaluating the consistency between the original image features and the occluded image features: in, represents semantic reconstruction loss; The dual reconstruction component forces the model to capture visual details and maintain semantic consistency at the same time, thereby solving the simplicity bias problem and improving the overall performance of long-tail object detection and large-scale fine-grained image feature learning.

4. The method for long-tail object detection and pre-training of fine-grained large models according to claim 3 is characterized in that: In step 3, the training method of step 2 is co-trained to build a pre-trained model for object detection suitable for long-tail distribution and a large model for fine-grained feature learning; Specifically, the loss of global-local contrastive learning joint training is expressed as: in, stands for global-local contrast loss; The loss of dual reconstruction joint training is expressed as: in, represents the dual reconstruction loss; The hyperparameter settings in formula (6) and formula (7) are as follows: c Set to 0.1, β c Set to 0.05, α r is set to 0.1; finally, the loss of the contrastive learning method combined with dual reconstruction is expressed as: in, Represents the final loss of model training, Representing detection loss is used to make training more stable and better adapt to pre-training tasks; in this way, the model's feature learning of tail data can be enhanced, the simplicity bias problem can be alleviated, and the model can be prompted to learn more fine-grained feature representations, thereby constructing a pre-training model for long-tail target detection and large-scale fine-grained image feature learning.

5. A computer storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

6. An electronic device, characterized in that: The method comprises a memory and one or more processors, wherein the memory is used to store one or more programs; when the one or more programs are executed by the one or more processors, the method according to any one of claims 1 to 4 is implemented.

Images