Target detection model training method, device, readable storage medium and program product

Abstract

The application provides a target detection model training method and device, readable storage medium and program product. The method comprises: obtaining a seed data set; generating a corresponding mask for an instance in a bounding box data set to obtain a first candidate data set; adding a bounding box and a corresponding class label to the instance in the instance mask data set to obtain a second candidate data set; extracting features of the seed data set, the first candidate data set and the second candidate data set, and respectively clustering according to a plurality of scales set to correspondingly obtain a plurality of scales of first clustering clusters, second clustering clusters and third clustering clusters; retrieving a set number of first samples for each scale of the first clustering clusters; constructing a first augmented data set based on second samples in the second clustering clusters and the third clustering clusters that have a similarity greater than a set threshold to the first samples and the seed data set; and training a first model based on the first augmented data set. The present scheme can improve the accuracy of model training.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a method, apparatus, readable storage medium, and program product for training an object detection model. Background Technology

[0002] The learning process of object detection models is essentially a data-driven implicit learning process, that is, automatically extracting features and optimizing decisions through massive amounts of data.

[0003] In related technologies, instance images labeled with masks, detection boxes, and detection box category labels are used as the training set. This training set is then input into the object detection model, and the model's parameters are updated based on its output. However, for specific downstream tasks, object detection models require specific training sets, which can easily lead to insufficient accuracy after training due to a lack of training samples. Summary of the Invention

[0004] This application provides a target detection model training method, device, readable storage medium, and program product, which can improve the accuracy of model training for specific downstream tasks.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides a method for training an object detection model, the method comprising: Obtain a seed dataset, wherein the seed dataset includes instances of multiple categories and the corresponding masks, detection boxes, and category labels of the instances; Based on the detection boxes in the instance detection box dataset, generate the corresponding mask for the instance in the detection box dataset, and use the instance detection box dataset with the generated mask as the first candidate dataset. Based on the mask corresponding to the instance in the instance mask dataset, add the detection box and the corresponding category label to the instance in the instance mask dataset, and use the instance mask dataset with the added detection box and category label as the second candidate dataset; Features of the seed dataset, the first candidate dataset, and the second candidate dataset are extracted, and clusters are performed on the features of the seed dataset, the first candidate dataset, and the second candidate dataset according to multiple set scales to obtain a first cluster, a second cluster, and a third cluster at multiple scales. For each scale of the first cluster, a set number of first samples are retrieved; The first augmented dataset is composed of second samples from the second cluster and the third cluster whose similarity to the first sample is greater than a set threshold, as well as the seed dataset; A first model is trained based on the first augmented dataset, wherein the first model is used as a teacher model to guide the training of downstream task models.

[0006] This application provides an object detection model training device, including: The seed dataset acquisition module is used to acquire a seed dataset, wherein the seed dataset includes instances of multiple categories and the corresponding masks, detection boxes and category labels of the instances; The first candidate dataset acquisition module is used to generate a corresponding mask for the instance in the instance detection box dataset based on the detection boxes in the instance detection box dataset, and use the instance detection box dataset with the generated mask as the first candidate dataset. The second candidate dataset acquisition module is used to add the detection box and the corresponding category label to the instance in the instance mask dataset according to the mask corresponding to the instance in the instance mask dataset, and use the instance mask dataset with the added detection box and the category label as the second candidate dataset. A multi-scale clustering module is used to extract features of the seed dataset, features of the first candidate dataset, and features of the second candidate dataset, and to cluster the features of the seed dataset, features of the first candidate dataset, and features of the second candidate dataset according to multiple set scales, thereby obtaining a first cluster, a second cluster, and a third cluster at multiple scales. The retrieval module is used to retrieve a set number of first samples for each scale of the first cluster. An enhanced dataset construction module is used to construct a first enhanced dataset based on second samples in the second cluster and the third cluster that have a similarity greater than a set threshold with the first sample, as well as the seed dataset; The training module is used to train a first model based on the first augmented dataset, wherein the first model is used as a teacher model to guide the training of downstream task models.

[0007] This application provides an electronic device, the electronic device comprising: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the target detection model training method provided in the embodiments of this application.

[0008] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the target detection model training method provided in this application.

[0009] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the target detection model training method provided in this application.

[0010] The embodiments of this application have the following beneficial effects: This application embodiment obtains a seed dataset containing annotations (masks, bounding boxes, and category labels) for instances of various categories, and supplements it with datasets that have different partial annotations. Then, it retrieves a highly class-aligned and balanced first augmented dataset from the supplemented dataset based on the seed dataset. Pre-training the first model using the first augmented dataset reduces the differences between the dataset and downstream tasks in terms of category distribution, appearance, and background features. This allows the first model (teacher model) to have domain adaptability to downstream tasks before encountering them, enabling it to accurately identify specific instances in downstream tasks. This reduces the risk of decreased model generalization performance due to inconsistent data distribution and improves the accuracy of model training. Attached Figure Description

[0011] Figure 1 This is a schematic diagram of the architecture of the target detection model training system provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the target detection model training device provided in the embodiments of this application; Figure 3 This is a schematic diagram of the first process of the target detection model training method provided in the embodiments of this application; Figure 4 This is a schematic diagram of the second process of the target detection model training method provided in the embodiments of this application; Figure 5 This is a schematic diagram illustrating the principle of obtaining seed datasets provided in an embodiment of this application; Figure 6 This is a schematic diagram illustrating the principle of obtaining the first candidate dataset provided in an embodiment of this application; Figure 7 This is a schematic diagram of the third process of the target detection model training method provided in the embodiments of this application; Figure 8 This is a schematic diagram illustrating the principle of obtaining the second candidate dataset provided in an embodiment of this application; Figure 9 This is a schematic diagram of the fourth process of the target detection model training method provided in the embodiments of this application; Figure 10 This is a schematic diagram of the fifth step of the target detection model training method provided in the embodiments of this application; Figure 11This is a schematic diagram of the sixth process of the target detection model training method provided in the embodiments of this application; Figure 12 This is a schematic diagram of the seventh process of the target detection model training method provided in the embodiments of this application; Figure 13 This is a schematic diagram of the data set processing flow in the application scenario provided in the embodiments of this application.

[0012] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0015] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0016] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0017] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0018] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0019] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0020] An instance refers to a single physical object or target entity in computer vision data processing that objectively exists within the image, possesses continuous physical boundary attributes, belongs to a specific business category, and is physically and logically independent and discretely countable. It constitutes the smallest logical and physical granularity for the computer vision system to perform perception tasks and data sampling operations. The system extracts a high-dimensional feature vector reflecting the individual target region and, during the cross-database feature retrieval stage, uses this entity feature as the query benchmark to perform nearest neighbor matching and equal-quantity sampling, fundamentally ensuring the structural consistency and category distribution stability of data augmentation and expansion. In the embodiments of this application, for example, it could be a specific mechanical part or grasping target objectively existing in an industrial scene image, or an element of a set of individual targets with specific independent visual representations constituting a large-scale data index.

[0021] A detection box is a basic data structure that uses a geometric rectangular boundary in a two-dimensional Cartesian coordinate system to initially identify and locate the absolute position and coverage area of ​​a specific instance in the digital image space.

[0022] A mask is a two-dimensional pixel-level data structure used in digital image processing tasks to accurately describe and define the true geometric contours and physical boundaries of a specific instance within a pixel matrix.

[0023] A seed dataset refers to a limited number of accurately labeled instances used in machine learning and computer vision model training, serving as an initial benchmark dataset for subsequent large-scale data expansion, feature matching, or model domain adaptation. In this embodiment, the seed dataset primarily serves as a query source for cross-database feature retrieval and nearest neighbor matching. It is used to perform high-dimensional spatial distance calculations in a heterogeneous data warehouse containing a massive number of the aforementioned detection boxes or masks, thereby guiding the system to extract evenly distributed and semantically aligned samples to construct an enhanced dataset, overcoming the technical limitation of the limited scale of original labeled data in specific business scenarios.

[0024] Clustering is an unsupervised data processing algorithm or data structuring mechanism that aims to divide a large number of unordered data samples in a multidimensional feature space into multiple independent sets, so that data samples within the same set maintain high similarity, while samples in different sets exhibit feature isolation. In the embodiments of this application, clustering can be a scheme that uses the K-Means algorithm, with cosine similarity as the feature distance metric, to generate multiple multi-granularity feature index clusters by independently running the algorithm on multiple scales set for massive visual features.

[0025] A segmentation model is a type of computer vision processing system built on deep convolutional networks or attention mechanism transformers. It is specifically configured to receive input in the form of digital images and output corresponding semantic classifications or region prediction results for independent instances at the pixel level.

[0026] The Image-Text Alignment Model (IPA) is a deep neural network architecture with cross-modal feature fusion and computation capabilities. It features a parallel visual encoder and text decoder, capable of mapping data from different modalities to the same unified high-dimensional vector space, and outputting a similarity metric representing the semantic consistency between the two. In this embodiment, the IPA is deployed as a quality control step after the data is automatically generated by the aforementioned segmentation model, providing semantic gating functionality (mechanism). The IPA rigorously calculates the distance between the local visual features of the instance region and its associated category text features, performing a deep semantic matching check on samples that have passed geometric verification. This thoroughly eliminates low-quality pseudo-label data with positioning errors, category confusion, and irrelevant visual content, ensuring that the augmented data clusters generated by the aforementioned clustering retrieval have extremely high label confidence. In this embodiment, when processing augmented data or un-category augmented data generated based on the aforementioned detection boxes, the IPA can accurately intercept and remove low-quality erroneous augmented data based on the cosine similarity calculation of features and combined with a set filtering threshold.

[0027] Knowledge distillation is a model compression and performance transfer algorithm in the field of machine learning. It constructs an asymmetric supervised learning path to guide a target network with fewer parameters and a lightweight structure to fit or mimic the output prediction distribution or intermediate layer feature representation of a complex network with a large number of parameters and stronger feature representation capabilities. In this embodiment, the knowledge distillation algorithm is applied to the network optimization stage after training using the aforementioned large-scale augmented dataset. Its main technical role is to overcome the hardware bottleneck of limited computing power on edge devices, which cannot directly deploy large-parameter networks. By reusing the soft-supervised labels containing non-rigid classification logic output by the complex network for specific images, it guides the lightweight target network to update its weights, thereby significantly improving its prediction accuracy and generalization ability for the aforementioned masks and detection boxes while reducing the model's computational scale.

[0028] The teacher model refers to the baseline deep learning network that serves as a knowledge source or guide in the aforementioned knowledge distillation architecture or pseudo-label self-training process. It is typically pre-trained intensively on large-scale heterogeneous data, possessing a relatively large parameter set, extremely high feature extraction accuracy, and a broad data generalization perspective. In this embodiment, it specifically refers to the first model, which primarily serves the dual functions of an automated high-level labeler and an implicit knowledge carrier. During the data construction and expansion phase, it performs high-precision forward inference computation on unlabeled or weakly labeled business data, utilizing the built-in segmentation model to output the aforementioned mask containing precise edges and the corresponding category prediction as soft labels or pseudo-ground truth signals, thereby providing a high-quality fitting target and parameter update guidance for a smaller network.

[0029] The student model refers to the target deep learning network, which serves as the knowledge receiver in the aforementioned knowledge distillation transfer learning system. Its hardware memory consumption, network computational complexity, and total parameter size are objectively lower than those of the corresponding teacher model. In the application embodiments, all downstream task models of the first model can become student models (e.g., the fourth model). The student model, as a specific application entity to be deployed in real-world industrial scenarios with limited computing power, calculates the error gradient to update its network structure parameters by inputting the same image samples and comparing its own prediction results with the high-quality soft-label data output by the aforementioned teacher model. In some embodiments, the student model can be a lightweight vision network directly deployed in the humanoid robot's edge processor, or it can represent the edge inference network in the aforementioned technical solution used to perform specific downstream industrial grasping tasks and requires the transfer of generalization representation capabilities from a complex large model.

[0030] In related technologies, the common practice for training object detection or instance segmentation models is "pre-training on a general dataset + fine-tuning on a specific business dataset". Specifically, firstly, a large-scale open-source general dataset (such as an image library with target location annotations) is used to pre-train the initial model (equivalent to the first model), enabling it to possess basic feature extraction and general target localization capabilities. Then, for specific downstream business scenarios such as humanoid robot environmental perception and industrial parts grasping, a small number of real-world scene images (or supplemented with some computer simulation images) are collected and manually annotated to construct a downstream task dataset. Based on this, the pre-trained model is directly fine-tuned and updated using this downstream task dataset. Alternatively, to adapt to the computing power and latency limitations of edge devices (such as robot controllers), knowledge distillation techniques are used, using the pre-trained large-scale complex model as guidance to transfer features to a lightweight edge model with smaller parameters, aiming to meet the perception and reasoning needs of specific scenarios.

[0031] However, open-source pre-training datasets and specific downstream business data often exhibit significant domain gaps in terms of target categories, visual appearance, surface texture, and background distribution. Furthermore, high-quality instance-level pixel annotations are extremely costly, resulting in a limited scale of real-world labeled data available for downstream applications. Therefore, directly fine-tuning the model based on downstream datasets with disparate distributions and small scales can easily lead to overfitting to a small number of specific samples. It can also cause a "catastrophic forgetting" of the generalization ability learned during pre-training. Moreover, even with the introduction of simulation data, the differences in visual representation of lighting and materials between simulation data and the real physical world are difficult to completely eliminate. These combined factors result in existing models struggling to achieve effective capability adaptation and transfer when faced with multiple independent downstream tasks lacking sufficient annotations. They exhibit weak cross-scene generalization capabilities and are unable to provide high-precision, high-stability target detection and instance mask segmentation results in complex industrial applications.

[0032] Based on the problems existing in related technologies, embodiments of this application provide a method, device, computer-readable storage medium, and computer program product for training an object detection model. This method provides a highly class-aligned and evenly distributed first augmented dataset, thereby improving the adaptability of the first model to different types of instances. The exemplary application of the object detection model training device provided in this application embodiment is described below. The device provided in this application embodiment can be implemented as various types of terminals such as laptops, tablets, and desktop computers, or as a server. The exemplary application when the device is implemented as a terminal will be described below.

[0033] See Figure 1 , Figure 1This is a schematic diagram of the architecture of the object detection model training system 100 provided in this application embodiment. To perform object detection operations, it can provide an object detection model training application, or it can be a functional module in other applications (such as a model training module in an object detection application). The object detection model training system 100 provided in this application embodiment includes at least a terminal 400, a network 300, and a server 200, wherein the server 200 is the server 200 for the object detection model training application. The server 200 can constitute the data processing device of this application embodiment, that is, the object detection model training method of this application embodiment is implemented through the server 200. The terminal 400 is connected to the server 200 through the network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0034] See Figure 1 The target detection model training method of this application embodiment can be executed by server 200, that is, the training of the first model can be completed in server 200. Server 200 obtains a seed dataset, wherein the seed dataset includes instances of multiple categories and the corresponding masks, detection boxes, and category labels of the instances; server 200 generates corresponding masks for the instances in the detection box dataset based on the detection boxes in the instance detection box dataset, and uses the instance detection box dataset with generated masks as the first candidate dataset; server 200 adds detection boxes and corresponding category labels to the instances in the instance mask dataset based on the masks corresponding to the instances in the instance mask dataset, and uses the instance mask dataset with added detection boxes and category labels as the second candidate dataset; server 200 extracts features from the seed dataset and features from the first candidate dataset. The server 200 uses the features of the seed dataset and the second candidate dataset, and clusters these features according to multiple set scales to obtain first clusters, second clusters, and third clusters at multiple scales. For each scale of the first cluster, the server 200 retrieves a set number of first samples. The server 200 then constructs a first augmented dataset based on second samples from the second and third clusters whose similarity to the first samples exceeds a set threshold, along with the seed dataset. The server 200 trains a first model based on the first augmented dataset, which serves as a teacher model to guide the training of downstream task models. After training the first model, the terminal 400 sends the model to be knowledge distilled to the server 200. After the server 200 trains the model based on the first model, it deploys the knowledge-distilled model to the terminal 400 via network 300.

[0035] As an example, when an object detection model is required to identify instances of category A and category B, a seed dataset containing categories A and B is first obtained. By annotating the dataset containing only instance detection boxes and the dataset containing only instance masks, a first candidate dataset and a second candidate dataset are obtained. Based on the seed dataset, a balanced first augmentation dataset is selected from the first candidate dataset and the second candidate dataset. This results in a training dataset containing rich information and balanced feature distribution of categories A and B, which is used to train the first model, thereby improving the generalization ability and stability of the first model.

[0036] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the server 200 provided in the embodiments of this application. Figure 2 The server 200 shown includes at least one processor 210, memory 250, at least one network interface 220, and a user interface 230. The various components in server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to implement communication between these components. In addition to a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 240.

[0037] Processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0038] User interface 230 includes one or more output devices 231 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0039] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices physically located away from the processor 210.

[0040] The memory 250 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 250 described in this application embodiment is intended to include any suitable type of memory.

[0041] In some embodiments, memory 250 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0042] Operating system 251 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 252 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc. The input processing module 254 is used to detect and translate one or more user inputs or interactions from one or more input devices 232.

[0043] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A training device 255 for an object detection model stored in memory 250 is shown. This device can be software in the form of programs or plugins, and includes the following software modules: a seed dataset acquisition module 2551, a first candidate dataset acquisition module 2552, a second candidate dataset acquisition module 2553, a multi-scale clustering module 2554, a retrieval module 2555, an augmented dataset construction module 2556, and a training module 2557. These modules are logically connected and can therefore be arbitrarily combined or further split according to their implemented functions. The functions of each module will be described below.

[0044] The target detection model training method provided in this application will be described in conjunction with exemplary applications and implementations of the server provided in the embodiments of this application.

[0045] See Figure 3 , Figure 3 This is a flowchart illustrating the target detection model training method provided in an embodiment of this application. Figure 3 The main component of the process is the server, which will be combined with... Figure 3The steps shown are explained, and the method includes steps 101 to 107.

[0046] In step 101, the seed dataset is obtained.

[0047] The seed dataset includes instances of multiple categories, along with the corresponding masks, bounding boxes, and category labels for each instance.

[0048] Here, the seed dataset refers to a basic data set that is constructed based on downstream business scenarios, has high-quality annotations, and can represent the data distribution of downstream business target tasks. In this embodiment of the application, it can be used as a "query source" for subsequent dataset expansion.

[0049] In some embodiments, the specific execution details for obtaining the seed dataset are as follows: A self-supervised large-scale vision model is used to extract features from all bounding boxes and corresponding instance regions in the dataset, resulting in high-dimensional feature vectors (e.g., 384-dimensional CLS token features). Simultaneously, it is ensured that each instance contains an accurate bounding box, a pixel-level mask, and a corresponding semantic category label, thereby forming a seed dataset rich in semantic and geometric information.

[0050] In some embodiments, see Figure 4 The seed dataset can be obtained by performing steps 108 to 109.

[0051] In step 108, the mask, detection box, and category label corresponding to the instance in the second downstream dataset are obtained by using the second model trained based on the first downstream dataset, thus obtaining the first downstream labeled dataset. The mask, detection box, and category label corresponding to the instance in the first downstream dataset are obtained by using the third model trained based on the second downstream dataset, thus obtaining the second downstream labeled dataset.

[0052] Here, the second and third models refer to expert models (Specialist Models) pre-trained for specific downstream business scenarios (such as business scenario A and business scenario B). They typically have the ability to segment instances in a specific domain (able to obtain the mask, detection box, and category label of a specific class of instance in an image based on an unlabeled image). The first and second downstream labeled datasets refer to intermediate datasets containing pseudo-labels generated through model inference.

[0053] In some embodiments, the first downstream dataset and the second downstream dataset can be two independent business datasets with different category distributions or scenario characteristics (e.g., one is a mechanical parts dataset and the other is a mechanical clamping structure dataset). The second model and the third model can be obtained by supervised training (cross-inference) on the corresponding datasets of each other using instance segmentation algorithms such as Mask R-CNN, Cascade R-CNN, or SOLOv2.

[0054] In some embodiments, the acquisition of the first and second downstream annotation datasets can be achieved through cross-inference: images from the second downstream dataset are input into a second model for forward propagation to predict instance masks, bounding boxes, and categories in the images; similarly, images from the first downstream dataset are input into a third model for prediction. During this process, to ensure data quality, a confidence threshold (e.g., 0.5) is typically set, retaining only instance annotation results with prediction confidence higher than this threshold, thereby filtering out low-quality pseudo-labels.

[0055] In some embodiments, the seed dataset can be constructed as follows: First, multiple downstream business datasets (e.g., business A dataset and business B dataset) are acquired, which contain manually labeled instance categories; then, the business B dataset is inferred and labeled using a model trained on business A dataset, and the business A dataset is inferred and labeled using a model trained on business B dataset, with category labels supplemented by cross-labeling; finally, the merged dataset is used as the initial seed data.

[0056] As an example, let's consider a first downstream dataset containing the mask, bounding box, and class label of instance A, and a second downstream dataset containing the mask, bounding box, and class label of instance B. A second model is trained using the first downstream dataset, enabling it to annotate instance A. A third model is trained using the second downstream dataset, enabling it to annotate instance B. When images from the second downstream dataset are input into the second model, it annotates instance A, resulting in the first downstream labeled dataset. Since the second downstream dataset itself contains the mask, bounding box, and class label of instance B, and also obtains the mask, bounding box, and class label of instance A through the second model, the first downstream labeled dataset contains instances of two different categories, instance A and instance B, along with their corresponding masks, bounding boxes, and class labels. When images from the first downstream dataset are input into the third model, it annotates instance B, resulting in the second downstream labeled dataset. Since the first downstream dataset itself has the mask, detection box and category label of instance A, and also obtains the mask, detection box and category label of instance B through the third model, the second downstream annotation dataset has instances of two different categories, instance A and instance B, as well as the corresponding mask, detection box and category label of the instances.

[0057] In step 109, the first downstream annotation dataset and the second downstream annotation dataset are combined into a seed dataset.

[0058] Here, "combination" refers to the operation of unifying the format, deduplicating, and merging labeled data from different sources to build a basic dataset with broader category coverage and a larger data volume.

[0059] In some embodiments, the first downstream annotation dataset and the second downstream annotation dataset to be combined can be initially cleaned and format standardized (e.g., uniformly converted to COCO format or VOC format) to ensure that the mask encoding method, coordinate system definition and category label mapping table are consistent.

[0060] In some embodiments, combining datasets into a seed dataset can be achieved by first establishing a unified category index table and mapping the category labels of the two datasets to this unified index; then, using a set union operation to merge the image samples and corresponding annotation information of the two datasets.

[0061] This application embodiment utilizes different models trained on different downstream datasets to cross-label the downstream datasets, thereby uncovering unlabeled instances in the first and second downstream datasets that belong to categories of interest to the other's business. This cross-labeling strategy unifies and integrates the originally independent category spaces, avoiding the "missed labeling" phenomenon caused by simple direct merging (i.e., objects of category A are mistakenly identified as background in dataset B due to lack of labeling), thus constructing a seed dataset with consistent semantic definitions and comprehensive category coverage. Furthermore, this application embodiment also automatically generates a large number of high-quality pseudo-labels by reusing the capabilities of existing downstream models (the second and third models). This not only increases the number and density of instances in the seed dataset but also enriches the distribution of appearance features for each category by leveraging the scene differences (such as different lighting, backgrounds, and viewpoints) of data from different sources, providing a more robust and diverse "query source" for subsequent data retrieval and expansion. Moreover, by combining the cross-labeled datasets, the seed dataset incorporates feature distributions from multiple downstream business scenarios. This integration approach effectively alleviates the problems of small scale and limited scenarios in single business datasets, enabling subsequent processes built on this seed dataset to be based on a wider range of data distributions, thereby improving the generalization potential and robustness of the dataset.

[0062] As an example, see Figure 5 Taking two downstream datasets as an example, Business A (first downstream dataset) and Business B (second downstream dataset), the two datasets contain a total of approximately 200k images. In Business A dataset, only specific instance categories related to Business A (e.g., gears) are manually labeled (solid boxes), while categories related to Business B are not labeled. Similarly, in Business B dataset, only specific instance categories related to Business B (e.g., screws) are manually labeled, while categories related to Business A are not labeled. To expand the coverage of labeled categories and construct a unified seed dataset, firstly, downstream model A (second model), trained on Business A dataset, is used to perform cross-inference on the images in Business B dataset, automatically filling in the missing masks, detection boxes, and category labels (dashed boxes) for Business A categories, thus obtaining the first downstream labeled dataset containing complete A and B category labels. Simultaneously, downstream model B (third model), trained on Business B dataset, is used to perform cross-inference on the images in Business A dataset, automatically filling in the missing Business B category labeling information, thus obtaining the second downstream labeled dataset. Finally, the two complementary labeled datasets are merged to form a seed dataset that fully covers the categories required for business A and business B, providing a foundation for subsequent large-scale feature retrieval and expansion.

[0063] In step 102, based on the detection boxes in the instance detection box dataset, a corresponding mask is generated for the instances in the detection box dataset, and the instance detection box dataset with generated masks is used as the first candidate dataset.

[0064] Here, the first candidate dataset refers to a dataset that is only a dataset containing bounding box annotations (such as the large-scale open-source dataset Objects365), and which is supplemented with instance segmentation masks based on the bounding box annotations through algorithm enhancement.

[0065] In some embodiments, the instance detection bounding box dataset may be derived from a publicly available large-scale object detection dataset (e.g., Objects365), which contains rich categories and accurate detection boxes, but lacks the masking information required for instance segmentation.

[0066] In some embodiments, the corresponding mask can be generated by using the coordinates of the detection boxes in the dataset as a prompt and calling a large segmentation model such as the SAM model to generate the corresponding instance mask.

[0067] As an example, see Figure 6 In constructing the first candidate dataset for post-pre-training, a public dataset containing a large number of labeled instance detection boxes (e.g., the Objects365 dataset) is first used as the input source. For each instance in the input image (Image A), its original detection box coordinates are used as geometric location cue information (BoxPrompt), which is input into the prompt encoder of the segmentation model and transformed into a feature vector. Subsequently, the SAM model performs fine-grained reasoning within the region defined by the detection box based on the cue features, automatically generating a pixel-level mask corresponding to the instance. Through the above steps, the original dataset containing only object detection annotations (detection boxes + categories) is expanded into a dataset that also contains instance-level mask annotations, thus constructing a first candidate dataset with a rich 365-class category space, providing high-quality segmentation supervision signals for subsequent model training.

[0068] In some embodiments, see Figure 7 In step 102, based on the detection boxes in the instance detection box dataset, a corresponding mask is generated for the instance in the detection box dataset, and the instance detection box dataset with generated mask is used as the first candidate dataset. This can be achieved by executing steps 1021 to 1023.

[0069] It should be noted that the detection boxes in the instance detection box dataset are set with corresponding category labels.

[0070] In step 1021, the images with the detection boxes annotated in the instance detection box dataset are input into the image segmentation model to obtain the instance mask.

[0071] Here, an image segmentation model, such as the SAM model, refers to a deep learning model that can accurately separate a target object (instance) from the image background at the pixel level based on given spatial cues (such as points or rectangles).

[0072] In some embodiments, the instance detection bounding box dataset can be an open-source, large-scale object detection dataset (such as Objects365), which natively contains a large number of accurate instance coordinate boxes and corresponding category labels, but lacks pixel-level mask annotation information.

[0073] In some embodiments, the image is input into an image segmentation model to obtain a mask, which can be achieved by using the coordinates of existing detection boxes in the image as prompts and calling a general segmentation model based on the Transformer architecture for forward inference. Specifically, the model uses the geometric boundary information of the detection boxes to locate the target region, automatically extracts features within the region, and generates a high-quality binary mask that fits the edge of the instance.

[0074] In step 1022, the features of the mask and the features of the category label are extracted, and a first similarity score is calculated between the features of the mask and the features of the category label.

[0075] Here, the First Similarity Score is a numerical metric used to quantify the degree of cross-modal semantic consistency between the generated visual instance (masked region image) and the pre-defined textual semantics (category label).

[0076] In some embodiments, the extraction of mask features and category label features relies on large multimodal models of image-text alignment (e.g., image-text alignment models, also known as cross-modal alignment models), which have the ability to map visual and textual modalities to the same shared high-dimensional feature space.

[0077] In some embodiments, calculating the first similarity score between the mask features and the category label features can be achieved as follows: First, the instance region in the original image is extracted using the mask generated in step 1021, and its visual feature vector is extracted using an image encoder. Simultaneously, the corresponding category label is converted into standard prompt text (e.g., "a photo of a [category name]"), and its text feature vector is extracted using a text encoder. Then, the visual feature vector and the text feature vector are mapped to the same high-dimensional feature space, and the cosine similarity between these two high-dimensional feature vectors is calculated. This cosine similarity is used as the first similarity score to measure whether the mask image matches the category label.

[0078] In step 1023, the masks of the first similarity scores below the set first threshold are removed to obtain the first candidate dataset.

[0079] Here, the first threshold refers to an empirical numerical boundary set manually to ensure the quality of automatically generated pseudo-labels (masks), and is used as a criterion for "semantic gating".

[0080] In some embodiments, in the original detection box dataset, some detection boxes may contain multiple overlapping objects of different categories (i.e., category confusion), or the mask generated by the segmentation model in a complex background may contain too many non-target regions. These anomalies will lead to a low image-text first similarity score calculated in step 1022.

[0081] In some embodiments, removing masks below a threshold to obtain the first candidate dataset can be achieved as follows: Iterate through all generated mask samples and compare the corresponding first similarity score (e.g., cosine similarity, or similarity scores calculated using Euclidean distance conversion) with a set first threshold (e.g., 0.3) (e.g., extracting the generated instance region image features and corresponding text label features using an image-text alignment model). If the score is greater than or equal to 0.3, the mask is considered semantically correct and meets the segmentation quality standard, and is retained; if the score is less than 0.3, the mask is considered to have a semantic conflict with the original category label of the detection box or to have serious false detections, and the mask sample is directly removed from the set. After this rigorous filtering operation, the remaining high-quality masks and corresponding instances constitute the first candidate dataset for subsequent multi-scale clustering.

[0082] This application's embodiments align and match image and text features (semantic gating mechanism). Compared to screening methods that rely solely on geometric overlap (IoU), this mechanism calculates the semantic similarity between mask image features and category text features. This allows it to accurately identify and eliminate low-quality mask samples that, while geometrically overlapping the detection box, have semantically incorrect content (such as category confusion or background missegmentation), efficiently removing noise and erroneous samples from the automatic annotation process. This process requires no manual intervention and yields massive amounts of high-precision instance segmentation annotation data, effectively compensating for the performance loss caused by random initialization of segmentation head parameters in downstream tasks and significantly improving the model's segmentation accuracy for complex scenes such as edges and occlusions. It also ensures that the final retained first candidate dataset has extremely high semantic purity and annotation confidence, avoiding interference from noisy data in subsequent model training. Furthermore, this semantically filtered, high-quality mask data enables the model to learn more robust instance feature representations during training, thereby improving its generalization segmentation performance when facing unseen complex scenes.

[0083] In step 103, based on the mask corresponding to the instance in the instance mask dataset, detection boxes and corresponding category labels are added to the instances in the instance mask dataset, and the instance mask dataset with added detection boxes and category labels is used as the second candidate dataset.

[0084] Here, the second candidate dataset refers to a dataset that is based on a large-scale classless masked dataset (such as SA-1B), and supplemented with semantic categories (represented by the category labels corresponding to the detection boxes) and detection boxes through automatic annotation and filtering mechanisms.

[0085] In some embodiments, the instance mask dataset is derived from a massive unlabeled mask library (e.g., SA-1B containing approximately 1 billion images), which natively contains only geometric masks without category semantics.

[0086] In some embodiments, adding detection boxes and category labels can be achieved by using a general detection model (such as DETR pre-trained on Objects365) and a downstream task-specific model to predict the image and generate candidate detection boxes and categories.

[0087] As an example, see Figure 8In constructing the second candidate dataset for post-pre-training, a large-scale instance mask dataset (e.g., the SA-1B dataset) is used as the original data source. Since the original images in this dataset (e.g., input layer Image A) only contain instance pixel masks without class attributes, lacking necessary semantic labels and geometric localization information, this embodiment employs a multi-model collaborative approach for automated information completion. Specifically, the original images are input into a pre-trained general detection model (trained based on Objects365), downstream task A model, or downstream task B model for processing. These models extract features and identify instances in the images, generating corresponding bounding boxes (as shown by the dashed boxes in the figure) and class confidence labels (e.g., "screw: 0.9", "gear: 0.9"). Through these steps, the original unsupervised mask data is transformed into structured data containing high-precision masks, bounding boxes, and multi-source class labels, thereby constructing a second candidate dataset with high coverage and rich semantics, solving the problem of lack of class supervision signals in large-scale mask data.

[0088] In some embodiments, see Figure 9 In step 103, based on the mask corresponding to the instance in the instance mask dataset, a detection box and the corresponding category label are added to the instance in the instance mask dataset, and the instance mask dataset with the added detection box and category label is used as the second candidate dataset. This can be achieved by executing the following steps 1031 to 1034.

[0089] In step 1031, multiple candidate detection boxes and candidate category labels corresponding to the candidate detection boxes are determined for each instance in the instance mask dataset.

[0090] Here, the candidate bounding box and candidate class label refer to the target location and semantic category information that have been initially predicted by the automatic annotation algorithm but have not yet undergone consistency verification.

[0091] In some embodiments, the instance mask dataset can be a large-scale classless mask library containing a massive number of images (e.g., the SA-1B dataset containing approximately 1 billion images), which natively provides only pixel-level geometric masks but lacks corresponding bounding boxes and category semantics.

[0092] In some embodiments, determining candidate bounding boxes and candidate class labels can be achieved by jointly invoking multiple pre-trained detection models. For example, open-source general object detection models (such as the DETR detector pre-trained on Objects365) and task models trained for specific downstream businesses (such as business A model and business B model) can be used to perform forward inference prediction on images in the instance mask dataset. The coordinates of all bounding boxes and their corresponding class predictions output by these models are then aggregated as candidate labels for mask instances in the image.

[0093] In step 1032, the intersection-union ratio (IUR) between the candidate detection box and the corresponding mask for each instance is calculated, and candidate detection boxes with IUR lower than the set second threshold are removed to obtain the initial second candidate dataset.

[0094] Here, Intersection over Union (IoU) is a geometric metric that measures spatial overlap and is used to determine whether automatically predicted candidate bounding boxes are precisely aligned in spatial location with the native instance mask.

[0095] In some embodiments, since the automatic annotation model may generate a large number of background false detections or location offsets, directly assigning a mask to the candidate detection box can lead to a serious "box-mask mismatch" problem.

[0096] In some embodiments, calculating the intersection-over-union ratio (IoU) and removing low-quality candidate boxes can be achieved as follows: First, obtain the minimum bounding rectangle of the native mask. Then, calculate the ratio of the overlap area between this bounding rectangle and the candidate detection boxes predicted by the model to the total union area. Compare this IoU with a set second threshold (e.g., 0.8), retaining only targets that are simultaneously covered by both the candidate detection boxes and the native mask, and whose IoU is greater than or equal to the second threshold. Through this geometric consistency check, predicted boxes with positioning errors or unrelated to the native mask are filtered out, thus obtaining the initial second candidate dataset after preliminary cleaning.

[0097] In step 1033, the features of candidate detection boxes and the features of corresponding candidate category labels are extracted from the initial second candidate dataset, and the second similarity score between the features of the candidate detection boxes and the features of the corresponding candidate category labels is calculated.

[0098] Here, the second similarity score is an indicator that measures the degree of cross-modal semantic matching between local image visual features and given text labels, and is used to determine whether the candidate category label truly describes the object within the detection box.

[0099] In some embodiments, after geometric filtering in step 1032, although the remaining candidate detection boxes are aligned with the mask in position, their predicted candidate category labels may still be incorrect due to classification errors in the general detection model or downstream model.

[0100] In some embodiments, the calculation of the second similarity score can be achieved by introducing a large-scale image-text multimodal model. Specifically, the local image of the corresponding instance is cropped from the original image based on the candidate detection boxes and fed into the visual encoder of the large-scale image-text multimodal model to extract visual feature vectors; simultaneously, the candidate category label text is fed into the text encoder of the large-scale image-text multimodal model to extract text feature vectors. Then, the cosine similarity (second similarity score) between these two high-dimensional feature vectors is calculated to obtain the second similarity score used to characterize the degree of semantic alignment.

[0101] In step 1034, candidate detection boxes and corresponding candidate category labels with second similarity scores lower than the set third threshold are removed from the initial second candidate dataset to obtain the second candidate dataset.

[0102] Here, the third threshold refers to the lower limit of judgment used to perform strict semantic gating, and its value is usually related to the size of the underlying dataset and the expected label confidence.

[0103] In some embodiments, since the instance mask dataset (such as SA-1B) is extremely large (far larger than Objects365), relatively strict semantic filtering criteria are required in order to control the final retained data size in the massive amount of data and significantly improve the accuracy of automatic annotation.

[0104] In some embodiments, removing data below a threshold can be achieved by comparing the second similarity score calculated in step 1033 with a set third threshold (e.g., 0.5). If the score is below 0.5, the image content within the candidate detection box is considered to have a significant semantic conflict with the candidate category label and is therefore removed; if the score is greater than or equal to 0.5, it is determined to be a high-confidence label and retained. After this rigorous semantic verification, the final output is a second candidate dataset consisting of high-precision "mask-detection box-category label" triples. It should be noted that since the SA-1B dataset is much larger than the Objects365 dataset, using a third threshold higher than the second threshold to filter the second similarity score helps control the total amount of data and improve the label confidence, further weakening the impact of automatic labeling errors and removing instances irrelevant to the target category.

[0105] This application addresses the issue of poorly labeled data that often arises in automated model prediction schemes. It uses the native mask boundary as an objective spatial physical benchmark and employs the Intersection over Union (IoU) metric to achieve rigorous geometric consistency verification, accurately filtering out samples with location offsets, background false detections, and "box-mask" spatial mismatches. Secondly, it introduces cross-modal image-text alignment capabilities and achieves deep semantic consistency verification through feature similarity scoring, accurately filtering out samples with incorrect classifications or inconsistencies between images and text. This dual approach of spatial localization and semantic classification improves the accuracy of the final generated second candidate dataset, thereby enhancing the stability and convergence of subsequent model training and the upper limit of final generalization performance.

[0106] In step 104, features of the seed dataset, the first candidate dataset, and the second candidate dataset are extracted, and clusters are performed on the features of the seed dataset, the first candidate dataset, and the second candidate dataset according to multiple set scales, resulting in first clusters, second clusters, and third clusters at multiple scales.

[0107] Here, "multi-scale" refers to setting different numbers of cluster centers (K value) during the clustering process to cover different semantic distribution levels from coarse-grained to fine-grained, thus constructing a multi-level index structure.

[0108] In some embodiments, feature extraction can be performed using a large visual model, mapping instances in an image to feature vectors of fixed dimensions.

[0109] In some embodiments, feature clustering can be implemented using the K-Means algorithm. As an example, for the feature vectors of the seed dataset, K values ​​can be set to 100, 500, 1k, 5k, 10k, etc. For the first and second candidate datasets, which are larger than the seed dataset, to ensure that a spatially uniform and more comprehensive augmented dataset (essentially using the augmented dataset as a new seed dataset) can be obtained, the feature vectors in the first and second candidate datasets are clustered using the same K value as the seed dataset. By calculating the cosine similarity or Euclidean distance between feature vectors, the feature vectors are assigned to their corresponding cluster centers, thereby constructing the first cluster (corresponding to the features of the seed dataset), the second cluster (corresponding to the features of the first candidate dataset, i.e., the dataset annotated with the instance detection box dataset), and the third cluster (corresponding to the features of the second candidate dataset, i.e., the dataset annotated with the instance mask dataset) for subsequent retrieval.

[0110] In step 105, for each scale of the first cluster, a set number of first samples are retrieved.

[0111] Here, the first sample refers to a representative instance sample extracted from the clustering results of the seed dataset, which is used as an anchor to retrieve large-scale external data.

[0112] In some embodiments, the first cluster is constructed based on the feature distribution of downstream business data, reflecting the categories and forms that the target task focuses on.

[0113] In some embodiments, retrieving a predetermined number of first samples can be achieved through uniform sampling within clusters. For example, traversing the first cluster at each scale, k samples are randomly selected from each cluster (e.g., k=4). It should be noted that if the total number of samples in a cluster is less than k, all samples in that cluster are retained. This sampling strategy ensures that when retrieving second samples from the second and third clusters, samples from long-tail categories and common categories have relatively balanced weights, avoiding data distribution skew and ultimately resulting in a first augmented dataset with balanced distribution and more comprehensive category coverage.

[0114] In step 106, a first augmented dataset is constructed based on second samples from the second and third clusters whose similarity to the first sample is greater than a set threshold, and the seed dataset.

[0115] Here, the first augmented dataset refers to a dataset specifically designed for the post-pre-training stage, which combines the original downstream data (since the seed dataset contains instances of multiple categories, it serves as the original downstream data here) with high-quality external data augmented through cross-database retrieval (first candidate dataset and second candidate dataset).

[0116] In some embodiments, the acquisition of the second sample depends on a pre-built multi-scale clustering index.

[0117] As an example, the first augmented dataset can be constructed as follows: The feature vector of the first sample is used as the query vector. A nearest neighbor search is performed on the cluster center indices of the second and third clusters to find the target cluster with the smallest distance. Then, an equal number of samples (e.g., N=4 samples) are taken within the target cluster to form the second sample. This method expands the downstream data size by approximately 8 times, and while maintaining semantic alignment (class alignment) with the downstream task, it introduces diversity in appearance and context. Finally, the original seed dataset is merged with the retrieved second samples to form a balanced first augmented dataset.

[0118] In some embodiments, see Figure 10 Step 106 can be achieved through the following steps 1061 to 1063.

[0119] In step 1061, for each first sample, a first distance metric is calculated between the cluster center in the second cluster and the first sample, and a second distance metric is calculated between the cluster center in the multiple third clusters and the first sample. The first and second distance metrics are used to measure similarity.

[0120] Here, distance metric refers to a numerical indicator that measures the similarity between two feature vectors in a vector space. The smaller the distance, the higher the similarity between the two samples in terms of semantic features, visual texture, or geometric topology.

[0121] In some embodiments, the first sample features, second cluster centers, and third cluster centers to be computed are typically vectors uniformly mapped to the same high-dimensional feature space (e.g., a 384-dimensional CLS feature space). These feature vectors are extracted through a pre-trained self-supervised visual model, which can effectively capture the global semantic information of instances.

[0122] In some embodiments, the calculation of the first and second distance metrics can be achieved by calculating the cosine similarity or Euclidean distance between the feature vector of the first sample and the candidate cluster center vector. For example, using the formula Calculate the cosine distance, which is used as a standard to measure the degree of affinity between the first sample and each cluster.

[0123] In step 1062, for each first sample, the cluster with the smallest distance to the first sample in the second cluster is determined according to the first distance metric, and the cluster with the smallest distance to the first sample in the third cluster is determined according to the second distance metric.

[0124] Here, the cluster with the smallest distance refers to the set of candidate samples that are semantically closest to and visually best match the current seed sample (first sample) in the corresponding clustering index (second cluster and third cluster).

[0125] In some embodiments, since the first cluster, the second cluster, and the third cluster are all constructed using the same preset scale (i.e., containing hierarchies with different K values), when matching, the cluster with the smallest distance is found for the first sample of each cluster scale.

[0126] In some embodiments, determining the cluster with the smallest distance can be achieved by sorting the distance vectors calculated in step 1061 in ascending order and extracting the cluster number with the first index value (i.e., Top-1). This process essentially maps each instance sample in the downstream business to the corresponding local manifold space in a large-scale external candidate library, establishing an index link for subsequent precise expansion.

[0127] In step 1063, a set number of second samples are retrieved from the second cluster and the third cluster that are closest to the first sample, and the second samples are combined with the seed dataset to construct the first augmented dataset.

[0128] Here, the first augmented dataset refers to a hybrid dataset formed by nearest neighbor matching and equal sampling (from the second and third clusters), which combines the characteristics of downstream business distribution with the diversity of open source data. Its size is usually several times that of the original seed dataset.

[0129] In some embodiments, the set quantity (e.g., N=4) is a hyperparameter used to control the proportion of data expansion. The retrieval process is performed within a defined minimum distance cluster. Intra-cluster sampling ensures that the introduced second samples maintain semantic alignment while having different lighting, viewing angles, or background interference.

[0130] In some embodiments, the specific execution logic for constructing the first augmented dataset is as follows: For each seed instance, N samples are randomly selected from the matched second cluster (from the first candidate dataset), and N samples are randomly selected from the matched third cluster (from the second candidate dataset); subsequently, all the selected second samples are concatenated with the original seed dataset. If the seed dataset contains 200k instances and N=4, the final constructed first augmented dataset will contain approximately 1.8M high-quality instance samples, thereby achieving approximately 8 times the effective data expansion.

[0131] This embodiment of the application retrieves an equal number of samples from the second and third clusters based on the first sample (samples retrieved from the features of the seed dataset). This ensures that the number of samples in the first sample is evenly expanded, regardless of whether the samples in the seed dataset are majority (head categories) or minority (long-tail categories). This equal expansion mechanism preserves the original category ratio structure of the seed dataset and avoids the "long-tail distribution imbalance" phenomenon common in large-scale data retrieval (i.e., common category samples are over-recalled while rare categories are submerged). This ensures a high degree of balance in the category distribution of the final generated first augmented dataset, providing a foundation for the model to learn various features in a balanced manner. Furthermore, selecting a "set number" of samples from the cluster with the smallest distance ensures that the final first augmented dataset maintains semantic consistency (derived from the nearest cluster) while introducing rich appearance diversity of samples within the cluster (derived from different second samples within the cluster). This strategy allows the model to learn the core semantic features of the category during training, while also being exposed to diverse backgrounds, lighting, and perspective changes. This improves the generalization ability of the first model without disrupting the data distribution balance, thereby enhancing its adaptability as a teacher model for training downstream tasks.

[0132] In step 107, the first model is trained based on the first augmented dataset.

[0133] The first model is used as a teacher model to guide the training of downstream task models.

[0134] Here, the first model (i.e., the teacher model) refers to a deep learning model that has been optimized through a "post-pre-training" stage between general pre-training and specific downstream fine-tuning, and has the ability to generalize across multiple downstream tasks.

[0135] In some embodiments, the first model may be an instance segmentation model based on the Transformer architecture, whose initial weights are derived from a large-scale object detection pre-trained model.

[0136] In some embodiments, training the first model can be achieved by supervising the model using a pre-constructed first augmentation dataset. The training loss function may include classification loss (e.g., Focal Loss), bounding box regression loss (e.g., L1 Loss and GIoU Loss), and mask segmentation loss (e.g., Dice Loss). After completing this stage of training, the first model can be used as a teacher model to guide the training of smaller edge models (i.e., the fourth model) through knowledge distillation, or as a high-starting-point pre-trained model for minor fine-tuning on specific downstream tasks A or B, thereby improving the model's convergence speed and final detection and segmentation accuracy.

[0137] This application embodiment obtains a seed dataset containing annotations (masks, bounding boxes, and category labels) of instances of various categories, and supplements it with datasets that have different partial annotations. A highly class-aligned augmented dataset is then retrieved from the supplemented dataset based on the seed dataset. Pre-training the first model using the augmented dataset reduces the differences between the dataset and the downstream task in terms of category distribution, appearance, and background features. This allows the first model (teacher model) to have domain adaptability to the downstream task before encountering it, enabling accurate identification of specific instances in the downstream task. This reduces the risk of decreased model generalization performance due to inconsistent data distribution and improves the accuracy of model training. Furthermore, by performing multi-scale clustering on the features of the seed dataset, the first candidate dataset, and the second candidate dataset at the same scale, and then performing equal sampling within each cluster on the results of the multi-scale clustering, a first augmented training set with balanced feature distribution is constructed. At each set scale, based on the first sample in the first cluster, an equal number of samples are retrieved from the second and third clusters. This forces the model trained on the first augmented dataset to focus on long-tail categories and hard examples, thus avoiding the problem of the model overfitting to common head categories and ignoring rare categories during large-scale data training.

[0138] In some embodiments, see Figure 11 After step 107, the fourth model can also be trained by performing steps 110 to 112 using the first model.

[0139] In step 110, the seed dataset is processed by the first model to obtain the teacher-annotated dataset, which includes a mask, detection boxes, and category labels.

[0140] Here, the first model usually refers to a general-purpose large model with strong generalization ability and high-precision segmentation performance, which is obtained after being fully trained on the large-scale augmented dataset in the aforementioned steps. It plays the role of "Teacher Model" in the knowledge distillation architecture. The teacher-annotated dataset is a set of high-quality pseudo-labels generated by the first model after performing forward inference on unlabeled or partially labeled data.

[0141] In some embodiments, after the images in the seed dataset are input into the first model, the first model outputs the prediction confidence, class label, bounding box coordinates, and pixel-level mask for each instance. To ensure the high quality of the teacher-annotated dataset, a high confidence threshold (e.g., 0.7 or 0.8) and a non-maximum suppression (NMS) operation are typically set to retain only the instance annotation results with high prediction scores and no redundancy.

[0142] In step 111, the seed dataset is processed by the fourth model to obtain the student labeled dataset.

[0143] The fourth model is a downstream task model of the first model.

[0144] Here, the fourth model refers to the network model designed for actual business needs (such as the deployment of downstream tasks on edge devices and real-time inference scenarios). Its network structure or number of parameters is usually smaller than that of the first model (in order to enable edge deployment). In the knowledge distillation architecture, it plays the role of "student model". The student-annotated dataset refers to the prediction output of the student model on the input image under the current network weights.

[0145] In some embodiments, the fourth model may employ Mask R-CNN or other efficient instance segmentation architectures built with a lightweight backbone network such as ResNet-50 or MobileNet. During training iterations, the same seed dataset images as in step 111 (or images after specific data augmentation) are input into the fourth model to perform forward propagation, thereby obtaining its predicted values ​​for the mask, detection boxes, and categories of the batch of images.

[0146] In step 112, the model parameters of the fourth model are updated based on the teacher-annotated dataset, the seed dataset, and the student-annotated dataset.

[0147] Here, updating model parameters refers to the optimization process of adjusting the weights of the fourth model network by calculating the error loss between the predicted values ​​(student-annotated dataset) and the supervised targets (original ground truth values ​​of teacher-annotated dataset and seed dataset) and using the backpropagation algorithm.

[0148] In some embodiments, the training process in this step can employ a “strong-weak consistency learning” strategy in semi-supervised learning: for the same seed image, the teacher model (first model) processes the images in the seed dataset to generate a stable and reliable teacher-labeled dataset (i.e., as a supervision signal or soft label); while the student model (fourth model) processes the images in the seed dataset to output the student-labeled dataset.

[0149] In some embodiments, the model parameters are updated as follows: the teacher-annotated dataset is treated as a "pseudo ground truth," and combined with the original manually annotated ground truth that may exist in the seed dataset, an overall loss function is calculated between the teacher-annotated dataset and the student-annotated dataset. This overall loss function typically includes the cross-entropy loss for category prediction, the L1 or GIoU regression loss for bounding box localization, and the Dice loss for mask generation. Subsequently, the parameters of the fourth model are updated based on the gradient of the loss function using an optimizer (such as SGD or AdamW). As iterations proceed, the "latent knowledge" and generalization ability contained in the first model are effectively distilled and transferred to the fourth model, which has relatively fewer parameters than the first model.

[0150] This application embodiment utilizes a first model trained on a first augmentation dataset as a teacher model. The implicit knowledge and generalization ability learned by the first model from the sufficiently large, balanced, and diverse first augmentation dataset are transferred to the fourth model through a teacher-labeled dataset. Although the fourth model is trained only on a limited seed dataset, it implicitly inherits the rich feature distribution and anti-interference logic inherent in the first augmentation dataset by learning the output of the first model. This allows the downstream task model (the fourth model) to obtain high-precision feature extraction and scene understanding capabilities without directly processing massive augmentation data, significantly reducing the training resource consumption of the downstream task. Furthermore, post-pre-training based on the first augmentation dataset constructed in this scheme enables the first model (teacher model) to possess excellent feature extraction and segmentation capabilities before entering the downstream fine-tuning stage. The post-pre-trained model converges faster during downstream task fine-tuning, requires less labeled data, and shows improvements in final detection and segmentation metrics (such as AP and mAP). Furthermore, as a guide for knowledge distillation, this teacher model can provide more accurate soft labels and feature supervision for lightweight edge models, effectively resolving the contradiction between the limited computing power of edge devices and the need for high precision.

[0151] In some embodiments, see Figure 12 The first model can also be trained through steps 113 to 117.

[0152] In step 113, the mask, detection box, and category label of the third downstream dataset are obtained through the trained first model to obtain the downstream candidate dataset.

[0153] In some specific embodiments, when a new downstream task C is introduced into a practical industrial application, it is necessary to incrementally supplement the post-pre-training dataset. In this case, the third downstream dataset (the dataset corresponding to the new downstream task) mainly includes a large-scale classless mask library (e.g., the SA-1B dataset containing approximately 1 billion images). As an example, the first model automatically labels the SA-1B data, generating initial detection boxes and class labels. In this embodiment, automatic labeling can also be performed using an open-source general detection model, or the third downstream dataset can be labeled using the second and third models. Subsequently, the IoU between the automatically labeled results and the native SA-1B mask is calculated to filter spatially mismatched regions, and a text-image alignment model is further used to calculate the similarity between the instance region and the text label (e.g., retaining samples with a retention threshold ≥ 0.5) for strict semantic gating. The high-quality instance segmentation data retained after the above dual screening of consistency and semantics constitutes the downstream candidate dataset for task C.

[0154] In step 114, the downstream candidate dataset is clustered according to multiple set scales to obtain the fourth cluster.

[0155] In some embodiments, to establish an efficient retrieval structure in massive datasets, a self-supervised visual large model is used to extract global semantic features (e.g., 384-dimensional CLS token feature vectors) from all samples in the downstream candidate dataset. Subsequently, the K-Means algorithm is used to perform multi-granularity clustering operations on these feature vectors. The number of cluster centers K can be set to different scales such as 100, 500, 1k, 5k, and 10k, depending on the data size (the same as the clustering scale of the seed dataset). Through this multi-scale clustering, a large-scale instance mask cluster index specifically for the new task C is generated, i.e., the fourth cluster.

[0156] In step 115, for each first sample, a third distance metric is calculated between the cluster center in the fourth cluster and the first sample, and the cluster in the fourth cluster with the smallest distance to the first sample is determined based on the third distance metric.

[0157] It should be noted that the "first sample" here can be a sample retrieved from the seed datasets of both business A and business B, or it can refer to a new "seed" dataset specifically constructed for the new task C (e.g., a small amount of real labeled data based on task C, or a class-aligned, evenly distributed set obtained by cross-completing missing labels using multiple downstream models). In this step, the first sample of task C is used as the query vector, and cosine similarity is used as the third distance metric to calculate its high-dimensional spatial distance with each cluster center in the fourth cluster mentioned above, thereby accurately locating the cluster in the large-scale candidate library that best matches the specific seed of task C in terms of semantics and appearance.

[0158] In step 116, a set number of third samples with a similarity greater than a set threshold to the first sample are retrieved from the fourth cluster and the second cluster that have the smallest distance to the first sample. The retrieved third samples and the seed dataset are then used to form the second augmented dataset.

[0159] In some embodiments, this step performs cross-database nearest neighbor matching and augmentation. On one hand, using the instance mask seed of downstream task C as the query, the nearest cluster is matched in the newly added SA-1B mask cluster (the fourth cluster) and intra-cluster equal sampling is performed (e.g., N=4 samples are sampled per cluster). On the other hand, using the detection box seed of downstream task C as the query, the same matching and sampling are performed in the existing open-source detection cluster (e.g., the second cluster built based on Objects365 and retaining the original settings). By fusing the third samples retrieved from the two heterogeneous feature libraries, and the new seed dataset of task C itself, a second augmented dataset containing "existing task + newly added task C" in the category space is constructed, achieving a significant expansion of the downstream sample size (e.g., an expansion of approximately 8 times) while maintaining the original data distribution.

[0160] In step 117, the first model is trained based on the second augmented dataset.

[0161] In some embodiments, step 117 involves incremental adaptation training of the teacher model (first model). The first model can be incrementally fine-tuned using a second augmented dataset that incorporates knowledge from the domain of task C, by loading existing network weights (i.e., weights that have already learned general features and features of downstream tasks A and B). Through this continuous learning method of loading existing weights, the first model can quickly adapt to the specific categories and scenario distributions of the newly added task C without forgetting its original task capabilities, thus further evolving its generalization ability.

[0162] This application embodiment reuses the existing index of the second cluster, requiring only automatic annotation and construction of new clusters (fourth clusters) for the new task (third downstream dataset) to complete dataset augmentation and obtain the second augmented dataset. After loading the existing weights and training on this incremental dataset, the first model not only quickly absorbs the feature representation of task C, but also maintains cross-task knowledge transfer capability through a shared general feature base, achieving simultaneous improvement in generalization ability and robustness in multiple fragmented industrial dataset scenarios (such as different robot grasping tasks). In addition, the effective training data scale of the new task is expanded without the cost of manual annotation, and the balanced class distribution and consistent instance attributes of the expanded data are ensured, solving the long-tail distribution and model overfitting problems that are easily encountered in newly added small industrial sample tasks.

[0163] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0164] In related technologies, the common practice for training object detection or instance segmentation models is "pre-training on a general dataset + fine-tuning on a specific business dataset". Specifically, firstly, a large-scale open-source general dataset (such as an image library with target location annotations) is used to pre-train the initial model (equivalent to the first model), enabling it to possess basic feature extraction and general target localization capabilities. Then, for specific downstream business scenarios such as humanoid robot environmental perception and industrial parts grasping, a small number of real-world scene images (or supplemented with some computer simulation images) are collected and manually annotated to construct a downstream task dataset. Based on this, the pre-trained model is directly fine-tuned and updated using this downstream task dataset. Alternatively, to adapt to the computing power and latency limitations of edge devices (such as robot controllers), knowledge distillation techniques are used, using the pre-trained large-scale complex model as guidance to transfer features to a lightweight edge model with smaller parameters, aiming to meet the perception and reasoning needs of specific scenarios.

[0165] However, open-source pre-training datasets and specific downstream business data often exhibit significant domain gaps in terms of target categories, visual appearance, surface texture, and background distribution. Furthermore, high-quality instance-level pixel annotations are extremely costly, resulting in a limited scale of real-world labeled data available for downstream applications. Therefore, directly fine-tuning the model based on downstream datasets with disparate distributions and small scales can easily lead to overfitting to a small number of specific samples. It can also cause a "catastrophic forgetting" of the generalization ability learned during pre-training. Moreover, even with the introduction of simulation data, the differences in visual representation of lighting and materials between simulation data and the real physical world are difficult to completely eliminate. These combined factors result in existing models struggling to achieve effective capability adaptation and transfer when faced with multiple independent downstream tasks lacking sufficient annotations. They exhibit weak cross-scene generalization capabilities and are unable to provide high-precision, high-stability target detection and instance mask segmentation results in complex industrial applications.

[0166] The following describes the specific implementation process of the target detection model training method provided in the embodiments of this application.

[0167] Step 1: Construct a seed dataset using the downstream labeled data.

[0168] Taking the existence of Business A dataset and Business B dataset downstream as an example, the following explanation is provided. The Business A dataset and Business B dataset together contain approximately 200k images, and these images have been manually labeled with different instance categories for Business A and Business B, respectively.

[0169] By using a downstream model trained on business dataset A to automatically label business dataset B, and then using a downstream model trained on business dataset B to label business dataset A, the category labeling is mutually complementary, thereby expanding the range of labeled categories. It should be noted that in the post-pre-training stage, the goal is to adapt the teacher model (the first model) to the downstream task distribution; therefore, a small number of errors from automatic labeling will not affect the overall training effect.

[0170] Step two: Expand the open-source dataset.

[0171] In this embodiment, since instance segmentation annotation is more cumbersome than object detection annotation, there is currently no large-scale open-source instance segmentation dataset that can be directly used for related tasks. Therefore, it is necessary to construct large-scale instance segmentation data using an automatic annotation method. In the field of object detection, Objects365 is usually used as a pre-training dataset, and instance segmentation models also use Objects365 to train their object detection-related structures; however, in subsequent training or fine-tuning stages, the segmentation-related parameters still need to be randomly initialized, which limits the performance of the model.

[0172] The detection boxes provided in the Objects365 (instance detection box dataset) can be used as input to call the Segment Anything (SAM) model to generate a corresponding instance segmentation mask for each detection box, thereby expanding the original object detection annotations into instance-level mask annotations (the first candidate dataset before filtering). The class space of the generated instance segmentation dataset (i.e., the first candidate dataset before filtering) consists of the original Objects365 detection box classes (365 classes) and the classes (N_A, N_B) contained in the downstream A and B business datasets, and the total number of classes can be represented as 365 + N_A + N_B. While ensuring class diversity and richness, it also ensures that the constructed dataset covers the specific classes required by the downstream tasks, providing sufficient semantic and instance information support for the subsequent post-pre-training stage.

[0173] Currently, the mainstream large-scale classless masked dataset is SA-1B (Instance Masked Dataset), containing approximately 1 billion images. However, due to the lack of category information, it is difficult to directly use it for instance segmentation training. Therefore, we employ open-source general detection models (such as the DETR detector pre-trained on Objects365) and downstream A and B models to automatically label SA-1B images. Since SA-1B is larger and more diverse than Objects365, downstream models can provide sufficient and highly comprehensive candidate samples, which will be referred to as the instance detection dataset (i.e., the second candidate dataset before filtering).

[0174] Step 3: Filter and clean the expanded data to build an enhanced dataset.

[0175] A self-supervised large-scale visual model was used to extract features from all bounding boxes and corresponding instance regions in the seed dataset, resulting in a 384-dimensional CLS token feature vector. The self-supervised large-scale visual model effectively extracts global semantic features of instances, and objects of different categories exhibit good separability in the high-dimensional feature space. Subsequently, the K-Means algorithm was used to cluster these features, specifically setting the number of clusters K to 100, 500, 1k, 5k, and 10k, and using cosine similarity as the feature distance metric. K-Means was run multiple times to obtain clustering results at different scales. At each clustering scale, data was sampled with a cluster size of k=4; if a cluster had fewer than k samples, all samples were retained.

[0176] The instance segmentation dataset generated in step two already contains bounding boxes before step two, and masks are generated in step two. A text-image alignment model is used to simultaneously extract features from the generated masks and the text labels (the category labels corresponding to the bounding boxes). The extracted features are mapped to the same high-dimensional space, and the cosine similarity between the mask features and the text label features is calculated in this high-dimensional space. Based on the cosine similarity, the similarity (or consistency) between the instance mask and the category label is determined. Only samples with a similarity score greater than 0.3 are retained, thus eliminating low-quality annotations caused by category confusion or false region detection within the bounding boxes. Objects365's bounding box category distribution is relatively accurate, but it lacks instance-level mask annotations. By generating masks using the SAM method described above and combining it with the text-image alignment model for filtering, samples with inconsistent categories within the bounding boxes can be effectively removed, and higher-quality instance annotations can be selected. For the filtered samples, K-Means clustering method is used for clustering. The number of clusters K is set to different scales such as 500, 1k, 5k, 10k, and 500k to construct a multi-granularity cluster instance detection box cluster structure (second cluster).

[0177] The instance detection dataset generated in step two already has a mask before step two, and detection boxes and their corresponding class labels were generated in step two. Since step two automatically annotates the SA-1B image using a general detection model (e.g., a DETR detector pre-trained on Objects365) and downstream A and B models, multiple candidate detection boxes and their corresponding class labels are generated. Since the candidate detection boxes may have excessively large errors or be of low quality, the intersection-over-union ratio (IoU) of the original mask and the generated candidate detection boxes can be calculated. Only targets simultaneously covered by both the detection box / instance annotation and the SA-1B mask are retained, filtering out mask errors and regions unrelated to the target class, thus reducing subsequent computation and storage overhead. After this step, the annotated class space is also denoted as 365+N_A+N_B, completing the initial screening. After initial screening, the instance detection dataset is processed using an image-text alignment model to extract features from both the generated bounding boxes and text labels (the category labels corresponding to the bounding boxes). The extracted features are then mapped to the same high-dimensional space, and the cosine similarity between the bounding box features and the text label features is calculated. Samples with a similarity score ≥ 0.5 are retained. Since SA-1B is significantly larger than Objects365, a higher similarity threshold is used during screening to control the total data volume and improve annotation confidence, thereby further mitigating the impact of automatic annotation errors and identifying instances unrelated to the target category. After semantic screening based on the image-text alignment model, a self-supervised large-scale visual model is used to extract features from all samples and save 384-dimensional CLS vectors. K-Means clustering is then employed, with the number of clusters K set to different scales (500, 1k, 5k, 10k, 500k, etc.) to construct a large-scale instance mask cluster (third cluster) index, providing a foundation for subsequent retrieval and expansion.

[0178] Using the seed dataset as the query source, feature matching retrieval is performed on large-scale instance mask clusters and instance detection box clusters. As an example, at each cluster scale, k=4 samples (i.e., the first samples in the above embodiment) are sampled within each cluster of the seed dataset. At the corresponding scales of the large-scale instance mask clusters (i.e., the second cluster in the above embodiment) and the instance detection box clusters (i.e., the third cluster in the above embodiment), the distance between the sample and the cluster center of each cluster is calculated to match the nearest cluster. N=4 augmented samples are then sampled within each nearest cluster and returned, completing the cross-database nearest neighbor matching and intra-cluster equal sampling process. By performing cross-database nearest neighbor matching and intra-cluster equal sampling on the large-scale instance mask clusters and instance detection box clusters based on samples collected from the seed dataset, it is possible to expand the downstream data scale by 8 times (i.e., 2N, the specific number can be set as needed) and expand the instance types to (365+N_A+N_B classes) while maintaining the original distribution and instance mask attributes of the seed dataset, resulting in an enhanced dataset, and ensuring semantic alignment and appearance diversity with downstream categories.

[0179] Step 4: Perform post-training using augmented datasets.

[0180] After pre-training the teacher model for object detection, it undergoes initial training using the augmented dataset obtained in step three. Subsequently, the pre-trained teacher model is fine-tuned using datasets from specific downstream tasks. Because the teacher model was initially trained on the augmented dataset, it already possesses domain-specific adaptability before encountering the actual downstream task datasets, thus enabling accurate identification of specific instances within the downstream tasks. It should be noted that a sample quality score is constructed by combining mask stability, similarity margin, and retrieval density. This sample quality score can be used for loss weighting or sample filtering during the training of the teacher model.

[0181] If a new downstream task C is subsequently introduced, the dataset is incrementally supplemented: multiple downstream models are used to supplement missing annotations (e.g., downstream models of business A and business B generate detection boxes, masks, and category labels for datasets containing business C), and a new seed dataset is constructed based on the newly added detection boxes and instance annotations; downstream model C is used to automatically annotate and perform consistency / semantic filtering on large-scale mask data (SA-1B) to generate new instance segmentation data clusters; the detection data clusters retain their existing settings. Finally, the downstream seeds are used as queries for matching and sampling in the new clusters, and the new samples are merged into the existing post-pre-training dataset; the teacher model loads the current weights for incremental training to complete the adaptation.

[0182] As an example, see Figure 13The data processing steps for constructing the augmented dataset in this application embodiment include: firstly, cross-labeling and merging of downstream dataset A and downstream dataset B to obtain a downstream multi-task dataset, and then constructing the core downstream seed dataset through K-means clustering (100-10k) and sampling. Subsequently, based on this seed dataset, content-based retrieval augmentation is performed in two parallel external data source paths: The left path uses a large-scale masked classless labeled dataset (instances) (i.e., the second candidate dataset in the above embodiment) to perform multi-model collaborative annotation (automatic annotation by downstream A / B business models or automatic annotation by open-source general detection models), and after filtering by K-means clustering (10k-1M) and image-text alignment model (CLIP score filtering (≥0.3)), the instance segmentation data clusters are retrieved using the "instance" feature in the seed dataset as the query source; The right path uses a large-scale detection box labeled dataset (detection boxes) (i.e., the first candidate dataset in the above embodiment) to perform detection data clusters enhanced by the SAM model (SAM model automatically annotates instance masks), and after filtering by K-means clustering (500-50k) and image-text alignment model (CLIP score filtering (≥0.5)), the "detection box" feature in the seed dataset is retrieved using the query source. Finally, the high-quality augmented data matching detection clusters (sampling N=4) retrieved from the two paths above are aggregated to generate a post-pre-training dataset (i.e., the first augmented dataset in the above embodiment) for improving the model's generalization ability.

[0183] The following description continues to illustrate the exemplary structure of the target detection model training device 255 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the target detection model training device 255 in the memory 250 may include: The seed dataset acquisition module 2551 is used to acquire the seed dataset, which includes instances of multiple categories and the corresponding masks, detection boxes and category labels of the instances; The first candidate dataset acquisition module 2552 is used to generate a corresponding mask for the instances in the instance detection box dataset based on the detection boxes in the instance detection box dataset, and use the instance detection box dataset with generated mask as the first candidate dataset. The second candidate dataset acquisition module 2553 is used to add detection boxes and corresponding category labels to the instances in the instance mask dataset according to the mask corresponding to the instances in the instance mask dataset, and use the instance mask dataset with added detection boxes and category labels as the second candidate dataset. The multi-scale clustering module 2554 is used to extract features of the seed dataset, the first candidate dataset, and the second candidate dataset, and to cluster the features of the seed dataset, the first candidate dataset, and the second candidate dataset according to multiple set scales, thereby obtaining a first cluster, a second cluster, and a third cluster at multiple scales. The retrieval module 2555 is used to retrieve a set number of first samples for each scale's first cluster. The augmented dataset construction module 2556 is used to construct a first augmented dataset based on second samples in the second and third clusters that have a similarity greater than a set threshold with the first sample, as well as a seed dataset. Training module 2557 is used to train a first model based on a first augmented dataset, wherein the first model is used as a teacher model to guide the training of downstream task models.

[0184] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the target detection model training method described in this application.

[0185] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the target detection model training method provided in this application. For example, ... Figure 3 The training method for the target detection model is shown.

[0186] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0187] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0188] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0189] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for training an object detection model, characterized in that, The method includes: Obtain a seed dataset, wherein the seed dataset includes instances of multiple categories and the corresponding masks, detection boxes, and category labels of the instances; Based on the detection boxes in the instance detection box dataset, generate the corresponding mask for the instance in the detection box dataset, and use the instance detection box dataset with the generated mask as the first candidate dataset. Based on the mask corresponding to the instance in the instance mask dataset, add the detection box and the corresponding category label to the instance in the instance mask dataset, and use the instance mask dataset with the added detection box and category label as the second candidate dataset; Features of the seed dataset, the first candidate dataset, and the second candidate dataset are extracted, and clusters are performed on the features of the seed dataset, the first candidate dataset, and the second candidate dataset according to multiple set scales to obtain a first cluster, a second cluster, and a third cluster at multiple scales. For each scale of the first cluster, a set number of first samples are retrieved; The first augmented dataset is composed of second samples from the second cluster and the third cluster whose similarity to the first sample is greater than a set threshold, as well as the seed dataset; A first model is trained based on the first augmented dataset, wherein the first model is used as a teacher model to guide the training of downstream task models.

2. The method according to claim 1, characterized in that, The method further includes: The seed dataset is obtained by performing the following processing: The first downstream labeled dataset is obtained by using a second model trained on the first downstream dataset to obtain the mask, the detection box and the category label corresponding to the instance in the second downstream dataset. The second downstream labeled dataset is obtained by using a third model trained on the second downstream dataset to obtain the mask, the detection box and the category label corresponding to the instance in the first downstream dataset. The first downstream annotation dataset and the second downstream annotation dataset are combined to form the seed dataset.

3. The method according to claim 1, characterized in that, The first augmented dataset, composed of second samples from the second and third clusters whose similarity to the first sample is greater than a set threshold, and the seed dataset, includes: For each of the first samples, a first distance metric is calculated between the cluster center in the second cluster and the first sample, and a second distance metric is calculated between the cluster center in the plurality of third clusters and the first sample. The first distance metric and the second distance metric are used to measure the similarity. For each of the first samples, the cluster with the smallest distance to the first sample in the second cluster is determined according to the first distance metric, and the cluster with the smallest distance to the first sample in the third cluster is determined according to the second distance metric. A set number of the second samples are retrieved from each of the second cluster and the third cluster, which are the clusters with the smallest distance to the first sample. The second samples are then combined with the seed dataset to construct a first augmented dataset.

4. The method according to claim 1, characterized in that, The detection boxes in the instance detection box dataset are set with corresponding category labels; The step of generating a corresponding mask for the instance in the instance detection box dataset based on the detection boxes in the instance detection box dataset, and using the instance detection box dataset with the generated mask as the first candidate dataset, includes: The images labeled with the detection boxes in the instance detection box dataset are input into the image segmentation model to obtain the mask of the instance; Extract the features of the mask and the features of the category label, and calculate a first similarity score between the features of the mask and the features of the category label; Remove the mask from the first similarity scores that are lower than a set first threshold to obtain the first candidate dataset.

5. The method according to claim 1, characterized in that, The step of adding detection boxes and corresponding category labels to the instances in the instance mask dataset based on the masks corresponding to the instances in the instance mask dataset, and using the instance mask dataset with the added detection boxes and category labels as the second candidate dataset, includes: For each instance in the instance mask dataset, determine multiple candidate detection boxes and candidate category labels corresponding to the candidate detection boxes; Calculate the intersection-union ratio (IUU) between the candidate detection box and the corresponding mask for each instance, and remove the candidate detection boxes whose IUU is lower than a set second threshold to obtain the initial second candidate dataset; Extract the features of the candidate detection boxes and the corresponding features of the candidate category labels from the initial second candidate dataset, and calculate the second similarity score between the features of the candidate detection boxes and the features of the corresponding candidate category labels; Remove the candidate detection boxes and their corresponding candidate category labels from the initial second candidate dataset where the second similarity score is lower than a set third threshold to obtain the second candidate dataset.

6. The method according to claim 1, characterized in that, After training the first model based on the first augmented dataset, the method further includes: The seed dataset is processed by the first model to obtain a teacher-annotated dataset, which includes the mask, the detection box, and the category label. The seed dataset is processed by the fourth model to obtain the student labeled dataset, wherein the fourth model is a downstream task model of the first model; The model parameters of the fourth model are updated based on the teacher-annotated dataset, the seed dataset, and the student-annotated dataset.

7. The method according to claim 2, characterized in that, After training the first model based on the first augmented dataset, the method further includes: The first model, after being trained, is used to obtain the mask, the detection box, and the category label of the third downstream dataset to obtain the downstream candidate dataset; The downstream candidate dataset is clustered according to multiple set scales to obtain a fourth cluster; For each first sample, calculate the third distance metric between the cluster center in the fourth cluster and the first sample, and determine the cluster in the fourth cluster that has the smallest distance to the first sample based on the third distance metric; From the fourth cluster and the cluster in the second cluster that is closest to the first sample, a set number of third samples that have a similarity greater than a set threshold to the first sample are retrieved, and the retrieved third samples and the seed dataset are used to form the second augmented dataset. The first model is trained based on the second augmented dataset.

8. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the target detection model training method according to any one of claims 1 to 7.

9. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the target detection model training method according to any one of claims 1 to 7 is implemented.

10. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the target detection model training method according to any one of claims 1 to 7 is implemented.

Images