Multi-label class balancing method and apparatus thereof

Abstract

A multi-label class balancing method, the method comprising: obtaining a sample set comprising a plurality of instances, each instance comprising one or more class labels; classifying the instances in the sample set according to the class labels to form a plurality of classes, each instance being assigned to one or more classes, each class corresponding to a class label; determining a target sampling weight of each instance in each sub-class according to a number of target class labels in each instance and a number of all class labels in each instance, wherein the sub-class is a class in the plurality of classes and the number of the class is less than a first threshold, the target class label is a class label corresponding to the sub-class in each instance, and the target sampling weight is a sampling weight corresponding to the sub-class in each instance; and performing weighted sampling on the instances in each sub-class according to the target sampling weight of each instance in the sub-class. The present case also provides a device, which can avoid scene loss and indirectly increase the number of other classes.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and computer vision technology, and in particular to a multi-label class equalization method and apparatus thereof. Background Technology

[0002] Object detection is an important research area in computer vision and image processing, and it is currently widely used in fields such as live video streaming, security monitoring, robotics, and human-computer interaction. However, most object detection data in real-world scenarios exhibit a long-tail effect, meaning that the head category constitutes the majority of samples, while the tail category has very few samples. For example, in the 2D dataset BDD, the number of train and car instances differs by a factor of 600, and in the 3D dataset nuScenes, the number of bicycle and car instances differs by a factor of 4. Currently, sampling techniques can address the long-tail effect. However, sampling techniques often lead to scene loss or indirectly increase the number of other categories. Summary of the Invention

[0003] In view of the above, it is necessary to provide a multi-label class balancing method and apparatus, which can avoid scene loss and indirectly increase the number of other categories.

[0004] An embodiment of the present invention provides a multi-label class equalization method, the method comprising: acquiring a sample set including multiple instances, each instance including one or more category labels; classifying the instances in the sample set into multiple classes according to the category labels, each instance being assigned to one or more classes, each class corresponding to a category; determining a target sampling weight for each instance in each sub-class based on the number of target category labels in each instance in the sub-class and the number of all category labels in each instance, wherein the sub-class is a class whose number is less than a first threshold among the multiple classes, the target category label is the category label corresponding to the sub-class in each instance, and the target sampling weight is the sampling weight of the instance corresponding to the sub-class in each instance; and performing weighted sampling on the instances of each sub-class according to the target sampling weight of each instance in the sub-class.

[0005] This application establishes a weight model for subclasses, calculates the target sampling weight for each instance in each category, and adds new instances by weighted sampling of instances in each subclass. Only new instances can be added, and all original instances will be retained without losing any instances. At the same time, instances with a higher number of target category labels in a subclass relative to the total number of category labels are more likely to be upsampled, with less impact on other categories. This avoids scene loss and indirectly increases the number of other categories.

[0006] According to some embodiments of this application, the method further includes: constructing the subclass by taking the target instance determined based on the weighted sampling result as a new instance.

[0007] This application increases the number of instances in a subclass by constructing a subclass, thereby further avoiding the long-tail effect.

[0008] According to some embodiments of this application, the method further includes: constructing instances in the subclass up to a first number; determining whether the number of instances in each subclass reaches a second number; if the number of instances in each subclass does not reach the second number, continuing to classify the instances in the sample set according to the category label to form multiple classes, continuing to determine the target sampling weight of each instance in each subclass, continuing to perform weighted sampling on the instances in each subclass, and continuing to construct the subclass with the target instance determined according to the weighted sampling result as a new instance until the number of instances in each subclass reaches the second number.

[0009] This approach can increase the amount of data in a specific subclass. Furthermore, based on an iterative sampling strategy, the number of target instances, the first quantity, and the second quantity will all be adjusted in each round of sampling due to the addition of new instances, so the total number of samples will not increase dramatically. At the same time, it further avoids indirectly increasing the number of other categories.

[0010] According to some embodiments of this application, the method further includes: if the number of instances in each subclass reaches a second number, outputting instances of all classes.

[0011] This invention outputs instances of all classes, enabling the training device to train an initial AI model based on these instances to obtain a target model.

[0012] According to some embodiments of this application, before constructing instances in the subclass to a first quantity, the method further includes: determining the first quantity.

[0013] By determining a first quantity, this case allows for adjustment of that first quantity during each round of sampling.

[0014] According to some embodiments of this application, determining the first quantity includes: determining the first quantity as the average number of current instances of all classes.

[0015] This case avoids the N by using the first quantity as the average of the current number of all classes. aim Too large an N value can easily lead to an excessively large total number of instances in the sample set, thus avoiding the aforementioned N. aim Too small a value can lead to too many iterations and slow convergence.

[0016] According to some embodiments of this application, before determining whether the number of instances in each subclass reaches a second number, the method further includes: determining the second number.

[0017] By determining a second quantity, this case allows for adjustment of the second quantity during each round of sampling.

[0018] According to some embodiments of this application, determining the second quantity includes: determining the second quantity as r*N max , where r∈[0.5,0.8], N max The number of instances of the largest class.

[0019] This case uses the second quantity as r*N max , where r∈[0.5,0.8], N max The number of instances in the largest class is used to avoid the second number being too large, which can easily lead to a large number of duplicate samples in the new sample set, which can easily lead to overfitting during the training process, and to avoid the second number being too small, which can easily lead to an insignificant balancing effect among different classes.

[0020] According to some embodiments of this application, the weighted sampling includes at least one of Weighted Random Sampling (WRS) and Naive Weighted Sampling.

[0021] This case uses weighted sampling, including at least one of the weighted random sampling method (WRS) and the naive weighted sampling method, so that the newly added instances mainly include instances with more target category labels, which will reduce the imbalance of each category.

[0022] According to some embodiments of this application, the step of weighted sampling of instances in each subclass based on the target sampling weight of each instance in the subclass includes: generating a random number within a preset range for each instance in the subclass; determining the sampling score of each instance in the subclass using the weighted random sampling method (WRS) based on the random number and the target sampling weight; and selecting the target instance with the largest sampling score as a new instance.

[0023] This case uses the weighted random sampling method (WRS) to make instances with larger target sampling weights have larger sampling scores. By using the target instance with the largest sampling score as a new instance, instances with larger target sampling weights are more likely to be sampled.

[0024] According to some embodiments of this application, the step of weighted sampling of instances in each subclass based on the target sampling weight of each instance in the subclass may include: sorting all instances in the subclass in ascending order of target sampling weight; generating random numbers within a preset range; and determining new instances using a naive weighted sampling method based on the random numbers and the sorting of the target sampling weights.

[0025] This case uses a naive weighted sampling method to make the cumulative value of instances with larger target sampling weights larger. By using target instances whose cumulative values ​​reach a random number as new instances, instances with larger target sampling weights are more easily sampled.

[0026] According to some embodiments of this application, before determining the target sampling weight of each instance in each subclass based on the number of target category labels in each instance in the subclass and the number of all category labels in each instance, the method further includes: determining subclasses among a plurality of said classes.

[0027] This case involves identifying subcategories, which then allow for sampling within those subcategories.

[0028] According to some embodiments of this application, determining the subclass among the plurality of classes includes: determining the sampling rate of each class based on the current number of instances of each class and a preset target number of instances; and determining the class with a sampling rate greater than zero as the subclass among the plurality of classes.

[0029] According to some embodiments of this application, determining the sampling rate of each class based on the current number of instances of each class and the preset target number of instances includes: according to the formula Determine the sampling rate, where f t Let N be the sampling rate for class t. aim N represents the preset target number of instances. i The number of current instances of class t.

[0030] This case uses a formula The sampling rate can be determined, and thus the subclass can be determined.

[0031] According to some embodiments of this application, determining the target sampling weight of each instance in each subclass based on the number of target category labels in each instance of the subclass and the number of all category labels in each instance includes: determining the proportion of the target category label of each instance in each subclass based on the number of target category labels in each instance of the subclass and the number of all category labels in each instance; and determining the target sampling weight of each instance in each subclass based on the proportion of the target category label of each instance in each subclass.

[0032] This case uses a weighted model to ensure that the more target category labels an instance in a subclass has relative to the total number of category labels in that instance, the higher the target sampling weight of that instance in that class; conversely, the fewer target category labels an instance in a subclass has relative to the total number of category labels in that instance, the lower the target sampling weight of that instance in that class.

[0033] According to some embodiments of this application, determining the proportion of target category labels for each instance in each subclass based on the number of target category labels in each instance of the subclass and the total number of category labels in each instance includes: according to the formula Determine the percentage of the target category label for each instance in each subclass, where p ti k represents the proportion of the target category label of instance i in subclass t. ti m is the number of target category labels for instance i in subclass t. ti Let be the number of all category labels in instance i within subclass t.

[0034] This case uses a formula The higher the number of target category labels for instances in each subclass relative to the total number of category labels in those instances, the higher the percentage of target category labels for those instances in that class; conversely, the lower the number of target category labels for instances in each subclass relative to the total number of category labels in those instances, the lower the percentage of target category labels for those instances in that class.

[0035] According to some embodiments of this application, determining the target sampling weight of each instance in each subclass based on the proportion of the target category label of each instance in each subclass includes: according to the formula Determine the target sampling weight for each instance in each subclass, where w ti p represents the target sampling weights for instance i in subclass t. ti Let ti be the percentage of the target category label of instance i in subclass t, and tn be the total number of all instances in subclass t. This represents the sum of the percentages of the target category labels for all instances in subclass t.

[0036] This case uses a formula The higher the proportion of the target category label of the instances in each subclass, the higher the target sampling weight of the instances in that class; conversely, the lower the proportion of the target category label of the instances in each subclass, the lower the target sampling weight of the instances in that class.

[0037] An embodiment of the present invention also provides a multi-label class equalization device, the device comprising: an acquisition unit, the acquisition unit being configured to acquire a sample set including multiple instances, each instance including one or more category labels; a sampling unit, the sampling unit being configured to classify the instances in the sample set into multiple classes according to the category labels, each instance being assigned to one or more classes, each class corresponding to a category; the sampling unit being configured to determine a target sampling weight for each instance in each sub-class based on the number of target category labels in each instance in the sub-class and the number of all category labels in each instance, wherein the sub-class is a class whose number is less than a first threshold among the multiple classes, the target category label is the category label corresponding to the sub-class in each instance, and the target sampling weight is the sampling weight corresponding to the sub-class in each instance; the sampling unit is further configured to perform weighted sampling on the instances of each sub-class according to the target sampling weight of each instance in the sub-class.

[0038] This application establishes a weight model for subclasses, calculates the target sampling weight for each instance in each category, and adds new instances by weighted sampling of instances in each subclass. Only new instances can be added, and all original instances will be retained without losing any instances. At the same time, instances with a higher number of target category labels in a subclass relative to the total number of category labels are more likely to be upsampled, with less impact on other categories. This avoids scene loss and indirectly increases the number of other categories.

[0039] According to some embodiments of this application, the sampling unit is further configured to construct the subclass by using the target instance determined based on the weighted sampling result as a new sample.

[0040] This application increases the number of instances in a subclass by constructing a subclass, thereby further avoiding the long-tail effect.

[0041] According to some embodiments of this application, the sampling unit is further configured to construct instances in the subclass up to a first number; the sampling unit is further configured to determine whether the number of instances in each subclass reaches a second number; the sampling unit is further configured to, if the number of instances in each subclass does not reach the second number, continue to classify the instances in the sample set according to the category label to form multiple classes, continue to determine the target sampling weight of each instance in each subclass, and continue to perform weighted sampling on the instances in each subclass until the number of instances in each subclass reaches the second number.

[0042] This approach can increase the amount of data in a specific subclass. Furthermore, based on an iterative sampling strategy, the number of target instances, the first quantity, and the second quantity will all be adjusted in each round of sampling due to the addition of new instances, so the total number of samples will not increase dramatically. At the same time, it further avoids indirectly increasing the number of other categories.

[0043] According to some embodiments of this application, the sampling unit is further configured to output instances of all classes if the number of instances in each subclass reaches a second number.

[0044] This invention outputs instances of all classes, enabling the training device to train an initial AI model based on these instances to obtain a target model.

[0045] According to some embodiments of this application, before constructing instances in the subclass to a first quantity: the sampling unit is further configured to determine the first quantity.

[0046] By determining a first quantity, this method allows for adjustment of that first quantity during each sampling.

[0047] According to some embodiments of this application, the sampling unit is further configured to determine the first quantity as the average number of current instances of all classes.

[0048] This case avoids the N by using the first quantity as the average of the current number of all classes. aim Too large an N value can easily lead to an excessively large total number of instances in the sample set, thus avoiding the aforementioned N. aim Too small a value can lead to too many iterations and slow convergence.

[0049] According to some embodiments of this application, before determining whether the number of instances in each subclass reaches the second number: the sampling unit is also used to determine the second number.

[0050] By determining a second quantity, this case allows for adjustment of the second quantity at each sampling time.

[0051] According to some embodiments of this application, the sampling unit is further configured to determine that the second quantity is r*N. max , where r∈[0.5,0.8], N max The number of instances of the largest class.

[0052] In this case, the second quantity is r*N. max , where r∈[0.5,0.8], N max The maximum number of instances in the largest class is used to avoid the problem that if the second number is too large, it will easily lead to a large number of duplicate samples in the new sample set, which may cause overfitting. Conversely, if the second number is too small, the balancing effect of each class may not be obvious.

[0053] According to some embodiments of this application, the weighted sampling includes at least one of Weighted Random Sampling (WRS) and Naive Weighted Sampling.

[0054] This case uses weighted sampling, including at least one of the weighted random sampling method (WRS) and the naive weighted sampling method, so that the newly added instances mainly include instances with more target category labels, which will reduce the imbalance of each category.

[0055] According to some embodiments of this application, the sampling unit is further configured to generate a random number within a preset range for each instance in the subclass; determine the sampling score of each instance in the subclass using the weighted random sampling method (WRS) based on the random number and the target sampling weight; and select the target instance with the largest sampling score as a new instance.

[0056] This case uses the weighted random sampling method (WRS) to make instances with larger target sampling weights have larger sampling scores. By using the target instance with the largest sampling score as a new instance, instances with larger target sampling weights are more likely to be sampled.

[0057] According to some embodiments of this application, the sampling unit is further configured to sort all instances in the subclass in ascending order of target sampling weight; generate random numbers within a preset range; and determine new instances using a naive weighted sampling method based on the random numbers and the sorting of the target sampling weights.

[0058] This case uses a naive weighted sampling method to make the cumulative value of instances with larger target sampling weights larger. By using target instances whose cumulative values ​​reach a random number as new instances, instances with larger target sampling weights are more easily sampled.

[0059] According to some embodiments of this application, before determining the target sampling weight of each instance in each subclass based on the number of target category labels in each instance in the subclass and the number of all category labels in each instance: the sampling unit is further configured to determine the subclasses in a plurality of the classes.

[0060] This case involves identifying subcategories, which then allow for sampling within those subcategories.

[0061] According to some embodiments of this application, the sampling unit is further configured to determine the sampling rate of each class based on the number of current instances of each class and the preset target number of instances; and to determine that the classes with sampling rates greater than zero are minor classes among the multiple classes.

[0062] According to some embodiments of this application, the sampling unit is further configured to, according to the formula Determine the sampling rate, where f t Let N be the sampling rate for class t. ai N represents the preset target number of instances. i The number of current instances of class t.

[0063] This case uses a formula The sampling rate can be determined, and thus the subclass can be determined.

[0064] According to some embodiments of this application, the sampling unit is used to determine the proportion of target category labels for each instance in each subclass based on the number of target category labels in each instance in the subclass and the total number of category labels in each instance; the sampling unit is also used to determine the target sampling weight for each instance in each subclass based on the proportion of target category labels for each instance in each subclass.

[0065] This case uses a weighted model to ensure that the more target category labels an instance in a subclass has relative to the total number of category labels in that instance, the higher the target sampling weight of that instance in that class; conversely, the fewer target category labels an instance in a subclass has relative to the total number of category labels in that instance, the lower the target sampling weight of that instance in that class.

[0066] According to some embodiments of this application, the sampling unit is further configured to, according to the formula Determine the percentage of the target category label for each instance in each subclass, where p ti k represents the proportion of the target category label of instance i in subclass t. ti m is the number of target category labels for instance i in subclass t. ti Let be the number of all category labels in instance i within subclass t.

[0067] This case uses a formula The higher the number of target category labels for instances in each subclass relative to the total number of category labels in those instances, the higher the percentage of target category labels for those instances in that class; conversely, the lower the number of target category labels for instances in each subclass relative to the total number of category labels in those instances, the lower the percentage of target category labels for those instances in that class.

[0068] According to some embodiments of this application, the sampling unit is further configured to, according to the formula Determine the target sampling weight for each instance in each subclass, where w ti p represents the target sampling weights for instance i in subclass t. ti Let ti be the percentage of the target category label of instance i in subclass t, and tn be the total number of all instances in subclass t. This represents the sum of the percentages of the target category labels for all instances in subclass t.

[0069] This case uses a formula The higher the proportion of the target category label of the instances in each subclass, the higher the target sampling weight of the instances in that class; conversely, the lower the proportion of the target category label of the instances in each subclass, the lower the target sampling weight of the instances in that class.

[0070] An embodiment of the present invention also provides an electronic device, the electronic device including a processor and a memory, the memory being used to store program instructions, and the processor, when calling the program instructions, implementing the multi-label class equalization method as described in any of the preceding claims.

[0071] An embodiment of the present invention also provides a vehicle that includes the electronic equipment described above.

[0072] An embodiment of the present invention also provides a computer-readable storage medium storing a program that causes a computer device to implement the multi-label class equalization method as described in any of the preceding claims.

[0073] An embodiment of the present invention also provides a computer program product, the computer program product including computer executable instructions stored in a computer-readable storage medium; at least one processor of the device can read the computer executable instructions from the computer-readable storage medium, and the at least one processor executes the computer executable instructions to cause the device to perform the multi-label class equalization method as described in any of the preceding claims. Attached Figure Description

[0074] Figure 1 This is a schematic diagram of the system architecture according to an embodiment of the present invention.

[0075] Figure 2 This is a flowchart illustrating the multi-label class balancing method of the first embodiment of the present invention.

[0076] Figure 3 This is a schematic diagram illustrating an example of an embodiment of the present invention.

[0077] Figure 4 This is a schematic diagram showing the number of instances in a corresponding category after the instance classification in the sample set of this embodiment of the invention.

[0078] Figure 5 This is a schematic diagram of the category label for category t in an embodiment of the present invention.

[0079] Figure 6 This is a flowchart illustrating the multi-label class balancing method according to the second embodiment of the present invention.

[0080] Figure 7 This is a schematic diagram illustrating how, according to an embodiment of the present invention, instances of the subclass are constructed such that the number of instances of each subclass reaches a first number.

[0081] Figure 8 This is a schematic diagram illustrating the iterative construction of subclasses according to an embodiment of the present invention.

[0082] Figure 9 This is a schematic diagram of the structure of a multi-label class equalizer provided in an embodiment of the present invention.

[0083] Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention.

[0084] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0085] Explanation of main component symbols

[0086] System Architecture 10

[0087] Data acquisition device 160

[0088] Database 130

[0089] Training equipment 120

[0090] Target Model 101

[0091] Execution device 110

[0092] I / O Interface 112

[0093] Customer equipment 140

[0094] Data storage system 150

[0095] Calculation Module 111

[0096] Multi-label class equalizer 900

[0097] Acquisition Unit 901

[0098] Sampling unit 902

[0099] Training Unit 903

[0100] 1000 electronic devices

[0101] Electronic device 1100

[0102] Memory 1001, 1101

[0103] Processor 1002, 1102

[0104] Communication interfaces 1003, 1103

[0105] Bus 1004, 1104 Detailed Implementation

[0106] In the description of the embodiments in this application, the word "for example" is used to indicate that it is an example, illustration, or illustration. Any embodiment or design scheme described as "for example" in the embodiments of this application should not be construed as being better or more advantageous than other embodiments or design schemes. Specifically, the use of the word "for example" is intended to present the relevant concepts in a specific manner.

[0107] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. It should be understood that, unless otherwise stated, "a plurality of" in this application means two or more.

[0108] Existing data balancing methods based on mixed sampling form a new dataset through instance classification, averaging, random upsampling, random downsampling, and combination. Instance classification involves counting the number of instances for each category. Averaging involves summing the instance counts for each category and then calculating the average. Random upsampling involves upsampling categories with fewer instances than the average using random sampling with replacement to bring their instance count to the average. Random downsampling involves downsampling categories with more instances than the average to bring their instance count to the average. Combining involves recombining instances from each category to form a new dataset. However, random downsampling may discard some scenes, resulting in scene loss. Random upsampling, because it's done with replacement, cannot guarantee that every instance will be selected, potentially leading to scene loss. Furthermore, since an instance may have multiple labels, upsampling based on one label may simultaneously increase the number of other labels, leading to class imbalance.

[0109] To better understand the methods of the embodiments of this application, some terms and concepts involved in this application will be introduced below.

[0110] Example: Generally refers to image samples in 2D datasets and point cloud samples in 3D datasets.

[0111] Sample set: A collection of instances, where each element in the set is an instance.

[0112] Category: refers to the distinction between different things, and is a distinction made according to different kinds.

[0113] Class: A subset defined on the sample set consisting of all instances, where samples of the same class are indistinguishable in terms of a certain property of interest, i.e., they have the same pattern.

[0114] Minor class: refers to a class whose number is less than the first threshold among multiple classes.

[0115] Before providing a detailed explanation of the embodiments of this application, the application scenarios involved in the embodiments of this application will be introduced first.

[0116] Currently, instances in the training dataset can be used to create a model. This model can be an AI model. An AI model is a machine learning model, essentially a mathematical model that includes a large number of parameters and mathematical formulas (or rules). Its purpose is to learn mathematical expressions that provide the correlation between input values ​​x and output values ​​y. The mathematical expression that provides the correlation between x and y is the trained AI model. Before using an AI model for detection, the initial AI model needs to be trained.

[0117] The main factors affecting AI model training include three aspects: training dataset, initial AI model, and machine computing power. With the increasing application scenarios of AI models, the scenarios they face are becoming more complex, leading to increasingly complex AI models for training. Simultaneously, to improve training effectiveness, the required amount of training datasets is also increasing, as are the types of data. This results in increased computational load during training, continuously increasing demands on machine computing power, and longer training times. Optimizing the AI ​​model training process to obtain a more accurate AI model in the shortest possible time is a key focus of the industry.

[0118] Therefore, to improve the performance of AI model training, after developers have written the initial neural network model to be trained, they can create instances using the multi-label class equalization method provided in this application's embodiments. They can then use the training dataset including the instances constructed in this application to train the AI ​​model to be trained, thereby improving the accuracy of model inference. Here, the neural network model to be trained is the initial AI model that needs to be trained.

[0119] The trained AI model possesses predictive analytics capabilities and can be used for various purposes. For example, it can identify data of interest (such as people or objects in images / videos) from target data (e.g., video streams, image streams, or images) to determine the location and size of objects within the image or video. The object of interest is called the target, and the type of target can be determined based on the application scenario and business focus. For instance, in the field of surveillance and security, targets can be, but are not limited to, vehicles, people, and faces. In the field of transportation, targets can be, but are not limited to, roads, license plate numbers, road facilities, traffic signs, and pedestrians. In the field of logistics, targets can be, but are not limited to, container numbers, tracking numbers, obstacles, and package numbers. In the field of military and defense, targets can be, but are not limited to, terrain and aircraft. In the field of medicine, targets can be, but are not limited to, organs and tissues.

[0120] Clearly, the trained AI model can also be applied to other scenarios besides object detection, such as recognition and segmentation. The following example illustrates the applicability of the trained AI model to object detection; however, this is not intended to limit it to object detection only.

[0121] refer to Figure 1 , Figure 1 This is a schematic diagram of the system architecture 10 according to an embodiment of the present invention. The data acquisition device 160 is used to collect training data. For the object detection method (an example of an AI model application) in this application embodiment, the training data may include training instances. The results of the training instances may be manually pre-annotated or automatically annotated using an existing annotation system; that is, each training instance includes one or more category labels. After collecting the training data, the data acquisition device 160 stores this training data in the database 130. The training device 120 trains the target model 101 based on the training data maintained in the database 130. The following will describe in more detail how the training device 120 trains the target model 101 based on the training data. The target model 101 can find data of interest (e.g., people or objects in images / videos) in target data (e.g., video streams, image streams, or images) to determine the position and size of objects in images or videos.

[0122] In this embodiment, the training device 120 can acquire a sample set including multiple instances, each instance including one or more category labels; classify the instances in the sample set into multiple classes according to the category labels, each instance being assigned to one or more classes, and each class corresponding to a category; determine the target sampling weight of each instance in each sub-class based on the number of target category labels in each instance in the sub-class and the total number of category labels in each instance, wherein the sub-class is a class whose number is less than a first threshold among the multiple classes, the target category label is the category label corresponding to the sub-class in each instance, and the target sampling weight is the sampling weight of the instance corresponding to the sub-class in each instance; perform weighted sampling on the instances in each sub-class according to the target sampling weight of each instance in the sub-class. The training device 120 can also train an initial AI model based on the instances to obtain a target model 101.

[0123] It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, the training device 120 may not necessarily train the target model 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training, such as from the 2D dataset BDD or the 3D dataset nuScenes. The above description should not be construed as limiting the embodiments of this application.

[0124] The target model 101 trained using training device 120 can be applied to different systems or devices, such as... Figure 1 The execution device 110 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, AR / VR, vehicle terminal, etc., or it can be a server or cloud service. Figure 1 In this embodiment, the execution device 110 is equipped with an I / O interface 112 for data interaction with external devices. The user can input data to the I / O interface 112 through the client device 140. The input data may include the image to be processed input by the client device.

[0125] The data storage system 150 is used to receive and store the parameters of the target model sent by the training device 120, and to store the data of the target detection results obtained by the target model 101. It may also include the program code (or instructions) required for the normal operation of the data storage system 150. The data storage system 150 can be a distributed storage cluster consisting of one or more devices deployed outside the execution device 110. In this case, when the execution device 110 needs to use data on the data storage system 150, the data storage system 150 can send the required data to the execution device 110, and the execution device 110 receives and stores (or caches) the data. Alternatively, the data storage system 150 can be deployed within the execution device 110. When deployed within the execution device 110, the distributed storage system can include one or more storage devices. Optionally, when multiple storage devices exist, different storage devices are used to store different types of data. For example, the model parameters of the target model generated by the training device and the data of the target detection results obtained by the target model 101 can be stored on two different storage devices.

[0126] The calculation module 111 uses the target model 101 to process the input image to be processed (e.g., video stream, image stream, or image), for example, to find data of interest (e.g., people or objects in the image / video) from the input image to be processed, in order to determine the position and size of the object in the image or video.

[0127] Finally, I / O interface 112 returns the processing results, such as the position and size of the object in the image or video obtained above, to client device 140, thereby providing them to the user.

[0128] It is worth noting that the training device 120 can generate corresponding target models 101 based on different training data for different objectives or tasks. The corresponding target models 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0129] exist Figure 1In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0130] It is worth noting that, attached Figure 1 This is merely a schematic diagram of a system architecture provided by an embodiment of the present invention. The positional relationships between the devices, components, modules, etc. shown in the diagram do not constitute any limitation. For example, in the attached diagram... Figure 1 In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0131] In this embodiment of the application, the training device 120, the execution device 110, and the client device 140 may be three different physical devices, or the training device 120 and the execution device 110 may be on the same physical device or a cluster, or the execution device 110 and the client device 140 may be on the same physical device or a cluster.

[0132] Please refer to Figure 2 This is a flowchart illustrating the multi-label class balancing method according to the first embodiment of the present invention. The method can specifically be as follows: Figure 1 The training device 120 shown executes the method, and the sample set of multiple instances in the method can be as follows: Figure 1 The training data maintained in the database 130 shown can optionally be processed in the training device 120, or the steps S202 to S206 of the method can be pre-processed by other functional modules before the training device 120. That is, the training data received or obtained from the database 130 is pre-processed, such as the sampling process described in steps S202 to S206, to obtain a new sample set, which is used as the input of the training device 120, and the training device performs model training.

[0133] Optionally, the method can be processed by a CPU, or by a combination of a CPU and a GPU, or it can be processed without a GPU, using other processors suitable for processing. This application does not impose any restrictions.

[0134] The method includes, but is not limited to, the following steps:

[0135] S201: Obtain a sample set that includes multiple instances, each instance including one or more category labels.

[0136] The sample set can be obtained through... Figure 1 The data is collected by the data acquisition device 160 shown. Each instance in the sample set collected by the data acquisition device 160 is either manually pre-annotated or automatically annotated using an existing annotation system. The sample set may also be obtained from 2D datasets such as BDD or 3D datasets such as nuScenes, or from datasets from other sources such as storage, and automatically annotated using an existing annotation system. The sample set may include all datasets in the 2D dataset BDD or the 3D dataset nuScenes, or it may include only a portion of the datasets. This case illustrates the example of including only a portion of the datasets.

[0137] This case uses the sample set as an example of an autonomous driving dataset for brief illustration. In this embodiment, the sample set contains 10 categories: bus, traffic light, sign, person, bicycle, car, truck, motorcycle, train, and rider. However, it is understood that for an autonomous driving dataset, the categories of the sample set are not limited to the above categories and can also be other categories. The sample set is not limited to autonomous driving datasets; it can also be a dataset of dogs from the Stanford Dogs Dataset, or a dataset of human face images from Labelled Faces in the Wild, etc. Obviously, the categories of the sample set will differ for different sample sets.

[0138] The instance can be an image sample from a 2D dataset. The instance can also be a point cloud sample from a 3D dataset. The instance can be, for example... Figure 3 As shown. In Figure 3In this example, the instance includes six category labels: traffic sign, traffic sign, rider, bike, truck, and car. In this embodiment, each object in the instance is indicated by a two-dimensional border. Each category label represents an object. These category labels are displayed on the instance, and their indication helps determine the type of each object. For example, the category label "car" indicates that the selected object is a car, the category label "traffic sign" indicates that the selected object is a traffic sign, the category label "rider" indicates that the selected object is a rider, the category label "bike" indicates that the selected object is a bike, and the category label "truck" indicates that the selected object is a truck. The same category label belongs to the same class. The above six category labels belong to five categories: car, traffic sign, rider, bike, and truck.

[0139] S202: Classify the instances in the sample set according to the category labels to form multiple classes. Each instance is assigned to one or more classes, and each class corresponds to a category.

[0140] In this embodiment, if an instance includes one category label, the instance's index is added to that category. If an instance includes multiple category labels, the instance's index is added to the corresponding category for each category. If an instance includes multiple instances of the same category label, the instance's index is added to that category only once. Figure 3 Taking the example above, which includes two traffic sign category labels, one car category label, one rider category label, one bike category label, and one truck category label, the index numbers of the above instances are added under the categories of traffic sign, car, rider, bike, and truck, respectively. The index numbers for adding instances under each category are shown in Table 1 below:

[0141] Table 1

[0142]

[0143]

[0144] After the instances in the sample set are classified, multiple classes are formed. The number of instances in different classes may be the same or different, such as... Figure 4 As shown. In Figure 4 Among them, the car class has the most instances, with 27,558, while the bicycle class has the fewest instances, with 6,263. Figure 4 In the data, the head category accounts for the majority of instances, while the tail category accounts for a very small number of instances. For example, the car and human categories account for the majority of instances, while the train, rider, motorcycle, and bicycle categories account for a very small number of instances.

[0145] S203: Determine the subclass among the multiple classes.

[0146] The subclass is the class whose number is less than a first threshold among the multiple classes. Specifically, determining the subclass among the multiple classes includes: determining the sampling rate of each class based on the current number of instances of each class and a preset target number of instances; determining the classes with a sampling rate greater than zero as the subclass among the multiple classes. Determining the sampling rate of each class based on the current number of instances of each class and the preset target number of instances includes: according to the formula Determine the sampling rate, where f t Let N be the sampling rate for class t. aim N represents the preset target number of instances. i N represents the number of current instances of class t. aim This can be the average number of instances in each category, or a value slightly greater than the average number of instances in each category, or a value slightly less than the average number of instances in each category. Wherein, if the N... aim If N is too large, it can easily lead to an excessively large total number of instances in the sample set. aim If the sampling rate is too small, it can easily lead to too many iterations and slow convergence. The class with a sampling rate greater than zero is the largest class among the multiple classes mentioned.

[0147] For example, with Figure 4 For example, the current number of instances of the bicycle class is 6263. If the preset target number of instances is 14000, then according to the formula... If the sampling rate is determined to be approximately 1.24, which is greater than zero, then the bicycle class can be identified as a subclass among the multiple classes mentioned.

[0148] S204: Determine the target sampling weight for each instance in each subclass based on the number of target category labels in each instance and the number of all category labels in each instance, wherein the target category label is the category label corresponding to the subclass in each instance, and the target sampling weight is the sampling weight corresponding to the subclass in each instance.

[0149] In this embodiment, each instance may include one or more category labels and can be classified into one or more classes. The proportion of different categories included in different instances may be the same or different. As described above... Figure 3Taking the examples above as an example, the categories traffic lights, cars, riders, bicycles, and trucks have proportions of 1 / 3, 1 / 6, 1 / 6, 1 / 6, and 1 / 6, respectively. Therefore, the proportions of the target category label for different instances within a category can be the same or different. Consequently, the new sample sets formed by upsampling from different instances within the same category will also be different. Therefore, before upsampling, it is necessary to determine the ratio of the proportion of the target category label of each instance in each sub-category to the sum of the proportions of the category labels of all instances in that sub-category; that is, to determine the target sampling weight for each instance in each sub-category.

[0150] In this embodiment, determining the target sampling weight of each instance in each subclass based on the number of target category labels in each instance of the subclass and the number of all category labels in each instance includes: determining the proportion of the target category label of each instance in each subclass based on the number of target category labels in each instance of the subclass and the number of all category labels in each embodiment; and determining the target sampling weight of each instance in each subclass based on the proportion of the target category label of each instance in each subclass.

[0151] In this embodiment, determining the proportion of target category labels for each instance in each subclass based on the number of target category labels in each instance of the subclass and the total number of category labels in each instance includes: according to the formula Determine the percentage of the target category label for each instance in each subclass, where p ti k represents the proportion of the target category label of instance i in subclass t. ti m is the number of target category labels for instance i in subclass t. ti Let be the number of all category labels in instance i within subclass t.

[0152] Let's take subclass m as an example for illustration, such as... Figure 5 As shown, subclass m is bicycle, and subclass m includes n instances, namely instance_1, ..., instance_i, ..., instance_n. Different instances may have the same or different category labels. For example, in... Figure 5 In the example, instance_1 includes the category labels bike, truck, bus; instance_i includes the category labels bike, bike, rider, motor; and instance_n includes the category labels bike, truck, car. Then, the percentage of the bike category label of instance_i in the bicycle class relative to all category labels of instance_i, i.e., the percentage of the bike category label of instance_i in the bicycle class, is: Therefore, if the number of target category labels of instances in the subclass is greater than the total number of category labels in the instances, the proportion of target category labels of the instances in the class is higher; if the number of target category labels of instances in the subclass is less than the total number of category labels in the instances, the proportion of target category labels of the instances in the class is lower.

[0153] In this embodiment, determining the target sampling weight of each instance in each subclass based on the proportion of the target category label of each instance in each subclass includes: according to the formula Determine the target sampling weight for each instance in each subclass, where w ti p represents the target sampling weights for instance i in subclass t. ti Let ti be the percentage of the target category label of instance i in subclass t, and tn be the total number of all instances in subclass t. This represents the sum of the percentages of the target category labels for all instances in subclass t.

[0154] Taking subcategory t as an example, subcategory t includes instance i and instance j. The proportion of the bike category label for instance i is... The percentage of the bike category label for instance j is The sum of the percentages of the bike category label for all instances of subclass t is: Therefore, the target sampling weight of instance i in subclass t is: Therefore, it can be seen that if the proportion of target category labels in instances of each subclass is higher, the target sampling weight of the instances of that class is higher; conversely, if the proportion of target category labels in instances of each subclass is lower, the target sampling weight of the instances of that class is lower. That is, if the number of target category labels in instances of each subclass is greater relative to the total number of category labels in those instances, the target sampling weight of the instances of that class is likely to be higher; conversely, if the number of target category labels in instances of each subclass is less relative to the total number of category labels in those instances, the target sampling weight of the instances of that class is likely to be lower.

[0155] S205: Perform weighted sampling on instances of each subclass according to the target sampling weight of each instance in the subclass.

[0156] In this embodiment, when upsampling the subclass, if the upsampling is mainly performed on instances in the subclass where the number of target category labels is greater than the number of all category labels in the instances, the newly added instances will mainly include instances with more target category labels, which will reduce the imbalance of each category.

[0157] In this embodiment, the weighted sampling includes at least one of Weighted Random Sampling (WRS) and Naive Weighted Sampling.

[0158] In this embodiment, the step of weighted sampling of instances of each subclass according to the target sampling weight of each instance in the subclass may include:

[0159] For each instance in the subclass, generate a random number within a preset range;

[0160] The sampling score of each instance in the subclass is determined by weighted random sampling (WRS) based on the random number and the target sampling weight.

[0161] The target instance with the highest sampling score is selected as the new instance.

[0162] In this embodiment, the preset interval is the (0, 1) interval. The preset interval can also be other intervals, such as (0, 2). This invention can utilize a random number generator to generate multiple random numbers within the (0, 1) interval. The step of determining the sampling score of each instance in the subclass using the weighted random sampling method (WRS) based on the random number and the target sampling weight includes: according to the formula... The sampling rate is determined by ti∈tn, where S ti R represents the sampled score of instance i in class t. ti w is a random number for instance i in class t. ti Let be the target sampling weight for instance i in subclass t, where ti is the i-th instance in subclass t, and tn is the n-th instance in subclass t. For example, continuing with the above example of class t, the target sampling weight for instance i in subclass t is: The target sampling weight of instance j in subclass t is: If the random number of instance i in subclass t is 0.8 and the random number of instance j in subclass t is 0.6, the sampling scores of instance i and instance j in subclass t can be determined to be 0.69 and 0.28 respectively. Then, instance i with a sampling score of 0.69 will be added as a new instance.

[0163] In this embodiment, the step of weighted sampling of instances of each subclass according to the target sampling weight of each instance in the subclass may include:

[0164] All instances in the aforementioned subclass are sorted in ascending order of target sampling weight;

[0165] Generate random numbers within a preset range;

[0166] New instances are determined by a naive weighted sampling method based on the sorting of the random numbers and the target sampling weights.

[0167] In this embodiment, the preset interval is the (0, 1) interval. The preset interval can also be other intervals, such as (0, 2). This invention can utilize a random number generator to generate multiple random numbers within the (0, 1) interval. The step of determining the new instance using a naive weighted sampling method based on the random number and the sorting of the target sampling weights includes: accumulating the target sampling weights of each instance sequentially from the first instance according to the sorting of the target sampling weights until the accumulated target sampling weight reaches the random number; and then using the target instance as the new instance. For example, continuing with the above example of class t, the target sampling weight of instance i in subclass t is... The target sampling weight of instance j in subclass t is: The target sampling weights are then sorted from smallest to largest as follows: If the random number is 0.5, the target sampling weight of the first instance is 0.4, which is less than 0.5. The target sampling weight of the second instance is accumulated, and the accumulated target sampling weight is 1, which reaches 0.5. Then the instance with a target sampling weight of 0.6 can be determined as the target instance, and the target instance is added as a new instance.

[0168] In this embodiment, the sampling score of each instance in the subclass is determined by the weighted random sampling method (WRS). Instances with higher target sampling weights will have higher sampling scores. By using the target instance with the highest sampling score as the new instance, instances with higher target sampling weights are more easily sampled, resulting in new instances that mainly include more target category labels, thus reducing the imbalance between categories.

[0169] By using a naive weighted sampling method, the cumulative values ​​of each instance are determined in ascending order of their target sampling weights. Instances with larger target sampling weights have larger cumulative values. By using target instances whose cumulative values ​​reach a random number as new instances, instances with larger target sampling weights are more easily sampled. This ensures that the new instances mainly include those with more target category labels, thus reducing class imbalance.

[0170] S206: Construct the subclass by using the target instance determined based on the weighted sampling results as a new instance.

[0171] In this embodiment, the step of adding new instances is repeated multiple times to construct the subclass. Specifically, the step of adding new instances is repeated multiple times, i.e., step S205, and the new instances are added to the original subclass to construct the subclass. For example, continuing to use subclass t as an example, subclass t includes instance i and instance j. If the new instances include instance k and instance l, then the constructed subclass includes: instance i, instance j, instance k, and instance l.

[0172] In this embodiment, the method can also train the initial AI model based on the instance to obtain the target model.

[0173] This case involves obtaining a sample set containing multiple instances, each instance including one or more category labels; classifying the instances in the sample set into multiple classes according to the category labels, with each instance assigned to one or more classes, and each class corresponding to a category; determining the target sampling weight for each instance in each sub-class based on the number of target category labels in each instance of the sub-class and the total number of category labels in each instance, wherein the sub-class is the class with a number less than a first threshold among the multiple classes, the target category label is the category label corresponding to the sub-class in each instance, and the target sampling weight is the sampling weight of the instance corresponding to the sub-class in each instance; and performing weighted sampling on the instances of each sub-class based on the target sampling weight of each instance in the sub-class, which can result in only new instances being added while all original instances are retained without losing any instances, and at the same time, it can make instances in the sub-class with a higher number of target category labels relative to the total number of category labels more likely to be upsampled, with less impact on other categories, thus avoiding scene loss and indirectly increasing the number of other categories.

[0174] Please refer to Figure 6 This is a flowchart illustrating the multi-label class balancing method of the second embodiment of the present invention.

[0175] The method can specifically be as follows: Figure 1 The training device 120 shown executes the method, and the sample set of multiple instances in the method can be as follows: Figure 1 The training data maintained in the database 130 shown can optionally be processed in the training device 120, or the steps S602 to S612 of the method can be pre-processed by other functional modules before the training device 120. That is, the training data received or obtained from the database 130 is pre-processed, such as the sampling process described in steps S602 to S612, to obtain a new sample set, which is used as the input of the training device 120, and the training device performs model training.

[0176] Optionally, the method can be processed by a CPU, or by a combination of a CPU and a GPU, or it can be processed without a GPU, using other processors suitable for processing. This application does not impose any restrictions.

[0177] The method includes, but is not limited to, the following steps:

[0178] S601: Obtain a sample set that includes multiple instances, each instance including one or more category labels.

[0179] Step S601 in the second embodiment is similar to step S201 in the first embodiment. Please refer to [link / reference needed] for details. Figure 2 The detailed description of step S201 in the first embodiment will not be repeated here.

[0180] S602: Classify the instances in the sample set according to the category label to form multiple classes. Each instance is assigned to one or more classes, and each class corresponds to a category.

[0181] Step S602 in the second embodiment is similar to step S202 in the first embodiment. Please refer to [link / reference needed] for details. Figure 2 The detailed description of step S202 in the first embodiment will not be repeated here.

[0182] S603: Determine the subclass among the multiple classes.

[0183] The process for determining the subclass among the multiple classes in the second embodiment is similar to the process for determining the subclass among the multiple classes in step S203 of the first embodiment. Please refer to [link / reference] for details. Figure 2 The detailed description of step S203 in the first embodiment will not be repeated here.

[0184] In this embodiment, an iterative sampling method is adopted. After each round of sampling, the number of instances of one or more classes will change. Therefore, in each round of sampling, it is necessary to preset the target number of instances and determine the subclasses among the multiple classes.

[0185] S604: Determine the target sampling weight for each instance in each subclass based on the number of target category labels in each instance and the total number of category labels in each instance, wherein the target category label is the category label corresponding to the subclass in each instance, and the target sampling weight is the sampling weight corresponding to the subclass in each instance.

[0186] Step S604 in the second embodiment is similar to step S204 in the first embodiment. Please refer to [link / reference] for details. Figure 2 The detailed description of step S204 in the first embodiment will not be repeated here.

[0187] S605: Perform weighted sampling on instances of each subclass according to the target sampling weight of each instance in the subclass.

[0188] Step S605 in the second embodiment is similar to step S205 in the first embodiment; please refer to [link / reference needed] for details. Figure 2 The detailed description of step S205 in the first embodiment will not be repeated here.

[0189] S606: Construct the subclass by using the target instance determined based on the weighted sampling results as a new instance.

[0190] Step S606 in the second embodiment is similar to step S206 in the first embodiment; please refer to [link / reference needed] for details. Figure 2 The detailed description of step S206 in the first embodiment will not be repeated here.

[0191] S607: Determine the first quantity.

[0192] Determining the first quantity includes: determining the first quantity as the average number of current instances of all classes. The first quantity may also be other quantities, such as a value slightly greater than the average number of instances in each class, or a value slightly less than the average number of instances in each class. The first quantity may be the same as or different from the preset target number of instances.

[0193] In this embodiment, an iterative sampling method is adopted. After each round of sampling, the number of instances of one or more classes will change. Therefore, the first number needs to be determined in each round of sampling.

[0194] S608: Construct instances of the subclass up to a first number.

[0195] In this embodiment, constructing instances of the subclass up to a first number includes:

[0196] Determine whether the number of instances in the subclass has reached the first quantity;

[0197] If the number of instances in the subclass does not reach the first number, continue to construct subclasses by weighting the instances of each subclass according to the target sampling weight of each instance in the subclass until the number of instances in the subclass reaches the first number, that is, continue to execute steps S605 to S606 until the number of instances in the subclass reaches the first number.

[0198] Specifically, before generating a random number within a preset range for each instance of the subclass, the method further includes: determining the number of instances to be constructed for the subclass based on the current number of instances of the subclass and the first number. The number of instances to be constructed can be the difference between the first number and the current number of instances of the subclass. For example, for subclass t, assuming its number of instances is n, the number of instances to be constructed for subclass t is delta = avg - n. Where delta is the number of instances to be constructed, avg is the first number, and n is the current number of instances of subclass t. In this embodiment, it is determined whether the number of instances in the subclass has reached the first number. If the number of instances in the subclass has not reached the first number, then the instances in the subclass are repeatedly weighted and sampled to obtain delta new instances. Thus, the number of instances in each subclass can reach the first number. Figure 4 For example, the diagram after weighted sampling of each subclass is as follows: Figure 7 As shown. In Figure 7 In this context, the number of instances for all subcategories, such as signs, buses, trains, riders, motorcycles, and bicycles, has increased to the highest number.

[0199] S609: Determine the second quantity.

[0200] Determining the second quantity includes: determining the second quantity as r*N max , where r∈[0.5,0.8], N max The value of r is the number of instances of the largest class. The value of r is not limited to the values ​​mentioned above; it can also be other values, such as 0.48, 0.81, etc. If the second number is too large, it can easily lead to a large number of duplicate samples in the new sample set, which can easily cause overfitting. If the second number is too small, the balancing effect between classes may not be significant.

[0201] In this embodiment, an iterative sampling method is adopted. After each round of sampling, the number of instances of one or more classes will change. Therefore, the second number needs to be determined in each round of sampling.

[0202] S610: Determine whether the number of instances in each subclass has reached the second number.

[0203] S611: If the number of instances in each subclass does not reach the second number, continue to classify the instances in the sample set according to the category label to form multiple classes, continue to determine the target sampling weight of each instance in each subclass, continue to perform weighted sampling on the instances in each subclass, and continue to construct the subclass by using the target instances determined according to the weighted sampling results as new instances until the number of instances in each subclass reaches the second number.

[0204] In this embodiment, if the number of instances in the subclass does not reach the second number, steps S602-S610 are repeated, thereby ensuring that the number of instances in each subclass reaches the second number. (Based on the above...) Figure 4 For example, the diagram illustrating the iterative construction of subclasses for each subclass is as follows: Figure 8 As shown. In Figure 8 In the middle, after 3 rounds of sampling, the number of instances of all subcategories, such as trucks, traffic lights, signs, buses, trains, riders, motorcycles, and bicycles, increased to the second largest number.

[0205] S612: If the number of instances in each subclass reaches the second number, output the instances in all classes.

[0206] In this embodiment, the method can also train the initial AI model based on the instance to obtain the target model.

[0207] This case involves obtaining a sample set containing multiple instances, each instance including one or more category labels; classifying the instances in the sample set into multiple classes according to the category labels, with each instance assigned to one or more classes, and each class corresponding to a category; determining the target sampling weight for each instance in each sub-class based on the number of target category labels in each instance of the sub-class and the total number of category labels in each instance, wherein the sub-class is the class with a number less than a first threshold among the multiple classes, the target category label is the category label corresponding to the sub-class in each instance, and the target sampling weight is the sampling weight of the instance corresponding to the sub-class in each instance; performing weighted sampling on the instances of each sub-class based on the target sampling weight of each instance in the sub-class; constructing the sub-class by using the target instances determined based on the weighted sampling results as new instances; constructing instances in the sub-class up to a first number; determining whether the number of instances in each sub-class reaches a second number; if the number of instances in each sub-class does not reach a second number... Upon reaching the second quantity, the instances in the sample set are further classified into multiple classes according to the category labels. The target sampling weight of each instance in each subclass is determined, and the instances in each subclass are subjected to weighted sampling. The target instances determined based on the weighted sampling results are used as new instances to construct the subclass until the number of instances in each subclass reaches the second quantity. Only new instances are added, and all original instances are retained without loss. This approach makes instances with a higher number of target category labels relative to the total number of category labels more likely to be upsampled, with less impact on other categories. Based on the iterative sampling strategy, due to the addition of new instances, the preset target instance count, the first quantity, and the second quantity are adjusted in each round of sampling. This ensures that the number of instances in each category reaches the second quantity before exiting the iteration, further avoiding the impact on other categories. This avoids scene loss and indirectly increases the number of other categories, thereby preserving the original data and solving the problem of class imbalance.

[0208] Please refer to Figure 9 , Figure 9 This is a schematic diagram of a multi-label class equalization device provided in an embodiment of the present invention. The multi-label class equalization device 900 may include an acquisition unit 901 and a sampling unit 902.

[0209] The multi-label class equalization device 900 can be used to execute the steps of the multi-label class equalization method of the embodiments of this application. For example, the acquisition unit 901 can be used to execute... Figure 2 In step S201 of the method shown, the sampling unit 902 can be used to perform... Figure 2 Steps S202 to S206 in the method shown. For example, the acquisition unit 901 can be used to perform... Figure 6In step S601 of the method shown, the sampling unit 902 can be used to perform... Figure 6 Steps S602 to S612 in the method shown.

[0210] The multi-label class equalization device 900 may further include a training unit 903. The training unit 903 is used to train an initial AI model based on the instance to obtain a target model.

[0211] Please see Figure 10 , Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Figure 10 The electronic device 1000 shown includes a memory 1001, a processor 1002, a communication interface 1003, and a bus 1004. The memory 1001, processor 1002, and communication interface 1003 are interconnected via the bus 1004.

[0212] The memory 1001 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1001 may store a program. When the program stored in the memory 1001 is executed by the processor 1002, the processor 1002 and the communication interface 1003 are used to execute the various steps of the multi-label class equalization method of this application embodiment.

[0213] The processor 1002 may be a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute related programs to implement the functions required by the units in the multi-label class equalization device of this application embodiment, or to execute the multi-label class equalization method of this application method embodiment.

[0214] The processor 1002 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the multi-tag class equalization method of this application can be completed by the integrated logic circuits in the hardware of the processor 1002 or by instructions in software form. The processor 1002 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 1001. The processor 1002 reads the information in the memory 1001 and, in conjunction with its hardware, performs the functions required by the units included in the multi-label class equalization device of this application embodiment, or executes the multi-label class equalization method of this application method embodiment.

[0215] The communication interface 1003 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the electronic device 1000 and other devices or communication networks. For example, data (such as the sample set including multiple instances in Embodiments 1 and 2 of this application) can be acquired through the communication interface 1003.

[0216] Bus 1004 may include a pathway for transmitting information between various components of electronic device 1000 (e.g., memory 1001, processor 1002, communication interface 1003).

[0217] Optionally, the electronic device 1000 may further include output components, such as a display, speakers, etc., which are used to display the parameters used to train the model to the developers. Therefore, the developers can obtain these parameters, modify them, and input the modified parameters into the electronic device 1000 via input components (e.g., mouse, keyboard, etc.). Additionally, the electronic device 1000 may also display the trained target model to the developers through the output components.

[0218] It should be understood that the acquisition unit 901 in the multi-label class equalization device 900 is equivalent to the communication interface 1003 in the electronic device 1000, and the sampling unit 902 can be equivalent to the processor 1002.

[0219] Please see Figure 11 , Figure 11 This is a schematic diagram of an electronic device according to an embodiment of the present invention. The electronic device 1100 includes a memory 1101, a processor 1102, a communication interface 1103, and a bus 1104. The memory 1101, the processor 1102, and the communication interface 1103 are interconnected via the bus 1104.

[0220] The memory 1101 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1101 may store a program, and when the program stored in the memory 1101 is executed by the processor 1102, the processor 1102 and the communication interface 1103 are used to execute the various steps of the target detection method.

[0221] The processor 1102 may be a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to perform the target detection method.

[0222] The processor 1102 can also be an integrated circuit chip with signal processing capabilities. During implementation, each step of the target detection method can be completed through integrated logic circuits in the hardware of the processor 1102 or through software instructions. The processor 1102 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 1101. The processor 1102 reads the information in the memory 1101 and combines it with its hardware to complete the target detection method.

[0223] The communication interface 1103 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the electronic device 1100 and other devices or communication networks. For example, the communication interface 1103 establishes a communication connection with the training device described above, and can receive parameters of the target model sent by the training device through the communication interface 1103. The parameters of the target model can be stored in the memory 1101 for later retrieval.

[0224] Bus 1104 may include a pathway for transmitting information between various components of electronic device 1100 (e.g., memory 1101, processor 1102, communication interface 1103).

[0225] Optionally, the electronic device 1100 may further include an output component, such as a display, an audio device, etc., which is used to display the target detection results obtained from the target model to the user.

[0226] The target detection method can be any existing target detection method, and will not be elaborated here.

[0227] It should be noted that, although Figure 10 and Figure 11The illustrated electronic device 1000 and electronic device 1100 only show the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, the electronic device 1000 and electronic device 1100 may also include other components necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the electronic device 1000 and electronic device 1100 may also include hardware components for implementing other additional functions. Moreover, those skilled in the art should understand that the device electronic device 1000 and electronic device 1100 may only include the components necessary for implementing the embodiments of this application, and may not necessarily include... Figure 10 or Figure 11 All the devices shown.

[0228] It can be understood that the electronic device 1000 is equivalent to the training device 120 in section 1, and the electronic device 1100 is equivalent to... Figure 1 The execution device 110 described herein. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0229] In addition to the methods and apparatus described above, embodiments of the present invention also provide a computer-readable storage medium storing instructions that, when executed on a processor, implement... Figure 2 or Figure 6 The multi-label class balancing method is shown.

[0230] A computer program product includes computer-executable instructions stored in a computer-readable storage medium; at least one processor of the device can read the computer-executable instructions from the computer-readable storage medium, and the at least one processor executes the computer-executable instructions to cause the device to perform... Figure 2 or Figure 6 The multi-label class balancing method is shown.

[0231] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0232] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0233] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0234] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0235] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0236] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multi-label class load balancing method, characterized in that, The method includes: Obtain a sample set comprising multiple images to be processed, each image including one or more category labels, the category labels being used to identify the type of objects in the image to be processed; The images to be processed in the sample set are classified into multiple classes according to the category labels, and each image to be processed is assigned to one or more classes. Based on the number of target category labels in each image to be processed in each sub-class and the number of all category labels in each image to be processed, the target sampling weight of each image to be processed in each sub-class is determined. The sub-class is the class whose number is less than a first threshold among the plurality of classes. The target category label is the category label corresponding to the sub-class in each image to be processed. The target sampling weight is the sampling weight corresponding to the sub-class in each image to be processed. Based on the target sampling weight of each image to be processed in the subclass, the images to be processed in each subclass are weighted and sampled.

2. The multi-label class equalization method as described in claim 1, characterized in that, The method further includes: The target image to be processed, determined based on the weighted sampling results, is used as a new image to be processed to construct the subclass.

3. The multi-label class equalization method as described in claim 2, characterized in that, The method further includes: Construct the number of images to be processed in the subclass to a first quantity; Determine whether the number of images to be processed in each sub-category has reached the second quantity; If the number of images to be processed in each subclass does not reach the second number, the images to be processed in the sample set are further classified into multiple classes according to the category label. The target sampling weight of each image to be processed in each subclass is further determined. The images to be processed in each subclass are further weighted and sampled. The target images to be processed determined according to the weighted sampling results are further used as new images to be processed to construct the subclass until the number of images to be processed in each subclass reaches the second number.

4. The multi-label class equalization method as described in claim 3, characterized in that, Before constructing the images to be processed in the subclass up to a first number, the method further includes: The first quantity is determined to be the average number of currently unprocessed images across all classes.

5. The multi-label class equalization method as described in claim 3, characterized in that, Before determining whether the number of images to be processed in each subclass has reached the second number, the method further includes: Determine the second quantity as Where r∈[0.5,0.8], The number of images to be processed in the largest class.

6. The multi-label class equalization method as described in any one of claims 1-5, characterized in that: The weighted sampling includes at least one of the weighted random sampling method (WRS) and the naive weighted sampling method.

7. The multi-label class equalization method as described in any one of claims 1-5, characterized in that, The determination of the target sampling weight for each image to be processed in each sub-class, based on the number of target category labels in each image to be processed within the sub-class and the total number of category labels in each image to be processed, includes: Based on the number of target category labels in each image to be processed in each sub-category and the total number of category labels in each image to be processed, determine the proportion of target category labels in each image to be processed in each sub-category; The target sampling weight for each image to be processed in each subclass is determined based on the proportion of the target category label for each image to be processed in each subclass.

8. The multi-label class equalization method as described in claim 7, characterized in that, The step of determining the proportion of target category labels in each image to be processed within each sub-category, based on the number of target category labels in each image to be processed within the sub-category and the total number of category labels in each image to be processed, includes: According to the formula Determine the proportion of the target category label for each image to be processed in each subclass, where, Let be the proportion of the target category label of the image i to be processed in subclass t. The number of target category labels for image i in subclass t. Let be the number of all category labels in the image i to be processed within subclass t.

9. The multi-label class equalization method as described in claim 7, characterized in that, The determination of the target sampling weight for each image to be processed in each sub-class based on the proportion of the target category label in each image to be processed in each sub-class includes: According to the formula Determine the target sampling weights for each image to be processed in each subclass, where, Let i be the target sampling weight of the image to be processed in subclass t. tn represents the proportion of the target category label of the image i to be processed in subclass t, and tn is the total number of all images to be processed in subclass t. This is the sum of the percentages of the target category labels for all images to be processed in subclass t.

10. A multi-label class equalization device, characterized in that, The device includes: An acquisition unit is used to acquire a sample set including multiple images to be processed, each image to be processed including one or more category labels, the category labels being used to indicate the type of objects in the image to be processed; A sampling unit is used to classify the images to be processed in the sample set into multiple classes according to the category labels, with each image to be processed being classified into one or more classes; The sampling unit is used to determine the target sampling weight of each image to be processed in each subclass based on the number of target category labels in each image to be processed and the number of all category labels in each image to be processed. The subclass is a class among a plurality of classes whose number is less than a first threshold, the target category label is the category label corresponding to the subclass in each image to be processed, and the target sampling weight is the sampling weight corresponding to the subclass in each image to be processed. The sampling unit is also used to perform weighted sampling on the images to be processed in each subclass according to the target sampling weight of each image to be processed in the subclass.

11. The multi-label class equalization device as described in claim 10, characterized in that: The sampling unit is also used to construct the subclass by using the target image to be processed determined based on the weighted sampling results as a new sample.

12. The multi-label class equalization device as described in claim 11, characterized in that: The sampling unit is also used to construct the images to be processed in the subclass up to a first quantity; The sampling unit is also used to determine whether the number of images to be processed in each subclass has reached the second number; The sampling unit is further configured to, if the number of images to be processed in each subclass does not reach the second number, continue to classify the images to be processed in the sample set according to the category label to form multiple classes, continue to determine the target sampling weight of each image to be processed in each subclass, and continue to perform weighted sampling on the images to be processed in each subclass until the number of images to be processed in each subclass reaches the second number.

13. The multi-label class equalization device as described in claim 12, characterized in that, Before constructing the images to be processed in the subclass up to the first number: The sampling unit is also used to determine that the first quantity is the average of the number of currently unprocessed images of all classes.

14. The multi-label class equalization device as described in claim 12, characterized in that, Before determining whether the number of images to be processed in each subclass has reached the second number: The sampling unit is further configured to determine the second quantity as... Where r∈[0.5,0.8], The number of images to be processed in the largest class.

15. The multi-label class equalization device as described in any one of claims 10-14, characterized in that: The weighted sampling includes at least one of the weighted random sampling method (WRS) and the naive weighted sampling method.

16. The multi-label class equalization device as described in any one of claims 10-14, characterized in that: The sampling unit is used to determine the proportion of target category labels in each image to be processed in each sub-class based on the number of target category labels in each image to be processed in each sub-class and the total number of category labels in each image to be processed. The sampling unit is also used to determine the target sampling weight of each image to be processed in each subclass based on the proportion of the target category label of each image to be processed in each subclass.

17. The multi-label class equalization device as described in claim 16, characterized in that: The sampling unit is also used to perform the following formula Determine the proportion of the target category label for each image to be processed in each subclass, where, For the image to be processed in subclass t The percentage of target category tags, The number of target category labels for image i in subclass t. Let be the number of all category labels in the image i to be processed within subclass t.

18. The multi-label class equalization device as described in claim 16, characterized in that: The sampling unit is also used to perform the following formula Determine the target sampling weights for each image to be processed in each subclass, where, For the image to be processed in subclass t The target sampling weights, tn represents the proportion of the target category label of the image i to be processed in subclass t, and tn is the total number of all images to be processed in subclass t. This is the sum of the percentages of the target category labels for all images to be processed in subclass t.

19. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory being used to store program instructions, and when the processor invokes the program instructions, it implements the multi-label class equalization method as described in any one of claims 1 to 9.

20. A vehicle, characterized in that, The vehicle includes the electronic equipment as described in claim 19.

21. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a program that causes a computer device to implement the multi-label class equalization method as described in any one of claims 1 to 9.

22. A computer program product, characterized in that, The computer program product includes computer-executable instructions stored in a computer-readable storage medium; at least one processor of the device can read the computer-executable instructions from the computer-readable storage medium, and the at least one processor executes the computer-executable instructions to cause the device to perform the multi-label class equalization method as described in any one of claims 1 to 9.

Images