Training sample screening method and device, equipment and storage medium
By using an attribute classification model to perform hierarchical clustering and active learning on the training samples, the problem of imbalanced training sample ratio in deep learning models is solved, improving model performance and reducing iteration time.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU HIKVISION DIGITAL TECHNOLOGY CO LTD
- Filing Date
- 2022-09-30
- Publication Date
- 2026-07-31
AI Technical Summary
Deep learning models suffer from an imbalance in the proportion of training samples, with easy samples making up the majority. This prevents the model from fully learning the features of difficult samples. Increasing the number of difficult samples increases the number of training samples, reducing the efficiency of iterative development.
By constructing an attribute classification model, the training samples are hierarchically clustered and divided into different categories. Difficult samples are selected from each category for active learning, and the sample ratio is adjusted to achieve weight reduction and performance improvement.
While balancing the proportion of training samples, we increase the proportion of difficult samples to improve model performance, reduce iteration time, and improve iteration development efficiency.
Smart Images
Figure CN115565030B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of deep learning, and in particular to a method, apparatus, device, and storage medium for selecting training samples. Background Technology
[0002] In practical industrial applications, deep learning models (such as object detection models) often accumulate a large number of training samples as projects iterate. These training samples can include simple and difficult samples. However, these numerous training samples often exhibit an imbalance in proportion; for example, simple samples may constitute the vast majority of the training samples, while difficult samples may make up a very small portion. Using such an imbalanced training sample structure will cause the deep learning model to learn only the features of the simple samples, while failing to learn the features of the difficult samples.
[0003] To address the imbalance in the training sample ratio, the number of difficult samples can be increased to achieve a balanced training sample ratio. For example, difficult samples can be resampled.
[0004] However, balancing the proportion of training samples by increasing the number of difficult samples will lead to a further increase in the number of training samples, reducing the efficiency of iterative development of the object detection model. Summary of the Invention
[0005] Based on the above-mentioned technical problems, this application provides a method, apparatus, device and storage medium for screening training samples. It can classify training samples by constructing an attribute classification model and adjust the sample ratio under each category, thereby reducing the number of training samples while balancing the proportion of training samples.
[0006] In a first aspect, this application provides a method for selecting training samples, the method comprising: obtaining a set of training samples to be processed; the set of training samples to be processed being used to train a deep learning model; the set of training samples to be processed including multiple training samples to be processed; using an attribute classification model to perform hierarchical clustering on the set of training samples to be processed to construct a classification tree; the attribute classification model being used to identify the target attributes of the training samples to be processed; the target attributes being used to characterize the types of influence of the training samples to be processed on the feature learning of the deep learning model; dividing the set of training samples to be processed into N categories according to the classification tree; each category including at least one training sample to be processed; N being an integer greater than 1; selecting a sample subset from each of the N categories to obtain N sample subsets; the sample subsets being obtained by actively learning from difficult samples in a category; and merging the N sample subsets to obtain a first training sample set.
[0007] It should be understood that training samples with different target attributes have varying effects on the bias of feature learning generated by the object detection model. The training sample selection method provided in this application can utilize an attribute classification model to perform hierarchical clustering of the training sample set to be processed, dividing the training samples into different categories, and selecting a subset of samples obtained by actively learning from difficult samples from the training samples in different categories. This reduces the size of the training sample set to be processed while increasing the proportion of difficult samples, thereby improving model performance and reducing the time consumed by model training iterations, while maintaining a balance among all categories.
[0008] Optionally, each training sample to be processed includes a ground truth label; after randomly selecting a first training sample set from each of the N categories according to a preset first ratio, the method further includes: training the target detection model based on the first training sample set until convergence; inputting the remaining training samples to be processed from the N categories other than the first training sample set into the target detection model trained to convergence, to obtain the detection value of each remaining training sample to be processed; selecting M remaining training samples to be processed from each of the N categories whose overlap between the detection value and the ground truth label is less than a preset threshold, and adding them to the first training sample set; M is a natural number.
[0009] In one possible implementation, for the first category among N categories, a sample subset is selected from each of the N categories, including: randomly selecting an initial training sample set from the first category according to a first proportion; training the deep learning model based on the initial training sample set until convergence; selecting difficult samples from the remaining unprocessed training samples in the first category other than the initial training sample set based on the converged deep learning model; and adding the selected difficult samples to the initial training sample set to obtain the sample subset.
[0010] In one possible implementation, each training sample to be processed includes a ground truth label; based on a deep learning model trained to convergence, difficult samples are selected from the remaining training samples to be processed in the first category excluding the initial training sample set, including: inputting the remaining training samples to be processed in the first category excluding the initial training sample set into the deep learning model trained to convergence to obtain a predicted value for each remaining training sample; selecting M% of the remaining training samples whose predicted value overlaps with the ground truth label with a value less than a threshold; where M is a natural number.
[0011] In one possible implementation, selecting M% of the training samples whose predicted values and ground truth labels have an overlap of less than a threshold from the remaining training samples to be processed includes: sorting the remaining training samples to be processed in ascending order of the overlap between the predicted values and ground truth labels to obtain an overlap ranking result; and selecting the top M% of the remaining training samples to be processed from the overlap ranking result whose overlap is less than a preset threshold.
[0012] Optionally, the overlap between predicted and true labels is the crossover ratio (CRO).
[0013] Optionally, the classification tree includes multiple layers; each layer includes multiple tree nodes; each tree node includes one or more training samples to be processed; the target attributes of the training samples to be processed in different tree nodes of the same layer are different. The training sample set to be processed is divided into N categories according to the classification tree, including: selecting the target layer in the classification tree; the target layer is any layer in the multiple layers of the classification tree; and dividing the training sample set to be processed into N categories according to the target attributes of different tree nodes in the target layer.
[0014] It should be understood that different layers in a classification tree have different classification granularities. Managers can select different target layers to divide the training sample sets to be processed according to different granularities, thereby improving the flexibility of the training sample selection process.
[0015] Optionally, the attribute classification model is obtained by training a preset initial model based on an attribute training sample set; the attribute training sample set includes multiple attribute training samples; each attribute training sample set includes a training sample to be processed and a label corresponding to the training sample to be processed; the label is the target attribute.
[0016] Secondly, this application provides a training sample screening device, which includes various modules for the method described in the first aspect above.
[0017] Thirdly, this application provides a training method for an attribute classification model, the method comprising: obtaining an attribute training sample set; the attribute training sample set including multiple attribute training samples; each attribute training sample set including a training sample to be processed and a label corresponding to the training sample to be processed; the label being a target attribute; the training sample to be processed being used to train an object detection model; the target attribute being used to characterize the type of influence of the training sample to be processed on the feature learning of the object detection model; and training a preset initial model based on the attribute training sample set to obtain an attribute classification model.
[0018] Fourthly, this application provides a training apparatus for an attribute classification model, which includes various modules for the method described in the third aspect above.
[0019] Fifthly, this application provides a computer program product that, when run on a computer, causes the computer to perform the steps of the related methods described in the first or third aspect above, so as to implement the methods described in the first or third aspect above.
[0020] In a sixth aspect, this application provides an electronic device including a processor and a memory; the memory stores processor-executable instructions; when the processor is configured to execute the instructions, the electronic device performs the method described in the first or third aspect above.
[0021] In a seventh aspect, this application provides a computer-readable storage medium comprising: computer software instructions; which, when executed in an electronic device, cause the electronic device to perform the method described in the first or third aspect above.
[0022] Eighthly, this application provides a chip including a processor and an interface. The processor is coupled to a memory through the interface. When the processor executes a computer program in the memory or an electronic device executes instructions, the methods described in the first or third aspect are executed.
[0023] The beneficial effects of aspects two through eight mentioned above can be referred to in aspect one, and will not be repeated here. Attached Figure Description
[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0025] Figure 1 This is a schematic diagram of the composition of the training sample screening system provided in the embodiments of this application;
[0026] Figure 2 A schematic diagram illustrating the composition of the second computing processing device 300 provided in an embodiment of this application;
[0027] Figure 3 A flowchart illustrating the training sample selection method provided in this application embodiment;
[0028] Figure 4 Another flowchart illustrating the training sample selection method provided in this application embodiment;
[0029] Figure 5 This is a schematic diagram of the IOU provided for an embodiment of this application;
[0030] Figure 6 A flowchart illustrating the training method for the attribute classification model provided in this application embodiment;
[0031] Figure 7 A schematic diagram illustrating the composition of the training sample screening device provided in the embodiments of this application;
[0032] Figure 8 This is a schematic diagram of the composition of the training device for the attribute classification model provided in the embodiments of this application. Detailed Implementation
[0033] Hereinafter, the terms "first," "second," and "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," or "third," etc., may explicitly or implicitly include one or more of that feature.
[0034] First, the terms used in the embodiments of this application will be explained.
[0035] 1. Simple samples: Training samples whose predicted values obtained by deep learning models have small errors compared to the true labels.
[0036] 2. Difficult samples: Training samples whose predicted values obtained using deep learning models have a large error compared to the true labels.
[0037] 3. Active learning: Difficult samples are obtained through machine learning methods, and then manually confirmed and reviewed. The training samples obtained by manual annotation are then used to train a supervised or semi-supervised learning model again, integrating human experience into the machine learning model to gradually improve the model's performance.
[0038] 4. Hierarchical Clustering: Hierarchical clustering refers to grouping samples into hierarchical categories when a hierarchical structure is assumed to exist between them. Hierarchical clustering can be divided into aggregate clustering and split clustering. Aggregate clustering involves initially assigning each sample to a single category, then merging two closely spaced categories to create a new category, repeating this process until a stopping condition is met to obtain hierarchical categories. Split clustering involves initially assigning all samples to a single category, then assigning samples from the existing categories that are furthest apart to two new categories, repeating this process until a stopping condition is met to obtain hierarchical categories.
[0039] Deep learning models can include object detection models in the field of computer vision.
[0040] Optionally, the object detection model can be applied to face detection. For example, it can be used for facial payment, facial recognition on mobile phones, facial recognition in access control systems, and facial temperature measurement at train stations.
[0041] Alternatively, object detection models can also be used for vehicle detection. For example, license plate segmentation and recognition, traffic condition assessment, autonomous driving, lane detection, etc.
[0042] Optionally, object detection models can also be used for people counting. For example, detecting the number of people at a meeting or the number of people waiting to cross a zebra crossing.
[0043] Optionally, the target detection model can also be used for target detection in remote sensing images. For example, it can be used for building detection, power tower detection, airport detection, aircraft detection, ship detection, playground detection, ground feature extraction, and route extraction in remote sensing images.
[0044] Optionally, object detection models can also be used in the medical field, such as medical image recognition and retinal segmentation.
[0045] In practical industrial applications, object detection models often accumulate a large number of training samples as projects iterate. These training samples can include both simple and difficult samples.
[0046] These large training samples often exhibit an imbalance in proportion; for example, easy samples constitute the vast majority of the training samples, while difficult samples make up a very small portion, with the small number of difficult samples being submerged among the vast majority of easy samples. Using such an imbalanced training sample structure causes the object detection model to learn only the features of the easy samples, failing to fully learn the features of the difficult samples.
[0047] To address the imbalance in the training sample ratio, the number of difficult samples can be increased to achieve a balanced training sample ratio. For example, difficult samples can be resampled.
[0048] However, balancing the proportion of training samples by increasing the number of difficult samples will lead to a further increase in the number of training samples, reducing the efficiency of iterative development of the object detection model.
[0049] Based on this, the embodiments of this application provide a method for screening training samples. By constructing an attribute classification model to classify the training samples and adjusting the sample ratio under each category, the proportion of training samples is balanced while the training sample size is reduced, thereby improving the iterative development efficiency of the target detection model.
[0050] The following description is provided in conjunction with the accompanying drawings.
[0051] Figure 1 This is a schematic diagram illustrating the composition of the training sample screening system provided in an embodiment of this application. Figure 1As shown, the system may include an image acquisition device 100, a first computing processing device 200, and a second computing processing device 300. The image acquisition device 100, the first computing processing device 200, and the second computing processing device 300 can be connected via a wired network or a wireless network.
[0052] The image acquisition device 100 can be a camera installed in various application scenarios of the above-mentioned target detection model.
[0053] Image acquisition device 100 is used to acquire images within a field of view. Examples include color images and / or depth images. A depth image is an image whose pixel values are the depth values (vertical distances) from the image acquisition device 100 to various points within the field of view.
[0054] In some embodiments, the image acquisition device 100 may use a color camera to acquire color images.
[0055] For example, a color camera can be a red-green-blue (RGB) camera. An RGB camera uses the RGB color mode, obtaining various colors through variations in the red, green, and blue color channels and their superposition. Typically, an RGB camera uses three different cables to provide the three basic color components, and three independent charge-coupled device (CCD) sensors to acquire the three color signals.
[0056] In some embodiments, the image acquisition device 100 may employ a depth camera to acquire depth images.
[0057] For example, a depth camera can be a time-of-flight (TOF) camera. A TOF camera uses a laser light source to emit modulated pulsed infrared light. The pulsed infrared light is reflected after hitting an object. The light source detector of the TOF camera can receive the light reflected from the object and calculate the distance between the TOF camera and the object being photographed by calculating the time difference or phase difference between the emission and reflection of the light source. Then, based on the distance between the TOF camera and the object being photographed, the depth value of each point in the field of view can be obtained.
[0058] It should be noted that the image acquisition device 100 has been described above using color images and depth images as examples. The image acquisition device 100 can also be used to acquire other types of images, such as infrared images. This application embodiment does not limit this.
[0059] In some embodiments, the image acquisition device 100 can also be used to send the acquired images within the field of view to the first computing processing device 200.
[0060] As described above, the image acquisition device 100, the first computing processing device 200, and the second computing processing device 300 can be connected via a wired network or a wireless network. This wired or wireless network may include one or more media, devices, or lines capable of transmitting images acquired by the image acquisition device 100 to the first computing processing device 200.
[0061] In some embodiments, the wired or wireless network may include one or more communication media that enable the image acquisition device 100 to transmit the acquired images directly to the first computing processing device 200 in real time. In this embodiment, the image acquisition device 100 may modulate the image according to a communication standard (e.g., a wireless communication protocol) and transmit the modulated image to the first computing processing device 200. The one or more communication media may include wireless and / or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines.
[0062] Optionally, the one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (e.g., the Internet).
[0063] The first computing processing device 200 can be an electronic device with computing processing capabilities, such as a computer or a server. The server can be a single server or a server cluster consisting of multiple servers. In some embodiments, the server cluster can also be a distributed cluster. Optionally, the server can also be implemented on a cloud platform, which can include, for example, private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, inter-cloud, and multi-cloud, or any combination thereof.
[0064] Optionally, the first computing processing device 200 may also be an electronic device such as a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc.
[0065] The first computing processing unit 200 may have a preset target detection model. The first computing processing unit 200 is used to detect the image acquired by the image acquisition device 100 using the preset target detection model and identify the target object in the image.
[0066] The specific form of the second computing processing device 300 can be referred to the first computing processing device 200 described above, and will not be repeated here.
[0067] The second computing processing unit 300 is used to perform the following... Figure 3 The training sample selection method shown obtains a first training sample set with a balanced ratio. Based on this first training sample set, a target detection model is trained, and the trained target detection model is sent to the first computing processing device 200. The specific process for obtaining the first training sample set can be referred to below. Figure 3 As mentioned above, it will not be repeated here.
[0068] It should be noted that the above example illustrates that the image acquisition device 100, the first computing processing device 200, and the second computing processing device 300 are each an independent device. Optionally, the image acquisition device 100, the first computing processing device 200, and the second computing processing device 300 can also be combined into one device, that is, the functions of the image acquisition device 100 or its corresponding functions, the functions of the first computing processing device 200 or its corresponding functions, and the functions of the second computing processing device 300 or its corresponding functions can be integrated into the same device. Alternatively, the first computing processing device 200 and the second computing processing device 300 can be combined into one device, that is, the functions of the first computing processing device 200 or its corresponding functions, and the functions of the second computing processing device 300 or its corresponding functions can be integrated into the same device. This application does not limit this aspect.
[0069] The execution entity of the training sample screening method provided in this application embodiment can be the aforementioned second computing processing device 300. As described above, the second computing processing device 300 can be an electronic device such as a computer, server, mobile phone, or tablet computer; or, the second computing processing device 300 can be an application (APP) installed in the aforementioned electronic device that provides training sample screening functionality; or, the second computing processing device 300 can be the central processing unit (CPU) in the aforementioned electronic device; or, the second computing processing device 300 can be a functional module in the aforementioned electronic device used to execute the training sample screening method. This application embodiment does not impose any limitations on this.
[0070] Figure 2 This is a schematic diagram illustrating the composition of the second computing processing device 300 provided in an embodiment of this application. Figure 2As shown, the second computing processing device 300 may include: a processor 10, a memory 20, a communication line 30, a communication interface 40, and an input / output interface 50.
[0071] The processor 10, memory 20, communication interface 40, and input / output interface 50 can be connected via communication line 30.
[0072] Processor 10 is used to execute instructions stored in memory 20 to implement the training sample screening method provided in the following embodiments of this application. Processor 10 may be a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller, a programmable logic device (PLD), or any combination thereof. Processor 10 may also be any other device with processing capabilities, such as a circuit, device, or software module; this application embodiment does not limit this. In one example, processor 10 may include one or more CPUs, for example... Figure 2 CPU0 and CPU1 are mentioned. As an optional implementation, the electronic device may include multiple processors; for example, in addition to processor 10, it may also include processor 60. Figure 2 (The example shown is a dashed line).
[0073] The memory 20 is used to store instructions. For example, the instructions may be computer programs. Optionally, the memory 20 may be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions; it may also be a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions; it may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, etc. The embodiments of this application do not limit this.
[0074] It should be noted that the memory 20 may exist independently of the processor 10 or may be integrated with the processor 10. The memory 20 may be located within the second computing processing device 300 or outside the second computing processing device 300, and the embodiments of this application do not impose any restrictions on this.
[0075] Communication line 30 is used to transmit information between the various components included in the electronic device. Communication line 30 can be an industry standard architecture (ISA) line, a peripheral component interconnect (PCI) line, or an extended industry standard architecture (EISA) line, etc. This communication line 30 can be divided into address lines, data lines, control lines, etc. For ease of representation, Figure 2 The line is represented by a single solid line, but this does not mean that there is only one line or one type of line.
[0076] The communication interface 40 is used to communicate with other devices (such as the image acquisition device 100 and the first computing processing device 200) or other communication networks. These other communication networks can be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 40 can be a module, circuit, transceiver, or any device capable of enabling communication.
[0077] The input / output interface 50 is used to realize human-computer interaction between the user and the second computing processing device 300. For example, it can realize action interaction, text interaction, or voice interaction between the user and the second computing processing device 300.
[0078] For example, the input / output interface 50 may be a mouse, keyboard, display screen or touch screen, etc., through which the user and the second computing processing device 300 can realize work interaction or text interaction.
[0079] It should be noted that, Figure 2 The structure shown does not constitute a limitation on the second computing processing device 300 (electronic device), except... Figure 2 In addition to the components shown, the second computing processing device 300 (electronic device) may include more or fewer components than shown, or combinations of certain components, or different component arrangements.
[0080] The method for selecting training samples provided in the embodiments of this application will now be described with reference to the accompanying drawings.
[0081] Figure 3 This is a flowchart illustrating the training sample selection method provided in an embodiment of this application. Optionally, this method can be implemented by someone with the above-described... Figure 2 The second computing processing unit of the hardware structure shown executes the operation. For example... Figure 3 As shown, the method includes steps S101 to S105.
[0082] S101, The second computing processing device acquires the training sample set to be processed.
[0083] The training sample set to be processed is used to train the deep learning model. The specific types of deep learning models can be found in the section on various application scenarios of object detection models mentioned above, and will not be repeated here. The training sample set to be processed includes multiple training samples. Taking the object detection model mentioned above as an example, each training sample to be processed includes an image and a ground truth label corresponding to that image. The ground truth label is the location information of the target object in the image. The images in the training samples to be processed can be acquired by the image acquisition device 100 mentioned above. The ground truth labels in the training samples to be processed can be manually labeled by the user through the input / output interface 50 mentioned above.
[0084] S102. The second computing processing device uses the attribute classification model to perform hierarchical clustering on the training samples to be processed and construct a classification tree.
[0085] The attribute classification model is used to identify the target attributes of the training samples to be processed, and the target attributes characterize the types of influence of the training samples on the target detection model. The attribute classification model can be pre-installed in the second computing processing device, and the training process of the attribute classification model can be referred to the following... Figure 6 As mentioned above, it will not be repeated here. Target attributes can also be manually set by administrators. For example, target attributes can be set by administrators when training the attribute classification model by labeling the ground truth labels of the training samples.
[0086] For example, the target attributes of the training samples to be processed may include any one of the following: normal, occluded, distorted, overexposed, and underexposed.
[0087] For example, taking the target attributes of the training samples to be processed as normal, occluded, distorted, overexposed, and underexposed as an example, assuming that the classification tree includes three layers of tree nodes from top to bottom, the first layer of tree nodes (or the root node) of the classification tree includes all the training samples to be processed, that is, the set of training samples to be processed.
[0088] The second-level tree nodes of the classification tree (or child nodes of the root node) can include normal child nodes, occluded child nodes, distorted child nodes, overexposed child nodes, and underexposed child nodes. Normal child nodes can include training samples whose target attribute is identified as normal using the attribute classification model. Occluded child nodes can include training samples whose target attribute is occluded using the attribute classification model. Distorted child nodes can include training samples whose target attribute is distorted using the attribute classification model. Overexposed child nodes can include training samples whose target attribute is overexposed using the attribute classification model. Underexposed child nodes can include training samples whose target attribute is underexposed using the attribute classification model.
[0089] The third-level tree nodes of the classification tree (or grandchildren of the root node) can include normal grandchildren, 5% occluded grandchildren, 10% occluded grandchildren, 20% occluded grandchildren, pincushion distortion distorted grandchildren, barrel distortion distorted grandchildren, 5% overexposed grandchildren, 10% overexposed grandchildren, 20% overexposed grandchildren, 5% underexposed grandchildren, 10% underexposed grandchildren, and 20% underexposed grandchildren, etc. The types of training samples to be processed in each grandchild node can be referred to as the types of training samples to be processed in the child nodes above, and will not be repeated here.
[0090] S103, The second computing and processing device divides the training sample set to be processed into N categories according to the classification tree.
[0091] Where N is an integer greater than 1. Each category includes at least one training sample to be processed.
[0092] In one possible implementation, as described above, the classification tree comprises multiple layers, each layer comprising multiple tree nodes; each tree node comprises one or more training samples to be processed; the target attributes of the training samples to be processed in different tree nodes of the same layer are different. In this case, S103 may specifically include: the second computing processing device selecting a target layer in the classification tree; the target layer being any layer in the multiple layers of the classification tree; the second computing processing device dividing the training sample set to be processed into N categories according to the target attributes of different tree nodes in the target layer.
[0093] For example, taking the above classification tree as having three layers of tree nodes, assuming the target layer is the second layer, the second computing processing device can divide the training sample set to be processed into five categories: normal, occluded, distorted, overexposed, and underexposed, that is, N is 5.
[0094] For example, taking the above classification tree as having three layers of tree nodes, assuming the target layer is the third layer, the second computing processing device can divide the training sample set to be processed into 12 categories: normal, 5% occlusion, 10% occlusion, 20% occlusion, pincushion distortion, barrel distortion, 5% overexposure, 10% overexposure, 20% overexposure, 5% underexposure, 10% underexposure, and 20% underexposure, that is, N is 12.
[0095] Optionally, the second computing processing device can obtain the number of training samples to be retained, and divide the training sample set to be processed into N categories according to the number of training samples to be retained and the classification tree.
[0096] It should be understood that different layers in a classification tree have different classification granularities. Managers can select different target layers to divide the training sample sets to be processed according to different granularities, thereby improving the flexibility of the training sample selection process.
[0097] S104. The second computing processing device selects a sample subset from each of the N categories to obtain N sample subsets.
[0098] The sample subset is obtained through active learning of difficult samples of a category. The ratio of difficult to easy samples in the sample subset can change with repeated active learning during the screening process, but the trend is to increase the proportion of difficult samples and decrease the proportion of easy samples. The specific process of active learning can be referred to in the following embodiments, and will not be repeated here.
[0099] S105, The second computing and processing device combines the N sample subsets to obtain the first training set.
[0100] It should be understood that training samples with different target attributes have varying effects on the bias of feature learning generated by the object detection model. The training sample selection method provided in this application can utilize an attribute classification model to perform hierarchical clustering of the training sample set to be processed, dividing the training samples into different categories, and selecting a subset of samples obtained by actively learning from difficult samples from the training samples in different categories. This reduces the size of the training sample set to be processed while increasing the proportion of difficult samples, thereby improving model performance and reducing the time consumed by model training iterations, while maintaining a balance among all categories.
[0101] The following is an introduction to the process of active learning.
[0102] Figure 4 This is another flowchart illustrating the training sample selection method provided in the embodiments of this application. Figure 4As shown, taking N categories including the first category as an example, for the first category, in S104 above, a sample subset is selected from each of the N categories, which can specifically include S201 to S204.
[0103] S201, The second computing processing device randomly selects an initial training sample set from the first category according to a first ratio.
[0104] In one possible implementation, the first percentage can be preset by the administrator. For example, the first percentage could be 10%, 15%, or 20%, etc. This application embodiment does not limit the specific value of the first percentage.
[0105] S202, the second computing processing device trains the deep learning model based on the initial training sample set until it converges.
[0106] The initial training sample set includes multiple training samples to be processed.
[0107] Optionally, the second computational processing unit can input one or more training samples to be processed into the deep learning model each time to obtain predicted values, calculate a loss function based on the predicted values and the labels (ground values) of the training samples to be processed, and adjust the parameters in the deep learning model. This process is repeated, inputting multiple training samples from the initial training sample set into the deep learning model sequentially, and iteratively training until the deep learning model converges.
[0108] Optionally, the convergence conditions of a deep learning model may include: the number of times the training samples to be processed input into the deep learning model reaches a preset first threshold, or the error between the predicted value and the label output by the deep learning model is less than a preset first error threshold.
[0109] Both the first quantity threshold and the first error threshold can be preset by the administrator. This application embodiment does not limit the specific values of the first quantity threshold and the first error threshold.
[0110] S203. The second computational processing is based on a deep learning model trained to convergence, selecting difficult samples from the remaining unprocessed training samples in the first category, excluding the initial training sample set.
[0111] Optionally, S203 may specifically include the following steps:
[0112] Step 1: The second computing and processing device inputs the remaining training samples to be processed in the first category, excluding the initial training sample set, into the deep learning model that has been trained to convergence, and obtains the predicted value of each remaining training sample to be processed.
[0113] Step 2: The second computing and processing device selects M% of the training samples whose predicted values and true labels have an overlap of less than a threshold from the remaining training samples to be processed.
[0114] Where M is a natural number. The preset threshold can be preset by the administrator. For example, the preset threshold can be 10% or 20%, etc. This application embodiment does not limit the specific value of the preset threshold.
[0115] Optionally, M can be a preset value. In this case, step 2 above can specifically include: the second calculation and processing device sorts the remaining training samples to be processed according to the order of the overlap between the predicted value and the true value label from smallest to largest, and obtains the overlap ranking result; the second calculation and processing device selects the top M% of the remaining training samples to be processed from the overlap ranking result.
[0116] Optionally, M can be a statistical value. In this case, step 2 above can specifically include: when the overlap between the predicted value and the true label of the remaining training sample to be processed is less than a preset threshold, the second computing processing device adds the remaining training sample to the first training sample set.
[0117] Optionally, M can be a statistical value. In this case, step 2 above can specifically include: the second calculation and processing device sorts the remaining training samples to be processed according to the order of overlap between the predicted value and the true value label from smallest to largest, and obtains the overlap ranking result; the second calculation and processing device selects the top M% of the remaining training samples to be processed in the overlap ranking result whose overlap is less than a preset (overlap) threshold.
[0118] In one possible implementation, the overlap between the detected value and the ground truth label is the intersection over union (IOU). That is, the second computational processing device can calculate the IOU of each remaining training sample as the overlap between the detected value and the ground truth label based on the detected value and the ground truth label of that remaining training sample.
[0119] For example, Figure 5 This is a schematic diagram of the IOU provided for an embodiment of this application. Figure 5 As shown, for a training sample with a ground truth label, the IOU of the training sample is equal to the quotient of the first intersection and the first union. The first intersection is the intersection of the predicted values and ground truth labels of the training sample. The first union is the union of the predicted values and ground truth labels of the training sample.
[0120] S204. The second calculation process adds the selected difficult samples to the initial training sample set to obtain a sample subset.
[0121] Here, the sample subset is the same as the sample subset corresponding to the first category.
[0122] In some embodiments, prior to S101, the second computing processing device may also acquire the trained attribute classification model.
[0123] The attribute classification model can be obtained by the second computing device training the initial model in advance based on the attribute training sample set. Optionally, the attribute classification model can also be obtained by any other computing device with computing processing capabilities training the initial model in advance based on the attribute training sample set and then sending it to the second computing device, or it can be transferred to the second computing device through an intermediate storage medium. This application embodiment does not impose any limitations on this.
[0124] Optionally, for the sake of simplicity, the following description will take the example of obtaining an attribute classification model after the second computing processing device has trained the initial model based on the attribute training samples.
[0125] Figure 6 This is a flowchart illustrating the training method for the attribute classification model provided in this application embodiment. Optionally, this method can be implemented by someone with the above-described... Figure 2 The second computing processing unit of the hardware structure shown executes, as follows: Figure 6 As shown, before S101, the method may also include S301 to S302.
[0126] S301, The second computing processing device acquires the attribute training sample set.
[0127] The attribute training sample set can include multiple attribute training samples. Each attribute training sample can include a training sample to be processed and a label corresponding to that training sample. The label is the target attribute. The target attribute can be obtained through manual annotation.
[0128] S302, the second computing processing device trains the preset initial model based on the attribute training sample set to obtain the attribute classification model.
[0129] Optionally, the second computing processing device can input one or more attribute training samples into the initial model each time to obtain the predicted value (the target attribute predicted by the initial model), calculate the loss function based on the predicted value and the label in the attribute training sample, and so on, input multiple attribute training samples in the attribute training sample set into the initial model in sequence, adjust the parameters of the initial model, and iteratively train until the initial model converges.
[0130] Optionally, the conditions for the initial model to converge (end training) may include: the second computing device inputting attribute training samples into the initial model a number of times reaches a preset second number threshold, or the error between the predicted value and the label is less than a preset second error threshold.
[0131] The second quantity threshold and the second error threshold can both be preset by the administrator. The second quantity threshold can be the same as or different from the first quantity threshold. The second error threshold can be the same as or different from the first error threshold. This application embodiment does not limit the specific values of the second quantity threshold and the second error threshold.
[0132] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the aforementioned functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0133] In an exemplary embodiment, this application also provides a training sample screening device, which can be applied to the second computing processing device described above. Figure 7 This is a schematic diagram illustrating the composition of the training sample screening device provided in an embodiment of this application. Figure 7 As shown, the device may include an acquisition module 701 and a processing module 702.
[0134] The acquisition module 701 is used to acquire the training sample set to be processed; the training sample set to be processed is used to train the deep learning model; the training sample set to be processed includes multiple training samples to be processed.
[0135] The processing module 702 is used to perform hierarchical clustering of the training sample set to be processed using an attribute classification model to construct a classification tree; the attribute classification model is used to identify the target attributes of the training samples to be processed; the target attributes are used to characterize the types of influence of the training samples to be processed on the feature learning of the deep learning model; the training sample set to be processed is divided into N categories according to the classification tree; each category includes at least one training sample to be processed; N is an integer greater than 1; a sample subset is selected from each of the N categories to obtain N sample subsets; the sample subsets are obtained by actively learning from the difficult samples in a category; the N sample subsets are merged to obtain the first training sample set.
[0136] In some possible embodiments, for the first category among N categories, the processing module 702 is specifically used to randomly select an initial training sample set from the first category according to a first ratio; train the deep learning model based on the initial training sample set until convergence; select difficult samples from the remaining unprocessed training samples in the first category other than the initial training sample set based on the deep learning model trained to convergence; and add the selected difficult samples to the initial training sample set to obtain a sample subset.
[0137] In other possible embodiments, each training sample to be processed includes a ground truth label. The processing module 702 is specifically configured to input the remaining training samples to be processed in the first category, excluding the initial training sample set, into a deep learning model trained to convergence, to obtain a predicted value for each remaining training sample; and to select M training samples from the remaining training samples whose predicted values have an overlap with the ground truth label less than a threshold; M is a natural number.
[0138] In some other possible embodiments, the processing module 702 is specifically used to sort the remaining training samples to be processed in ascending order of the overlap between the predicted value and the true value label to obtain the overlap sorting result; and select the top M remaining training samples to be processed in the overlap sorting result whose overlap is less than a preset threshold.
[0139] In some other possible embodiments, the degree of overlap between the predicted value and the true value label is the intersection-union ratio (IOU).
[0140] In some other possible embodiments, the classification tree includes multiple layers; each layer includes multiple tree nodes; each tree node includes one or more training samples to be processed; the target attributes of the training samples to be processed in different tree nodes of the same layer are different. The processing module 702 is specifically used to select the target layer in the classification tree; the target layer is any layer in the multiple layers of the classification tree; and according to the target attributes of different tree nodes of the target layer, the training sample set to be processed is divided into N categories.
[0141] In some other possible embodiments, the attribute classification model is obtained by training a preset initial model based on an attribute training sample set; the attribute training sample set includes multiple attribute training samples; each attribute training sample set includes a training sample to be processed and a label corresponding to the training sample to be processed; the label is the target attribute.
[0142] In an exemplary embodiment, this application also provides a training device for an attribute classification model, which can be applied to the second computing processing device described above. Figure 8 This is a schematic diagram illustrating the composition of a training apparatus for an attribute classification model provided in an embodiment of this application. Figure 8 As shown, the device may include an acquisition module 801 and a processing module 802.
[0143] The acquisition module 801 is used to acquire an attribute training sample set; the attribute training sample set includes multiple attribute training samples; each attribute training sample set includes a training sample to be processed and a label corresponding to the training sample to be processed; the label is the target attribute; the training sample to be processed is used to train the target detection model; the target attribute is used to characterize the type of influence of the training sample to be processed on the feature learning of the target detection model.
[0144] The processing module 802 is used to train a preset initial model based on the attribute training sample set to obtain an attribute classification model.
[0145] It should be noted that, Figure 7 and Figure 8 The module division shown is illustrative and represents only one logical functional division; in actual implementation, other division methods are possible. For example, two or more functions can be integrated into a single processing module. These integrated modules can be implemented either in hardware or as software functional modules.
[0146] In an exemplary embodiment, this application also provides a computer-readable storage medium including computer-executable instructions that, when executed on an electronic device (second computing processing device), cause the electronic device (second computing processing device) to perform any of the methods provided in the above embodiments.
[0147] In an exemplary embodiment, this application also provides a computer program product containing computer execution instructions, which, when run on an electronic device (second computing processing device), causes the electronic device (second computing processing device) to execute any of the methods provided in the above embodiments.
[0148] In an exemplary embodiment, this application also provides a chip, including: a processor and an interface, wherein the processor is coupled to a memory through the interface, and when the processor executes computer programs in the memory or electronic devices (second computing processing devices) execute instructions, any of the methods provided in the above embodiments are executed.
[0149] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer-executable instructions. When these computer-executable instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-executable instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer-executable instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks, SSDs).
[0150] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.
[0151] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.
[0152] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A method for selecting training samples, characterized in that, The method includes: Obtain a training sample set to be processed; the training sample set to be processed is used to train the target detection model; the training sample set to be processed includes multiple training samples to be processed; the training samples to be processed include images and ground truth labels corresponding to the images, the ground truth labels are used to represent the position information of the target objects in the images; Hierarchical clustering is performed on the training sample set to be processed using an attribute classification model to construct a classification tree; the attribute classification model is used to identify the target attribute of the training sample to be processed; the target attribute includes at least one of the following: normal, occluded, distorted, overexposed, and underexposed. The training sample set to be processed is divided into N categories according to the classification tree; each category includes at least one training sample to be processed; N is an integer greater than 1; A subset of samples is selected from each of the N categories to obtain N sample subsets; each sample subset is obtained by actively learning from the difficult samples in a category. For the first category among the N categories, the step of selecting a sample subset from each of the N categories includes: randomly selecting an initial training sample set from the first category according to a first ratio; training the target detection model based on the initial training sample set until convergence; selecting difficult samples from the remaining unprocessed training samples in the first category other than the initial training sample set based on the converged target detection model; and adding the selected difficult samples to the initial training sample set to obtain a sample subset. The N sample subsets are combined to obtain the first training sample set.
2. The method according to claim 1, characterized in that, Each of the training samples to be processed includes a ground truth label; the selection of difficult samples from the remaining training samples to be processed in the first category, excluding the initial training sample set, based on the target detection model trained to convergence, includes: The remaining training samples in the first category, excluding the initial training sample set, are input into the target detection model to train until convergence, thereby obtaining the predicted value for each of the remaining training samples. From the remaining training samples to be processed, select M% of the training samples whose predicted values have an overlap of less than a threshold with the true labels; M is a natural number.
3. The method according to claim 2, characterized in that, The step of selecting M% training samples from the remaining training samples to be processed, where the overlap between the predicted value and the ground truth label is less than a threshold, includes: The remaining training samples to be processed are sorted in ascending order of the degree of overlap between the predicted values and the true labels to obtain the overlap ranking result. Select the top M% of the remaining training samples whose overlap is less than a preset threshold from the overlap ranking results.
4. The method according to claim 2, characterized in that, The degree of overlap between the predicted value and the true value label is the crossover union ratio.
5. The method according to claim 1, characterized in that, The classification tree comprises multiple layers; each layer comprises multiple tree nodes; each tree node comprises one or more training samples to be processed; the target attributes of the training samples to be processed in different tree nodes of the same layer are different. The step of dividing the training sample set to be processed into N categories according to the classification tree includes: Select the target layer in the classification tree; the target layer is any layer in the multiple layers of the classification tree; Based on the target attributes of different tree nodes in the target layer, the training sample set to be processed is divided into N categories.
6. The method according to any one of claims 1-5, characterized in that, The attribute classification model is obtained by training a preset initial model based on an attribute training sample set; the attribute training sample set includes multiple attribute training samples; each attribute training sample set includes the training sample to be processed and a label corresponding to the training sample to be processed; the label is the target attribute.
7. A training sample screening device, characterized in that, The device includes: an acquisition module and a processing module; The acquisition module is used to acquire a training sample set to be processed; the training sample set to be processed is used to train the target detection model; the training sample set to be processed includes multiple training samples to be processed; the training samples to be processed include images and ground value labels corresponding to the images, the ground value labels are used to represent the position information of the target objects in the images; The processing module is used to perform hierarchical clustering on the training sample set to be processed using an attribute classification model to construct a classification tree; the attribute classification model is used to identify the target attributes of the training samples to be processed; the target attributes include at least one of the following: normal, occluded, distorted, overexposed, and underexposed; the training sample set to be processed is divided into N categories according to the classification tree; each category includes at least one training sample to be processed; N is an integer greater than 1; a sample subset is selected from each of the N categories to obtain N sample subsets; the sample subsets are obtained by actively learning from difficult samples in a category; the N sample subsets are merged to obtain a first training sample set; for the first category of the N categories, the processing module is specifically used to randomly select an initial training sample set from the first category according to a first ratio; based on the initial training sample set, the target detection model is trained until convergence; based on the target detection model trained to convergence, difficult samples are selected from the remaining training samples to be processed in the first category other than the initial training sample set; the selected difficult samples are added to the initial training sample set to obtain a sample subset.
8. The apparatus according to claim 7, characterized in that, Each of the training samples to be processed includes a ground truth label; the processing module is specifically used to input the remaining training samples to be processed in the first category, excluding the initial training sample set, into the target detection model trained to convergence, and obtain the predicted value of each of the remaining training samples to be processed; and select M% of the training samples to be processed whose predicted value and ground truth label overlap is less than a threshold; M is a natural number. The processing module is specifically used to sort the remaining training samples to be processed according to the order of overlap between the predicted value and the true value label from smallest to largest, and obtain the overlap sorting result; and select the top M% of the remaining training samples to be processed in the overlap sorting result whose overlap is less than a preset threshold. The degree of overlap between the predicted value and the true value label is the intersection-union ratio; The classification tree includes multiple layers; each layer includes multiple tree nodes; each tree node includes one or more training samples to be processed; the target attributes of the training samples to be processed in different tree nodes of the same layer are different; the processing module is specifically used to select the target layer in the classification tree; the target layer is any layer in the multiple layers of the classification tree; according to the target attributes of different tree nodes of the target layer, the training sample set to be processed is divided into N categories; The attribute classification model is obtained by training a preset initial model based on an attribute training sample set; the attribute training sample set includes multiple attribute training samples. Each of the attribute training sample sets includes the training sample to be processed and the label corresponding to the training sample to be processed; the label is the target attribute.
9. An electronic device, characterized in that, The electronic device includes: a processor and a memory; The memory stores instructions that the processor can execute; When the processor is configured to execute the instructions, the electronic device performs the method as described in any one of claims 1-6.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes: computer software instructions; When the computer software instructions are executed in an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-6.