Method and device for generating difficult sample set, storage medium and electronic device

Abstract

The invention discloses a difficult sample set generation method and device, a storage medium and an electronic device, and relates to the field of computers.The method comprises the steps that multiple first image samples are classified through a classifier set, and first difficult samples in the multiple first image samples are determined, the classifier set comprises a plurality of classifiers of different types, the plurality of classifiers are all used for carrying out fault classification on the plurality of first image samples, and the first difficult samples are used for indicating image samples which fail in classification; the first difficult sample is input into a sample enhancement model to be processed, a plurality of second difficult samples are generated, and the second difficult samples comprise the first difficult sample; generating a difficult sample set according to the plurality of second difficult samples; by adopting the scheme, the problem that the difficult sample set obtained based on a traditional difficult sample mining and generating method lacks diversity in related technologies is solved.

Description

Technical Field

[0001] The present application relates to the field of computers, and more specifically, to a method and device for generating a difficult sample set, a storage medium, and an electronic device. Background Art

[0002] As the world continues to face climate change and energy demand, the development and utilization of renewable energy has become crucial. As important components of renewable energy, wind power and photovoltaic power generation have shown a rapid growth in installed capacity and power generation. However, in actual operation, wind turbines and photovoltaic systems will inevitably encounter various faults, such as wind turbine blade damage and photovoltaic module dust blocking. These faults not only affect power generation efficiency, but may also cause permanent damage to the equipment and increase operation and maintenance costs. Therefore, how to efficiently and accurately diagnose and solve these faults has become an important issue that needs to be solved in the current new energy field.

[0003] Traditional fault diagnosis methods mostly rely on manual inspections and regular maintenance, which is not only inefficient but also difficult to cover all potential fault points. With the rapid development of technologies such as big data and artificial intelligence, computer vision-based methods have shown great potential in the field of fault diagnosis. Computer vision technology mainly collects image data of wind turbines or photovoltaic modules, uses image processing technology to extract key features, and combines machine learning models to automatically identify and warn faults. However, in computer vision-based fault diagnosis methods, when faced with complex and changeable fault scenarios, the "difficult sample" problem is an important challenge. Difficult samples refer to samples that are difficult to be correctly classified or predicted during model training, and they often contain rich fault information. In order to solve this problem, methods such as hard sample mining are proposed. Through the hard sample mining method, the impact of simple examples on model training can be effectively reduced, so that the model can pay more attention to those truly challenging hard examples (i.e., hard samples).

[0004] Traditional difficult sample mining usually relies on the prediction results of a single classifier or manually set rules to identify difficult samples. This method may be limited by the performance of the classifier or the subjectivity of the rules, resulting in inaccurate or incomplete identification of difficult samples. Traditional difficult sample enhancement techniques usually generate new samples through simple transformations (such as rotation, scaling, cropping, etc.). Although these methods can increase the number of samples, the generated samples are often very similar to the original samples, lacking diversity, and it is difficult to improve the classification effect of the model.

[0005] In related technologies, there is a problem that the difficult sample sets obtained based on traditional difficult sample mining and generation methods lack diversity, and no effective solution has been proposed so far.

[0006] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology. Summary of the invention

[0007] The embodiments of the present application provide a method and device for generating a difficult sample set, a storage medium, and an electronic device to at least solve the problem of lack of diversity in the difficult sample set obtained based on the traditional difficult sample mining and generation method in the related art.

[0008] According to one aspect of an embodiment of the present application, a method for generating a difficult sample set is provided, comprising: classifying a plurality of first image samples through a classifier set to determine a first difficult sample from the plurality of first image samples, wherein the classifier set includes a plurality of classifiers of different types, the plurality of classifiers are all used to perform fault classification on the plurality of first image samples, and the first difficult sample is used to indicate an image sample that fails classification; inputting the first difficult sample into a sample enhancement model for processing to generate a plurality of second difficult samples, wherein the plurality of second difficult samples include the first difficult sample; and generating a difficult sample set based on the plurality of second difficult samples.

[0009] In an exemplary embodiment, a plurality of first image samples are classified through a set of classifiers to determine a first difficult sample from the plurality of first image samples, including: preprocessing the plurality of first image samples to obtain a plurality of second image samples; performing fault classification on each second image sample in turn through the plurality of classifiers to obtain a plurality of groups of fault classification results, wherein the plurality of second image samples correspond one-to-one to the plurality of groups of fault classification results, each group of fault classification results includes a plurality of fault classification results, the plurality of fault classification results correspond one-to-one to the plurality of classifiers, and the fault classification results are used to indicate a fault category of the second image sample; and determining the first difficult sample from the plurality of second image samples according to the plurality of groups of fault classification results.

[0010] In an exemplary embodiment, determining the first difficult sample from the multiple second image samples based on the multiple groups of fault classification results includes: determining a final classification result of a third image sample based on the multiple first fault classification results, wherein the multiple second image samples include the third image sample, a group of fault classification results corresponding to the third image sample includes the multiple first fault classification results, and the multiple first fault classification results include the final classification result; counting the proportion of second fault classification results in the multiple first fault classification results, wherein the second fault classification result is different from the final classification result; and determining the third image sample as the first difficult sample when it is determined that the proportion of the second fault classification result is greater than a preset threshold.

[0011] In an exemplary embodiment, the first difficult sample is input into the sample enhancement model for processing, and before generating multiple second difficult samples, the method also includes: a training step: training the first discriminator according to the first generator to obtain the second discriminator, and training the first generator according to the second discriminator and the classification discriminator to obtain the second generator, wherein the first generator and the second generator are used to generate the second difficult sample, the first discriminator and the second discriminator are used to evaluate whether the second difficult sample is available, the classification discriminator is obtained based on the classifier set, and the classification discriminator is used to determine the classification probability distribution of the sample, and the classification probability distribution is used to indicate the probability that the sample belongs to different fault categories; the training step is executed in a loop for a preset number of times, and the sample enhancement model is generated according to the second discriminator and the second generator finally obtained.

[0012] In an exemplary embodiment, training the first discriminator according to the first generator to obtain the second discriminator includes: inputting a plurality of real samples into the first generator to generate a plurality of generated samples, wherein the plurality of real samples are difficult samples; and Calculate the first loss of the first discriminator, where L D is the first loss, D(x) is the first discrimination result of the first discriminator on the multiple real samples, G(z) is the multiple generated samples, D(G(z)) is the second discrimination result of the first discriminator on the multiple generated samples, E x～pdata(x) is the distribution probability of each real sample, The distribution probability of each generated sample is adjusted according to the first loss to obtain the second discriminator.

[0013] In an exemplary embodiment, the first generator is trained according to the second discriminator and the classification discriminator to obtain the second generator, including: calculating the third discrimination results of multiple real samples by the second discriminator, and calculating the fourth discrimination results of multiple generated samples by the second discriminator, wherein the multiple generated samples are generated by the first generator for the multiple real samples; calculating the first classification probability distribution of the multiple real samples by the classification discriminator, and calculating the second classification probability distribution of the multiple generated samples by the classification discriminator; by formula Calculate the second loss of the first generator, where L G For the second loss, is the distribution probability of each generated sample, G(z) is the multiple generated samples, D(G(z)) is the fourth discrimination result, L dist(C(G(z)), C(x ref ) distribution difference loss, used to indicate the difference between the second classification probability distribution and the first classification probability distribution of the target real sample, the multiple real samples include the target real sample, x ref is the target true sample, α is the weight of the distribution difference loss; the parameters of the first generator are adjusted according to the second loss to obtain the second generator.

[0014] According to another aspect of an embodiment of the present application, a device for generating a difficult sample set is also provided, including: a processing module, used to classify a plurality of first image samples through a classifier set, and determine a first difficult sample among the plurality of first image samples, wherein the classifier set includes a plurality of classifiers of different types, and the plurality of classifiers are all used to perform fault classification on the plurality of first image samples, and the first difficult sample is used to indicate an image sample that fails classification; a first generating module, used to input the first difficult sample into a sample enhancement model for processing, and generate a plurality of second difficult samples, wherein the plurality of second difficult samples include the first difficult sample; and a second generating module, used to generate a difficult sample set based on the plurality of second difficult samples.

[0015] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above method for generating a difficult sample set when running.

[0016] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method for generating the difficult sample set through the computer program.

[0017] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps of the method described in each embodiment of the present application are implemented.

[0018] Through the present application, multiple first image samples are first classified by multiple classifiers to determine the first difficult samples among the multiple first image samples, wherein the multiple classifiers are of different types and are all used to perform fault classification on the multiple first image samples, and the first difficult samples are used to indicate image samples that fail classification; the first difficult samples are input into a sample enhancement model for processing to generate multiple second difficult samples; finally, a difficult sample set is generated based on the multiple second difficult samples; by adopting the above scheme, multiple classifiers are used for comprehensive evaluation during difficult sample mining to obtain accurate difficult samples, and then more realistic difficult samples are generated through an improved GAN model (i.e., sample enhancement model) to expand the difficult sample set; thereby solving the problem of lack of diversity in the difficult sample set obtained based on traditional difficult sample mining and generation methods in the related art. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 It is a hardware structure block diagram of a computer terminal for a method for generating a difficult sample set according to an embodiment of the present application;

[0022] Figure 2 is a flowchart of a method for generating a difficult sample set according to an embodiment of the present application;

[0023] Figure 3 is a schematic diagram of a training process of a sample enhancement model according to an embodiment of the present application;

[0024] Figure 4 is a flowchart of a method for generating a difficult sample data set according to an embodiment of the present application;

[0025] Figure 5 It is a structural block diagram of a device for generating a difficult sample set according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] The method embodiments provided in the embodiments of the present application can be executed in a computer terminal or a similar computing device. Taking running on a computer terminal as an example, Figure 1 is a hardware structure block diagram of a computer terminal for a method for generating a difficult sample set according to an embodiment of the present application. Figure 1 As shown, the computer terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor (Central Processing Unit, MCU) or a programmable logic device (Field Programmable Gate Array, FPGA)) and a memory 104 for storing data, wherein the above-mentioned computer terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0029] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for generating a difficult sample set in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] A wireless network provided by a communication provider of a computer terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0031] In this embodiment, a method for generating a difficult sample set is provided. Figure 2 is a flow chart of a method for generating a difficult sample set according to an embodiment of the present application, such as Figure 2 As shown, the process includes the following steps S202-S206:

[0032] Step S202: performing classification processing on the plurality of first image samples through a classifier set, and determining a first difficult sample among the plurality of first image samples, wherein the classifier set includes a plurality of classifiers of different types, and the plurality of classifiers are all used to perform fault classification on the plurality of first image samples, and the first difficult sample is used to indicate an image sample that fails classification;

[0033] Step S204: inputting the first difficult sample into a sample enhancement model for processing to generate a plurality of second difficult samples, wherein the plurality of second difficult samples include the first difficult sample;

[0034] Step S206: generating a difficult sample set according to the plurality of second difficult samples.

[0035] Through the above steps, multiple first image samples are first classified by multiple classifiers to determine the first difficult samples among the multiple first image samples, wherein the multiple classifiers are of different types and are all used to perform fault classification on the multiple first image samples, and the first difficult samples are used to indicate image samples that fail classification; the first difficult samples are input into the sample enhancement model for processing to generate multiple second difficult samples; finally, a difficult sample set is generated based on the multiple second difficult samples; using the above scheme, multiple classifiers are used for comprehensive evaluation during difficult sample mining to obtain accurate difficult samples, and then more realistic difficult samples are generated through the improved GAN model (i.e., the sample enhancement model) to expand the difficult sample set; thereby solving the problem of lack of diversity in the difficult sample set obtained based on the traditional difficult sample mining and generation method in the related art.

[0036] In an exemplary embodiment, a plurality of first image samples are classified through a set of classifiers to determine a first difficult sample from the plurality of first image samples, including: preprocessing the plurality of first image samples to obtain a plurality of second image samples; performing fault classification on each second image sample in turn through the plurality of classifiers to obtain a plurality of groups of fault classification results, wherein the plurality of second image samples correspond one-to-one to the plurality of groups of fault classification results, each group of fault classification results includes a plurality of fault classification results, the plurality of fault classification results correspond one-to-one to the plurality of classifiers, and the fault classification results are used to indicate a fault category of the second image sample; and determining the first difficult sample from the plurality of second image samples according to the plurality of groups of fault classification results.

[0037] First, the first image sample needs to be preprocessed to obtain multiple second image samples. The preprocessing process includes data standardization and noise filtering. Data standardization: standardize the input image data, including adjusting the image size, normalizing pixel values, etc. to ensure the consistency of the input data; noise filtering: use image denoising technology (such as Gaussian filtering, median filtering) to reduce image noise and improve the accuracy of the classifier.

[0038] The second image sample is predicted by n trained classifiers in turn to obtain multiple groups of fault classification results. Finally, the first difficult sample is determined based on the multiple groups of fault classification results.

[0039] It should be noted that the training process of these n classifiers includes: selecting n different types of classifiers (such as deep convolutional neural network CNN, support vector machine SVM, random forest, etc.) and training them separately. These classifiers have different learning capabilities and feature extraction capabilities, so as to evaluate the difficulty of samples from multiple perspectives.

[0040] These n classifiers are trained separately until convergence or the preset training rounds are reached, and the performance of each classifier is evaluated on the validation set to ensure that they all have a certain classification ability.

[0041] Based on the above steps, the first difficult sample is determined from the multiple second image samples according to the multiple groups of fault classification results, including: determining the final classification result of the third image sample according to the multiple first fault classification results, wherein the multiple second image samples include the third image sample, and a group of fault classification results corresponding to the third image sample includes the multiple first fault classification results, and the multiple first fault classification results include the final classification result; counting the proportion of the second fault classification results in the multiple first fault classification results, wherein the second fault classification result is different from the final classification result; and when it is determined that the proportion of the second fault classification result is greater than a preset threshold, determining the third image sample as the first difficult sample.

[0042] The final classification result of each third image sample is determined by majority voting. Specifically, the number of times each fault category is predicted is counted, and the fault category with the largest number of predictions is selected as the final classification result of the third image sample. For example, suppose there are 5 classifiers (n=5), respectively labeled as classifiers 1, 2, 3, 4, and 5, which have been trained for a certain three-classification task (fault A, fault B, and fault C). An image sample X to be classified is input into these 5 classifiers after preprocessing. The output results of classifiers 1-5 are faults A, B, A, A, and C, respectively. According to the majority voting method, the sample X is finally classified as fault A.

[0043] Then calculate the number k of classifiers that are inconsistent with the final classification result. For each sample, traverse the fault classification results of its n classifiers and count the number of fault classification results that are different from the final category; set a threshold (for example, n / 5, that is, the above-mentioned preset threshold), and when k is greater than or equal to the threshold, the sample is regarded as a difficult sample, that is, the above-mentioned first difficult sample, and is added to the difficult sample set.

[0044] Optionally, before the first difficult sample is input into the sample enhancement model for processing and multiple second difficult samples are generated, the method also includes: a training step: training the first discriminator according to the first generator to obtain the second discriminator, and training the first generator according to the second discriminator and the classification discriminator to obtain the second generator, wherein the first generator and the second generator are used to generate the second difficult sample, the first discriminator and the second discriminator are used to evaluate whether the second difficult sample is available, the classification discriminator is obtained based on the classifier set, and the classification discriminator is used to determine the classification probability distribution of the sample, and the classification probability distribution is used to indicate the probability that the sample belongs to different fault categories; the training step is executed in a loop for a preset number of times, and the sample enhancement model is generated according to the second discriminator and the second generator finally obtained.

[0045] The sample enhancement model of the present application has a basic architecture of a GAN structure, including a generator G and a discriminator D. The generator is responsible for generating image samples from random noise, while the discriminator is responsible for distinguishing the generated images from the real images. In the model training process, the embodiment of the present application adopts a method of alternating training of the discriminator and the generator. In each iterative training process, the parameters of the generator G are first fixed, the discriminator D is trained, and then the parameters of the discriminator D are fixed, and the generator G is trained.

[0046] Optionally, a classification discriminator is also integrated in the sample enhancement model of the present application to evaluate the classification quality of the generated samples, wherein the classification discriminator can adopt one of multiple classifiers with better performance and fix its parameters to remain unchanged during the GAN training process. In this way, the trained classifier can be used to evaluate the classification quality of the generated samples while avoiding affecting the performance of the classifier during the GAN training process.

[0047] Optionally, training the first discriminator according to the first generator to obtain the second discriminator includes: inputting a plurality of real samples into the first generator to generate a plurality of generated samples, wherein the plurality of real samples are difficult samples; and obtaining the second discriminator by formula Calculate the first loss of the first discriminator, where L D is the first loss, D(x) is the first discrimination result of the first discriminator on the multiple real samples, G(z) is the multiple generated samples, D(G(z)) is the second discrimination result of the first discriminator on the multiple generated samples, E x～pdata(x) is the distribution probability of each real sample, The distribution probability of each generated sample is adjusted according to the first loss to obtain the second discriminator.

[0048] The loss of the discriminator D, for real images, D should output as high a value as possible (close to 1), indicating that these images are real, and for images generated by the generator G, D should output as low a value as possible (close to 0), indicating that these images are fake. Use binary cross entropy loss to calculate: L D is the first loss, D(x) is the first discrimination result of the first discriminator on the multiple real samples, G(z) is the multiple generated samples, D(G(z)) is the second discrimination result of the first discriminator on the generated samples, E x～pdata(x) is the distribution probability of each real sample, The distribution probability of each generated sample is calculated, and finally the parameters of the first discriminator are adjusted according to the calculated first loss to complete the training of the discriminator in this iterative training.

[0049] Further, the first generator is trained according to the second discriminator and the classification discriminator to obtain the second generator, including: calculating the third discrimination results of multiple real samples by the second discriminator, and calculating the fourth discrimination results of multiple generated samples by the second discriminator, wherein the multiple generated samples are generated by the first generator for the multiple real samples; calculating the first classification probability distribution of the multiple real samples by the classification discriminator, and calculating the second classification probability distribution of the multiple generated samples by the classification discriminator; by formula Calculate the second loss of the first generator, where L G For the second loss, is the distribution probability of each generated sample, G(z) is the multiple generated samples, D(G(z)) is the fourth discrimination result, L dist (C(G(z)), C(x ref ) is a distribution difference loss, used to indicate the difference between the second classification probability distribution and the first classification probability distribution of the target real sample, wherein the multiple real samples include the target real sample, x ref is the target true sample, α is the weight of the distribution difference loss; the parameters of the first generator are adjusted according to the second loss to obtain the second generator.

[0050] Loss of generator G: The goal of the original generator G is to generate images that deceive the discriminator D as much as possible, so its loss function corresponds to the generated image discrimination loss of D, but the goal is to minimize it. In this embodiment, a classification discriminator is introduced to calculate the distribution difference loss, which is used to measure the classification probability distribution of the generated sample G(z) and a certain reference real sample x ref The difference between the classification probability distributions of the target real samples mentioned above. Among them, Ldist (C(G(z)), C(x ref ) is the distribution difference loss based on L2 distance, and α is the weight of the distribution difference loss; finally, the parameters of the first generator are adjusted according to the calculated second loss to complete this iterative training.

[0051] Based on the above model training process, a sample enhancement model is finally obtained. Optionally, the present application embodiment also provides an optional training method for the sample enhancement model, such as Figure 3 As shown, the training method of the sample enhancement model includes the following process: random noise is input into the generator G to obtain generated samples, the generated samples and real samples are classified by the classification discriminator C, and the loss L of the generator G is calculated based on the obtained classification probability distribution G ; and input the generated samples and real samples into the discriminator D for discrimination to calculate the loss L of the discriminator D D , using L G and L D Complete the training of the classifier and discriminator.

[0052] Through the above embodiment, multiple classifiers are used to predict the same sample, and difficult samples are screened out by quantifying the inconsistency between these prediction results. This method overcomes the bias that may exist in a single classifier and improves the accuracy and robustness of difficult sample identification. In addition, based on the traditional GAN ​​model, a classification discriminator C is introduced to evaluate the classification quality of generated samples. At the same time, by defining the distribution difference loss, the generator is guided to generate samples with similar classification distribution to the real samples, thereby alleviating the mode collapse problem in GAN training and improving the quality and diversity of generated samples. During the GAN training process, the parameters of the classification discriminator C are fixed to avoid the potential impact of GAN training on the classifier performance and ensure the accuracy and stability of the classification discriminator's evaluation of generated samples.

[0053] In an optional embodiment, the present application embodiment provides an optional method for generating a difficult sample data set, the process is as follows: Figure 4 As shown, the following steps are included:

[0054] 1. Data preparation and preprocessing.

[0055] 1.1. Data standardization: Standardize the input image data, including adjusting the image size, normalizing the pixel values, etc., to ensure the consistency of the input data;

[0056] 1.2. Noise filtering: Use image denoising techniques (such as Gaussian filtering and median filtering) to reduce image noise and improve the accuracy of the classifier.

[0057] 2. Difficult sample mining: After preprocessing the input data, it is first input into n trained classification models respectively, and the difficult samples are screened according to the prediction results of multiple classifiers. The evaluation criteria in this stage are mainly based on the inconsistency of the prediction results of multiple classifiers. When there are significant differences in the prediction results of the classifiers for the same sample, the sample is likely to contain complex features or be near the classification boundary, and therefore is more likely to become a difficult sample. By quantifying this inconsistency, we can effectively screen out samples that need special attention. First, the final category of the sample (that is, the final classification result mentioned above) is determined by the majority voting method, and then the number of classifiers that are inconsistent with the final category is counted (assuming it is k). This method can intuitively reflect how many classifiers disagree with the opinions of the majority of classifiers. The more the number, the higher the inconsistency between the prediction results of different classifiers. Set a certain threshold (that is, the preset threshold mentioned above), and when k is greater than or equal to the threshold, the sample is regarded as a difficult sample. This method is based on the prediction results of multiple classifiers, avoiding the bias of a single classifier. At the same time, it has a higher recognition accuracy for samples located at the classification boundary or with complex features.

[0058] 3. Difficult sample generation: By improving GAN (generative adversarial network), the screened difficult samples are enhanced to generate more realistic and challenging new samples to enrich the training data set and improve model performance. The specific improvements are as follows: A classification discriminator C is combined with the original GAN ​​model. The discriminator is a high-performance discriminator in the classifier set (its parameters remain unchanged during the GAN training process) to output the probability that the generated samples belong to different fault categories. In order to encourage the generator to generate samples with similar distribution to the original samples, an additional loss term L is introduced. dist , this loss term measures the difference between the classification probability distribution of the generated samples and the classification probability distribution of the real samples based on the L2 distance. By introducing the distribution difference loss, the generator can be guided to learn the distribution characteristics of the real data more accurately, which helps to generate more realistic and challenging new samples, thereby improving the overall quality of the generated samples and alleviating the mode collapse problem in the GAN training process to a certain extent.

[0059] 4. Add the original data set: Add the difficult samples determined in step 2 and the newly generated difficult samples in step 3 to the difficult sample data set.

[0060] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0061] In this embodiment, a device for generating a difficult sample set is also provided, which is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0062] Figure 5 is a structural block diagram of a device for generating a difficult sample set according to an embodiment of the present application, the device comprising:

[0063] A processing module 52 is used to classify the plurality of first image samples through a classifier set to determine a first difficult sample among the plurality of first image samples, wherein the classifier set includes a plurality of classifiers of different types, the plurality of classifiers are all used to perform fault classification on the plurality of first image samples, and the first difficult sample is used to indicate an image sample that fails classification;

[0064] A first generating module 54 is used to input the first difficult sample into a sample enhancement model for processing to generate a plurality of second difficult samples, wherein the plurality of second difficult samples include the first difficult sample;

[0065] The second generating module 56 is configured to generate a difficult sample set according to the plurality of second difficult samples.

[0066] Through the above-mentioned device, multiple first image samples are first classified by multiple classifiers to determine the first difficult samples among the multiple first image samples, wherein the multiple classifiers are of different types and are all used to perform fault classification on the multiple first image samples, and the first difficult samples are used to indicate image samples that fail classification; the first difficult samples are input into the sample enhancement model for processing to generate multiple second difficult samples; finally, a difficult sample set is generated based on the multiple second difficult samples; by adopting the above-mentioned scheme, multiple classifiers are used for comprehensive evaluation during difficult sample mining to obtain accurate difficult samples, and then more realistic difficult samples are generated through the improved GAN model (i.e., the sample enhancement model) to expand the difficult sample set; thereby solving the problem of lack of diversity in the difficult sample set obtained based on the traditional difficult sample mining and generation method in the related technology.

[0067] Optionally, the above-mentioned processing module 52 is also used to preprocess the multiple first image samples to obtain multiple second image samples; perform fault classification on each second image sample in turn through the multiple classifiers to obtain multiple groups of fault classification results, wherein the multiple second image samples correspond one-to-one to the multiple groups of fault classification results, each group of fault classification results includes multiple fault classification results, the multiple fault classification results correspond one-to-one to the multiple classifiers, and the fault classification results are used to indicate the fault category of the second image sample; and determine the first difficult sample from the multiple second image samples according to the multiple groups of fault classification results.

[0068] First, the first image sample needs to be preprocessed to obtain multiple second image samples. The preprocessing process includes data standardization and noise filtering. Data standardization: standardize the input image data, including adjusting the image size, normalizing pixel values, etc. to ensure the consistency of the input data; noise filtering: use image denoising technology (such as Gaussian filtering, median filtering) to reduce image noise and improve the accuracy of the classifier.

[0069] The second image sample is predicted by n trained classifiers in turn to obtain multiple groups of fault classification results. Finally, the first difficult sample is determined based on the multiple groups of fault classification results.

[0070] It should be noted that the training process of these n classifiers includes: selecting n different types of classifiers (such as deep convolutional neural network CNN, support vector machine SVM, random forest, etc.) and training them separately. These classifiers have different learning capabilities and feature extraction capabilities, so as to evaluate the difficulty of samples from multiple perspectives.

[0071] These n classifiers are trained separately until convergence or the preset training rounds are reached, and the performance of each classifier is evaluated on the validation set to ensure that they all have a certain classification ability.

[0072] Optionally, the above-mentioned processing module 52 is also used to determine the final classification result of the third image sample based on multiple first fault classification results, wherein the multiple second image samples include the third image sample, a group of fault classification results corresponding to the third image sample includes the multiple first fault classification results, and the multiple first fault classification results include the final classification result; count the proportion of the second fault classification results in the multiple first fault classification results, wherein the second fault classification result is different from the final classification result; and when it is determined that the proportion of the second fault classification result is greater than a preset threshold, determine the third image sample as the first difficult sample.

[0073] The final classification result of each third image sample is determined by majority voting. Specifically, the number of times each fault category is predicted is counted, and the fault category with the largest number of predictions is selected as the final classification result of the third image sample. For example, suppose there are 5 classifiers (n=5), respectively labeled as classifiers 1, 2, 3, 4, and 5, which have been trained for a certain three-classification task (fault A, fault B, and fault C). An image sample X to be classified is input into these 5 classifiers after preprocessing. The output results of classifiers 1-5 are faults A, B, A, A, and C, respectively. According to the majority voting method, the sample X is finally classified as fault A.

[0074] Then calculate the number k of classifiers that are inconsistent with the final classification result. For each sample, traverse the fault classification results of its n classifiers and count the number of fault classification results that are different from the final category; set a threshold (for example, n / 5, that is, the above-mentioned preset threshold), and when k is greater than or equal to the threshold, the sample is regarded as a difficult sample, that is, the above-mentioned first difficult sample, and is added to the difficult sample set.

[0075] Optionally, the first generation module 54 is further used to perform a training step: training the first discriminator according to the first generator to obtain a second discriminator, and training the first generator according to the second discriminator and the classification discriminator to obtain a second generator, wherein the first generator and the second generator are used to generate the second difficult sample, the first discriminator and the second discriminator are used to evaluate whether the second difficult sample is available, the classification discriminator is obtained based on the classifier set, the classification discriminator is used to determine the classification probability distribution of the sample, and the classification probability distribution is used to indicate the probability that the sample belongs to different fault categories; the training step is performed repeatedly for a preset number of times, and the sample enhancement model is generated according to the second discriminator and the second generator finally obtained. .

[0076] The sample enhancement model of the present application has a basic architecture of a GAN structure, including a generator G and a discriminator D. The generator is responsible for generating image samples from random noise, while the discriminator is responsible for distinguishing the generated images from the real images. In the model training process, the embodiment of the present application adopts a method of alternating training of the discriminator and the generator. In each iterative training process, the parameters of the generator G are first fixed, the discriminator D is trained, and then the parameters of the discriminator D are fixed, and the generator G is trained.

[0077] Optionally, a classification discriminator is also integrated in the sample enhancement model of the present application to evaluate the classification quality of the generated samples, wherein the classification discriminator can adopt one of multiple classifiers with better performance and fix its parameters to remain unchanged during the GAN training process. In this way, the trained classifier can be used to evaluate the classification quality of the generated samples while avoiding affecting the performance of the classifier during the GAN training process.

[0078] Optionally, the first generating module 54 is further configured to input a plurality of real samples into the first generator to generate a plurality of generated samples, wherein the plurality of real samples are difficult samples; Calculate the first loss of the first discriminator, where L D is the first loss, D(x) is the first discrimination result of the first discriminator on the multiple real samples, G(z) is the multiple generated samples, D(G(z)) is the second discrimination result of the first discriminator on the multiple generated samples, E x～pdata(x) is the distribution probability of each real sample, The distribution probability of each generated sample is adjusted according to the first loss to obtain the second discriminator.

[0079] The loss of the discriminator D, for real images, D should output as high a value as possible (close to 1), indicating that these images are real, and for images generated by the generator G, D should output as low a value as possible (close to 0), indicating that these images are fake. Use binary cross entropy loss to calculate: L D is the first loss, D(x) is the first discrimination result of the first discriminator on the multiple real samples, G(z) is the multiple generated samples, D(G(z)) is the second discrimination result of the first discriminator on the generated samples, E x～pdata(x) is the distribution probability of each real sample, The distribution probability of each generated sample is calculated, and finally the parameters of the first discriminator are adjusted according to the calculated first loss to complete the training of the discriminator in this iterative training.

[0080] Optionally, the first generation module 54 is further configured to calculate a third discrimination result of a plurality of real samples through the second discriminator, and to calculate a fourth discrimination result of a plurality of generated samples through the second discriminator, wherein the plurality of generated samples are generated by the first generator for the plurality of real samples; calculate a first classification probability distribution of the plurality of real samples through the classification discriminator, and calculate a second classification probability distribution of the plurality of generated samples through the classification discriminator; and calculate the first classification probability distribution of the plurality of generated samples through the classification discriminator by formula Calculate the second loss of the first generator, where L G For the second loss, is the distribution probability of each generated sample, G(z) is the multiple generated samples, D(G(z)) is the fourth discrimination result, L dist (C(G(z)), C(x ref ) is a distribution difference loss, used to indicate the difference between the second classification probability distribution and the first classification probability distribution of the target real sample, wherein the multiple real samples include the target real sample, x ref is the target true sample, α is the weight of the distribution difference loss; the parameters of the first generator are adjusted according to the second loss to obtain the second generator.

[0081] Loss of generator G: The goal of the original generator G is to generate images that deceive the discriminator D as much as possible, so its loss function corresponds to the generated image discrimination loss of D, but the goal is to minimize it. In this embodiment, a classification discriminator is introduced to calculate the distribution difference loss, which is used to measure the classification probability distribution of the generated sample G(z) and a certain reference real sample x ref The difference between the classification probability distributions of the target real samples mentioned above. Among them, L dist (C(G(z)), C(x ref ) is the distribution difference loss based on L2 distance, and α is the weight of the distribution difference loss; finally, the parameters of the first generator are adjusted according to the calculated second loss to complete this iterative training.

[0082] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0083] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0084] S1, classifying a plurality of first image samples through a classifier set to determine a first difficult sample among the plurality of first image samples, wherein the classifier set includes a plurality of classifiers of different types, the plurality of classifiers are all used to perform fault classification on the plurality of first image samples, and the first difficult sample is used to indicate an image sample that fails classification;

[0085] S2, inputting the first difficult sample into a sample enhancement model for processing to generate a plurality of second difficult samples, wherein the plurality of second difficult samples include the first difficult sample;

[0086] S3: Generate a difficult sample set according to the multiple second difficult samples.

[0087] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0088] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0089] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0090] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0091] S1, classifying a plurality of first image samples through a classifier set to determine a first difficult sample among the plurality of first image samples, wherein the classifier set includes a plurality of classifiers of different types, the plurality of classifiers are all used to perform fault classification on the plurality of first image samples, and the first difficult sample is used to indicate an image sample that fails classification;

[0092] S2, inputting the first difficult sample into a sample enhancement model for processing to generate a plurality of second difficult samples, wherein the plurality of second difficult samples include the first difficult sample;

[0093] S3: Generate a difficult sample set according to the multiple second difficult samples.

[0094] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0095] An embodiment of the present application further provides a computer program product, comprising a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program product, and when the computer program is executed by a processor, the steps of the method described in each embodiment of the present application are implemented.

[0096] Optionally, in this embodiment, the above computer program may be configured to implement the following steps when executed by a processor:

[0097] S1, classifying a plurality of first image samples through a classifier set to determine a first difficult sample among the plurality of first image samples, wherein the classifier set includes a plurality of classifiers of different types, the plurality of classifiers are all used to perform fault classification on the plurality of first image samples, and the first difficult sample is used to indicate an image sample that fails classification;

[0098] S2, inputting the first difficult sample into a sample enhancement model for processing to generate a plurality of second difficult samples, wherein the plurality of second difficult samples include the first difficult sample;

[0099] S3: Generate a difficult sample set according to the multiple second difficult samples.

[0100] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0101] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0102] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for generating a difficult sample set, characterized in that: include: Classify the plurality of first image samples through a classifier set to determine a first difficult sample among the plurality of first image samples, wherein the classifier set includes a plurality of classifiers of different types, the plurality of classifiers are all used to perform fault classification on the plurality of first image samples, and the first difficult sample is used to indicate an image sample that fails classification; Inputting the first difficult sample into a sample enhancement model for processing to generate a plurality of second difficult samples, wherein the plurality of second difficult samples include the first difficult sample; A difficult sample set is generated according to the plurality of second difficult samples.

2. The method according to claim 1, characterized in that: Classifying the plurality of first image samples through a classifier set to determine a first difficult sample from the plurality of first image samples includes: Preprocessing the plurality of first image samples to obtain a plurality of second image samples; Performing fault classification on each second image sample in turn through the multiple classifiers to obtain multiple groups of fault classification results, wherein the multiple second image samples correspond one-to-one to the multiple groups of fault classification results, each group of fault classification results includes multiple fault classification results, the multiple fault classification results correspond one-to-one to the multiple classifiers, and the fault classification results are used to indicate the fault category of the second image sample; The first difficult sample is determined from the plurality of second image samples according to the plurality of groups of fault classification results.

3. The method according to claim 2, characterized in that Determining the first difficult sample from the plurality of second image samples according to the plurality of groups of fault classification results comprises: Determining a final classification result of a third image sample according to a plurality of first fault classification results, wherein the plurality of second image samples include the third image sample, a group of fault classification results corresponding to the third image sample includes the plurality of first fault classification results, and the plurality of first fault classification results include the final classification result; Counting the proportion of the second fault classification results in the multiple first fault classification results, wherein the second fault classification result is different from the final classification result; When it is determined that the proportion of the second fault classification results is greater than a preset threshold, the third image sample is determined as the first difficult sample.

4. The method according to claim 1, characterized in that: Before inputting the first difficult sample into the sample enhancement model for processing to generate a plurality of second difficult samples, the method further includes: Training step: training the first discriminator according to the first generator to obtain a second discriminator, and training the first generator according to the second discriminator and a classification discriminator to obtain a second generator, wherein the first generator and the second generator are used to generate the second difficult sample, the first discriminator and the second discriminator are used to evaluate whether the second difficult sample is usable, the classification discriminator is obtained based on the classifier set, the classification discriminator is used to determine the classification probability distribution of the sample, and the classification probability distribution is used to indicate the probability that the sample belongs to different fault categories; The training step is executed repeatedly for a preset number of times, and the sample enhancement model is generated according to the second discriminator and the second generator finally obtained.

5. The method according to claim 4, characterized in that The first discriminator is trained according to the first generator to obtain a second discriminator, including: Inputting a plurality of real samples into the first generator to generate a plurality of generated samples, wherein the plurality of real samples are difficult samples; By formula Calculate the first loss of the first discriminator, where L D is the first loss, D(x) is the first discrimination result of the first discriminator on the multiple real samples, G(z) is the multiple generated samples, D(G(z)) is the second discrimination result of the first discriminator on the multiple generated samples, E x～pdata(x) is the distribution probability of each real sample, The distribution probability of each generated sample; The parameters of the first discriminator are adjusted according to the first loss to obtain the second discriminator.

6. The method according to claim 5, characterized in that The first generator is trained according to the second discriminator and the classification discriminator to obtain a second generator, comprising: Calculating a third discrimination result of a plurality of real samples by the second discriminator, and calculating a fourth discrimination result of a plurality of generated samples by the second discriminator, wherein the plurality of generated samples are generated by the plurality of real samples by the first generator; Calculating a first classification probability distribution of the plurality of real samples by the classification discriminator, and calculating a second classification probability distribution of the plurality of generated samples by the classification discriminator; By formula Calculate the second loss of the first generator, where L G For the second loss, is the distribution probability of each generated sample, G(z) is the multiple generated samples, D(G(z)) is the fourth discrimination result, L dist (C(G(z)),C(x ref ) is a distribution difference loss, used to indicate the difference between the second classification probability distribution and the first classification probability distribution of the target real sample, wherein the multiple real samples include the target real sample, x ref is the target true sample, α is the weight of the distribution difference loss; The parameters of the first generator are adjusted according to the second loss to obtain the second generator.

7. A device for generating a difficult sample set, characterized in that: include: a processing module, configured to classify the plurality of first image samples through a classifier set, and determine a first difficult sample among the plurality of first image samples, wherein the classifier set includes a plurality of classifiers of different types, the plurality of classifiers are all configured to perform fault classification on the plurality of first image samples, and the first difficult sample is configured to indicate an image sample that fails classification; a first generating module, configured to input the first difficult sample into a sample enhancement model for processing, and generate a plurality of second difficult samples, wherein the plurality of second difficult samples include the first difficult sample; The second generating module is configured to generate a difficult sample set according to the plurality of second difficult samples.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 6 when executed.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 6 through the computer program.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images