A method and system for downsampling training samples for neural network image classification tasks

Abstract

This invention proposes a method for downsampling training samples for neural network image classification tasks. The method includes the following steps: collecting and acquiring sample data information; training a neural network model based on the acquired sample data information; performing pre-defined operations on the sample data during the inference phase to further obtain downsampled sample data; and completing the sample data classification training task. By using the downsampling method, the diversity of sample distribution is ensured while also considering the gain effect on model training. This effectively alleviates the underfitting problem of the model for a small number of sample categories in image classification tasks with imbalanced sample labels.

Description

Technical Field

[0001] This invention belongs to the technical field of data sampling, specifically relating to a method and system for downsampling training samples for neural network image classification tasks. Background Technology

[0002] During the training of neural network image classification tasks, if the training sample labels are unbalanced, it may result in the model not fitting the smaller number of categories adequately, or even cause the model to treat all samples of the smaller category as training noise.

[0003] Currently, widely used methods can be broadly categorized into three types: those targeting the training data itself, those targeting the model training process, and those targeting the model inference stage. Among these, methods targeting the training data itself have become the mainstream approach due to their ability to incorporate model inference feedback. These methods can be further subdivided into: oversampling or creating new samples from a small number of labeled data, downsampling from multiple categories, and adaptive sampling.

[0004] However, currently widely used data sampling methods often only consider the distribution of data features themselves, and various data augmentations are used in the model training phase but not in the inference phase. As a result, the sample sets obtained through various sampling methods cannot provide a significant gain effect in the model training phase.

[0005] Therefore, it is of great significance to propose a method and system for downsampling training samples for neural network image classification tasks. Summary of the Invention

[0006] To address the problem that existing sample sets obtained through various sampling methods cannot provide significant gain during model training, this invention provides a method and system for downsampling training samples for neural network image classification tasks, thereby resolving the aforementioned technical deficiencies.

[0007] In a first aspect, this invention proposes a method for downsampling training samples for neural network image classification tasks, the method comprising the following steps:

[0008] Collect and acquire sample data information;

[0009] A neural network model is trained based on the acquired sample data information;

[0010] During the inference phase, the sample data undergoes pre-defined processing to further obtain downsampled sample data;

[0011] Complete the sample data classification training task.

[0012] Preferably, it further includes: collecting and acquiring image sample data information of different categories, wherein the sample data of a certain category l is greater than X times the sample data of the other categories, and the total number of samples of category l is N.

[0013] Further preferably, the method also includes: performing the preset operation on each sample i in the N images of category l, specifically including:

[0014] Perform n different data augmentations on sample i to obtain n images, which are then input into the feature layer F of the trained neural network model. M This yields n output results;

[0015] Calculate the relative feature layer F of the nodes corresponding to the labels of n output results in the output layer l. M gradient

[0016] Normalize the n feature gradients and calculate the feature expectation μ of these n feature gradients. i and variance σ i ;

[0017] Calculate the modulus of the characteristic variance to obtain ||σ|| i ||;

[0018] The calculated feature expectations μ = {μ 1 ,μ 2 ,…,μ N The clusters are divided into S classes. For each class, the image samples corresponding to the features with the largest variance modulus are selected, and finally, S images are obtained.

[0019] Further preferred methods include: performing K-means clustering based on the feature expectation μ.

[0020] Secondly, this invention also proposes a downsampling system for training samples in a neural network image classification task, the system comprising:

[0021] The data acquisition module is used to collect and acquire sample data information;

[0022] The model training module is used to train a neural network model based on the acquired sample data information;

[0023] The operation processing module is used to perform preset operation processing on the sample data during the inference stage to further obtain downsampled sample data.

[0024] A further preferred embodiment includes: acquiring image sample data of different categories through an acquisition module, wherein the sample data of a certain category l is greater than X times the sample data of the other categories, and the total number of samples of category l is N.

[0025] Further preferably, it also includes: performing the preset operation processing on each sample u in the N images of category l through the operation processing module, specifically including:

[0026] Perform n different data augmentations on sample i to obtain n images, which are then input into the feature layer F of the trained neural network model. M This yields n output results;

[0027] Calculate the relative feature layer F of the nodes corresponding to the labels of n output results in the output layer l. M gradient

[0028] Normalize the n feature gradients and calculate the feature expectation μ of these n feature gradients. i and variance σ i ;

[0029] Calculate the modulus of the characteristic variance to obtain ||σ|| i ||;

[0030] The calculated feature expectations μ = {μ 1 ,μ 2 ,…,μ N The clusters are divided into S classes. For each class, the image samples corresponding to the features with the largest variance modulus are selected, and finally, S images are obtained.

[0031] Further preferred options include:

[0032] The data augmentation module is used to perform n different data augmentations on sample i to obtain n images, which are then input into the feature layer F of the trained neural network model. M This yields n output results;

[0033] The calculation module is used to calculate the relative position of the nodes corresponding to the labels in the output layer l to the feature layer F for n output results. M gradient Calculate the expected value of the feature μ i and variance σ i Calculate the modulus of the characteristic variance ||σ i ||;

[0034] The clustering module is used to cluster all the calculated feature expectations μ = {μ} using K-means. 1 ,μ 2 ,…,μ N The clusters are divided into S classes. For each class, the image samples corresponding to the features with the largest variance modulus are selected, and finally, S images are obtained.

[0035] Thirdly, embodiments of the present invention provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation of the first aspect.

[0036] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the implementations of the first aspect.

[0037] Compared with the prior art, the beneficial results of the present invention are as follows:

[0038] (1) The technical solution of the present invention proposes a sample downsampling method based on sample uncertainty and distribution diversity, which ensures the distribution diversity among samples while taking into account the gain effect on model training. It can effectively alleviate the underfitting problem of the model to a small number of sample categories in the image classification task with unbalanced sample labels.

[0039] (2) The downsampling method of the present invention has been applied to the "whether the face is masked" task and the "skin color judgment" task in the face attribute judgment model of the masked attribute branch of the face multi-attribute model. After downsampling, the number of images in the dataset is reduced to less than half of the original, reducing the training time. At the same time, the F1 value of the model for the face masking task is increased from 0.919 to 0.937, and the F1-macro index for the skin color judgment task is increased from 0.944 to 0.951. Attached Figure Description

[0040] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments and, together with the description, serve to explain the principles of the invention. Other embodiments and many anticipated advantages of the embodiments will be readily recognized as they become better understood through reference to the following detailed description. Elements in the drawings are not necessarily to scale. The same reference numerals refer to corresponding similar parts.

[0041] Figure 1 This is an exemplary device architecture diagram in which an embodiment of the present invention can be applied;

[0042] Figure 2 This is a flowchart illustrating the method for downsampling training samples in a neural network image classification task according to an embodiment of the present invention.

[0043] Figure 3 This is a flowchart illustrating the sampling method in the neural network image classification task training sample downsampling method according to an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of the structure of a neural network image classification task training sample downsampling system according to an embodiment of the present invention;

[0045] Figure 5 This is a schematic diagram of the structure of a computer device suitable for implementing electronic devices according to embodiments of the present invention. Detailed Implementation

[0046] In the following detailed description, reference is made to the accompanying drawings, which form part of the detailed description and are illustrated by specific illustrative embodiments in which the invention may be practiced. In this regard, directional terms such as “top,” “bottom,” “left,” “right,” “up,” “down,” etc., are used with reference to the orientation of the described figures. Because components of the embodiments can be positioned in several different orientations, directional terms are used for illustrative purposes and are by no means limiting. It should be understood that other embodiments may be utilized or logical changes may be made without departing from the scope of the invention. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the invention is defined by the appended claims.

[0047] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0048] Figure 1 An exemplary system architecture 100 for processing information, or for processing information, to which embodiments of the present invention can be applied, is shown.

[0049] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0050] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0051] Terminal devices 101, 102, and 103 can be various electronic devices with communication functions, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0052] Server 105 can be a server that provides various services, such as a background information processing server that processes verification request information sent by terminal devices 101, 102, and 103. The background information processing server can analyze and process the received verification request information and obtain processing results (such as verification success information used to indicate that the verification request is a valid request).

[0053] It should be noted that the information processing method provided in the embodiments of the present invention is generally executed by server 105, and correspondingly, the device for processing information is generally disposed in server 105. Furthermore, the information sending method provided in the embodiments of the present invention is generally executed by terminal devices 101, 102, and 103, and correspondingly, the device for sending information is generally disposed in terminal devices 101, 102, and 103.

[0054] It should be noted that a server can be either hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software programs or software modules (for example, used to provide distributed services), or as a single software program or multiple software modules; no specific limitations are made here.

[0055] Currently widely used data sampling methods often only consider the distribution of data features themselves, and various data augmentations are used in the model training phase but not in the inference phase. As a result, the sample sets obtained through various sampling methods cannot provide significant gains in the model training phase.

[0056] To address the aforementioned shortcomings, we propose a sample downsampling method based on sample uncertainty and distribution diversity, which ensures the distribution diversity among samples while also taking into account the gain effect on model training.

[0057] Firstly, Figure 2 An embodiment of the present invention discloses a method for downsampling training samples for a neural network image classification task, as shown in the example. Figure 2 As shown, the method includes the following steps:

[0058] S101. Collect and acquire sample data information;

[0059] S102. Train a neural network model based on the acquired sample data information;

[0060] S103. In the inference stage, the sample data is subjected to preset operation processing to further obtain downsampled sample data;

[0061] S104. Complete the sample data classification training task.

[0062] In a preferred embodiment, suppose that in an image classification training task, there is a class l with a much larger number of samples than the other classes, and the total number of samples of class l is N. Given a target downsampling number S, where S << N.

[0063] First, a neural network model M is trained using all sample data D. During the inference phase, for each sample i in the N images of category l, the following operation is performed: Figure 3 As shown:

[0064] S1. Perform n different data augmentations to obtain n images, which are then processed by the model feature layer F. M The output result is obtained;

[0065] S2. Calculate the relative feature layer F of the nodes corresponding to the labels of n output results in the output layer l. M gradient

[0066] S3. Normalize the n feature gradients and calculate the feature expectation μ of these n feature gradients. i and variance σ i ;

[0067] S4. Calculate the modulus of the characteristic variance to obtain ‖σ. i ||;

[0068] S5. Use K-means to estimate the value of all features, μ = {μ} 1 ,μ 2 ,…,μ N The clusters are divided into S classes. For each class, the image samples corresponding to the feature with the largest variance modulus are selected, and finally, S images are obtained.

[0069] The technical solution of this invention can effectively alleviate the problem of underfitting of the model to a small number of sample categories in image classification tasks with imbalanced sample labels.

[0070] As a preferred embodiment, the downsampling method of the present invention has been applied to the "whether it is masked" task and the "skin color judgment" task in the face attribute judgment model of the masked attribute branch of the face multi-attribute model. Since the number of face images belonging to the "no masked" category is much greater than the number of face images belonging to the "masked" category in the "whether it is masked" task, and the number of face images belonging to the "yellow skin color" category is much greater than the number of face images belonging to the "no masked" category in the "skin color judgment" task, downsampling is performed on the "no masked" and "yellow skin color" category images respectively.

[0071] Downsampling reduced the number of images in the dataset to less than half of the original, reducing training time. At the same time, the F1 score of the model for the task of determining whether a face is masked increased from 0.919 to 0.937, and the F1-macro score for the task of determining skin color increased from 0.944 to 0.951.

[0072] Secondly, embodiments of the present invention also disclose a downsampling system for training samples in a neural network image classification task, such as... Figure 4 As shown, the system includes: an acquisition module 41, a model training module 42, an operation processing module 43, a data augmentation module 44, a calculation module 45, and a clustering module 46.

[0073] In one embodiment, a data acquisition module 41 is used to collect and acquire sample data information; a model training module 42 is used to train a neural network model based on the acquired sample data information; an operation processing module 43 is used to perform preset operation processing on the sample data during the inference phase to further obtain downsampled sample data; and a data augmentation module 44 is used to perform n different data augmentations on sample i to obtain n images, which are then input into the feature layer F of the trained neural network model. M This yields n output results; the calculation module 45 is used to calculate the relative feature layer F of the n output results at the node corresponding to the label in the output layer l. M gradient Calculate the expected value of the feature μ i and variance σ i Calculate the modulus of the characteristic variance ||σ i ||; Clustering module 46, used to use K-means to calculate the expected value μ of all features μ={μ 1 ,μ 2 ,…,μ N The clusters are divided into S classes. For each class, the image samples corresponding to the features with the largest variance modulus are selected, and finally, S images are obtained.

[0074] The following is for reference. Figure 5 It illustrates an electronic device suitable for implementing embodiments of the present invention (e.g., Figure 1 The diagram shows the structure of a computer device 500 (a server or terminal device). Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0075] like Figure 5As shown, the computer device 500 includes a central processing unit (CPU) 501 and a graphics processing unit (GPU) 502, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 503 or programs loaded from storage section 509 into random access memory (RAM) 506. Various programs and data required for the operation of device 500 are also stored in RAM 504. The CPU 501, GPU 502, ROM 503, and RAM 504 are interconnected via bus 505. Input / output (I / O) interface 506 is also connected to bus 505.

[0076] The following components are connected to I / O interface 506: an input section 507 including a keyboard, mouse, etc.; an output section 508 including an LCD, speakers, etc.; a storage section 509 including a hard disk, etc.; and a communication section 510 including a network interface card, such as a LAN card or modem. The communication section 510 performs communication processing via a network such as the Internet. A drive 511 may also be connected to I / O interface 506 as needed. A removable medium 512, such as a hard disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 511 as needed so that computer programs read from it can be installed into storage section 509 as needed.

[0077] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 510, and / or installed from removable medium 512. When the computer program is executed by central processing unit (CPU) 501 and graphics processing unit (GPU) 502, it performs the functions defined in the methods of this invention.

[0078] It should be noted that the computer-readable medium described in this invention can be a computer-readable signal medium, a computer-readable medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution apparatus, device, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than a computer-readable medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution device, apparatus, or apparatus. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0079] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0080] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using dedicated hardware-based devices that perform the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0081] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be located in a processor.

[0082] In another aspect, the present invention also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to: collect and acquire sample data information; train a neural network model based on the acquired sample data information; in the inference phase, perform preset operations on the sample data to further obtain downsampled sample data; and complete the sample data classification training task.

[0083] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention is not limited to the specific combination of the above-described technical features, but also includes other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. A method for down-sampling training samples of a neural network picture classification task, characterized in that, The method comprises the following steps: Collect and acquire sample data information; the sample data information includes sample data of a certain category , the total sample number of the category is ; training a neural network model based on the obtained sample data information; During the reasoning phase, regarding categories of Each sample in the image To further obtain downsampled sample data, preset processing steps are performed, including: processing the samples... Perform n different data augmentations to obtain n images, and input them into the feature layer of the trained neural network model. This yields n output results; the relative position of the n output results to the corresponding nodes of the output layer labels in the feature layer is calculated. gradient Normalize the n eigengradations and calculate the eigenvalues ​​of these n eigengradations. and variance Calculate the modulus of the characteristic variance to obtain ; Calculate the expected value of all features Gather as For each class, the image sample corresponding to the feature with the largest variance magnitude is selected, ultimately yielding... Zhang's required images; completing a sample data classification training task.

2. The method of claim 1, wherein, Further comprising: Collect and obtain different categories of picture sample data information, wherein the sample data of a certain category is more than X times the sample data of the remaining categories.

3. The method of claim 1, wherein the method further comprises: Also included: based on feature expectations K-means is performed to cluster.

4. A neural network picture classification task training sample down-sampling system, characterized in that, The system comprises: The acquisition module is configured to acquire sample data information, wherein the sample data information comprises sample data of a certain category , the total number of samples of the category ;​ a model training module, configured to train a neural network model based on the obtained sample data information; The operation processing module is configured to perform a preset operation processing on each sample in the category of the plurality of pictures in the inference stage, and further obtain the down-sampling sample data. The operation processing module includes a data enhancement module, a gradient calculation module, and a clustering screening module.​​ Data augmentation module, used to augment samples Perform n different data augmentations to obtain n images, and input them into the feature layer of the trained neural network model. This yields n output results; The calculation module is used to calculate the relative position of n output results at the corresponding nodes of the output layer labels to the feature layer. gradient Calculate the expected value of features and variance Calculate the modulus of the characteristic variance ; a clustering module, configured to use K-means to cluster all the calculated feature expectations into K classes, and select, for each class, a picture sample corresponding to a feature with the largest variance, and finally obtain K required pictures. ​ 5. The neural network picture classification task training sample down-sampling system of claim 4, wherein, Further comprising: The acquisition module collects and obtains image sample data information of different categories, including a specific category. The sample data of the other categories is X times greater than the sample data of the other categories. 6.An electronic device, comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 3.

7. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of any one of claims 1 to 3.

Images