Image data enhancement method, device, medium and equipment

Abstract

The present invention discloses an image data enhancement method, apparatus, medium and equipment, including training a deep learning network according to an image training set to obtain a soft label generator, wherein the image training set includes a set of image samples of K categories; obtaining a soft label and category of a first image sample, wherein the soft label represents the category association between the first image sample and the K categories; obtaining the category representativeness of image samples in an image sample set of K-1 categories according to the soft label generator, wherein the K-1 categories do not include the category of the first image sample; selecting a second image sample from the image sample set of K-1 categories according to the soft label of the first image sample and the category representativeness of the image sample; and fusing the first image sample and the second image sample to obtain a target image sample. The present invention relates to the field of computer vision, and by generating new image samples with controllable difficulty, the difficulty of training a classification network model can be adjusted to improve the generalization ability of the model.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image data enhancement method, device, medium and equipment. Background Art

[0002] With the development of artificial intelligence, deep learning models have been widely applied across industries and are playing a vital role in various scenarios. Deep learning network models require a large amount of sample data for training. However, insufficient sample data can lead to overfitting during model training. Therefore, data augmentation methods for training data have been proposed. These methods generate more sample data by performing operations such as geometric and color transformations on existing sample data.

[0003] In the field of computer vision, for image classification networks and image training samples, mixed sample data enhancement methods such as Mixup, Cutout, and CutMix have been proposed. The image after mixed samples can be Figure 1 As shown in Figure 2 . All three methods improve the generalization of deep learning classification networks by generating difficult examples. However, these methods all randomly extract two samples from the training set and fuse them to synthesize new difficult examples. This lacks consideration of the correlation between categories, making it difficult to measure and control the difficulty of training the network model with the synthesized samples. For example, using CutMix to fuse images of bicycles and cats, the model can easily distinguish between the two categories. However, if the samples are fused from images of cats and dogs, the model will have difficulty accurately distinguishing between the two categories in the images, because dogs are more similar in appearance to cats than to bicycles. Summary of the Invention

[0004] In order to solve the problems of the prior art, the present invention provides an image data enhancement method, device, medium and equipment. The technical solution is as follows:

[0005] In a first aspect, the present invention provides an image data enhancement method, the method comprising:

[0006] Training a deep learning network based on an image training set to obtain a soft label generator, wherein the image training set includes a set of image samples of K categories;

[0007] Obtain a first image sample, and obtain a soft label and a category of the first image sample according to the soft label generator, wherein the soft label represents a category association between the first image sample and the K categories;

[0008] Obtaining, according to the soft label generator, category representativeness of image samples in an image sample set of K-1 categories, wherein the K-1 categories do not include the category of the first image sample;

[0009] Selecting at least one image sample from the K-1 categories of image sample sets as a second image sample based on the soft label of the first image sample and the category representativeness of the image sample;

[0010] The first image sample and the second image sample are fused to obtain a target image sample, thereby completing image data enhancement of the image training set.

[0011] Furthermore, obtaining the category representativeness of image samples in the image sample set of K-1 categories according to the soft label generator includes:

[0012] Mapping the image samples in the K-1 categories of image sample sets to the same feature space by the soft label generator;

[0013] Obtaining the image sample space vector sets of the K-1 categories in the feature space;

[0014] Calculating based on the image sample space vector sets of the K-1 categories to obtain the center vectors of the K-1 categories in the feature space;

[0015] The category representativeness of the image samples in the image sample sets of the K-1 categories is calculated according to the center vectors of the K-1 categories.

[0016] Furthermore, the calculating, based on the center vectors of the K-1 categories, the category representativeness of the image samples in the image sample set of the K-1 categories includes:

[0017] Determine the center vector and image sample space vector set of the target category in the K-1 categories;

[0018] Based on a vector similarity measurement algorithm, for each image sample space vector in the image sample space vector set of the target category, calculating the similarity with the central vector of the target category;

[0019] Determine the category representativeness of the image samples in the image sample set of the target category according to the similarity.

[0020] Furthermore, selecting at least one image sample as the second image sample from the image sample set of the K-1 categories according to the soft label of the first image sample and the category representativeness of the image sample includes:

[0021] According to the soft label of the first image sample, select a category that satisfies a first correlation degree from K-1 categories as an optional category;

[0022] Acquire an image sample set of the optional category and the category representativeness of the image samples in the image sample set of the optional category;

[0023] At least one image sample is selected from the set of image samples of the optional category as the second image sample, and the category representativeness of the at least one image sample satisfies the second degree of association.

[0024] Furthermore, selecting at least one image sample as the second image sample from the image sample set of the K-1 categories according to the soft label of the first image sample and the category representativeness of the image sample further includes:

[0025] sorting the image samples among classes according to the soft labels of the first image samples to obtain a first sequence;

[0026] sorting the image samples within a category according to their category representativeness to obtain a second sequence;

[0027] Obtaining a comprehensive sequence of image samples according to the first sequence and the second sequence;

[0028] At least one image sample is selected as a second image sample according to the comprehensive ranking of the image samples, and a sequence position of the second image sample in the comprehensive sequence meets a preset threshold.

[0029] Furthermore, the fusing the first image sample and the second image sample to obtain a target image sample includes:

[0030] fusing the first image sample and the second image sample by interpolation or filling to obtain a target image sample;

[0031] Acquire a soft label of the second image sample by the soft label generator;

[0032] The soft label of the first image sample and the soft label of the second image sample are calculated to obtain a label of the target image sample.

[0033] Furthermore, the acquiring of the first image sample and the acquiring of the soft label and category of the first image sample according to the soft label generator include:

[0034] Extracting an image sample from the sample sets of the K categories as the first image sample;

[0035] Acquire a soft label for the first image sample according to the soft label generator;

[0036] The category of the first image sample is calculated and determined according to the soft label of the first image sample, where the category of the first image sample is any one of the K categories.

[0037] In a second aspect, the present invention provides an image data enhancement device, the device comprising:

[0038] A training module is used to train the deep learning network based on an image training set to obtain a soft label generator, wherein the image training set includes a set of image samples of K categories;

[0039] a first acquisition module, configured to acquire a first image sample, and acquire a soft label and a category of the first image sample according to the soft label generator, wherein the soft label represents a degree of association between the first image sample and the K categories;

[0040] A second acquisition module is configured to acquire, according to the soft label generator, category representativeness of image samples in a set of K-1 categories of image samples, wherein the K-1 categories do not include the category of the first image sample;

[0041] a selection module, configured to select at least one image sample as a second image sample from the image sample set of the K-1 categories based on the soft label of the first image sample and the category representativeness of the image sample;

[0042] The fusion module is used to fuse the first image sample and the second image sample to obtain a target image sample, thereby completing image data enhancement of the image training set.

[0043] In a third aspect, the present invention provides a computer-readable storage medium, which stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement an image data enhancement method as described in the first aspect.

[0044] In a fourth aspect, the present invention provides a computer device comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement an image data enhancement method as described in the first aspect.

[0045] The present invention provides an image data enhancement method, device, equipment, and storage medium, which have the following technical effects:

[0046] (1) The solution provided by the present invention performs data enhancement of mixed samples based on the image dataset to generate more training samples, solve the problem of insufficient training data, and reduce overfitting caused by model training;

[0047] (2) The solution provided by the present invention constructs soft labels to describe the category relationship of image data and uses them as the basis for selecting fusion samples to synthesize training samples with controllable difficulty, thereby adjusting the difficulty of classification network training and improving the generalization ability of deep learning classification networks;

[0048] (3) The present invention can better determine the direction of model training by controlling the difficulty of training samples, reduce the time and computing power consumption caused by too many simple samples in model training, and enable the deep network model to converge better and faster.

[0049] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0051] Figure 1 This is a schematic diagram of randomly selecting samples for image data enhancement provided by an embodiment of the present invention;

[0052] Figure 2 Schematic diagram of an implementation environment of an image data enhancement method provided by an embodiment of the present invention;

[0053] Figure 3 1 is a flow chart of an image data enhancement method provided by an embodiment of the present invention;

[0054] Figure 4 is a schematic diagram of a training classification network in an image data enhancement method provided by an embodiment of the present invention;

[0055] Figure 5 1 is a flow chart of obtaining category representativeness of image samples in a set of K-1 categories of image samples according to the soft label generator provided by an embodiment of the present invention;

[0056] Figure 6 is a schematic diagram of a process for calculating the category representativeness of image samples provided by an embodiment of the present invention;

[0057] FIG7(1) is a schematic diagram of a process of selecting at least one image sample as a second image sample from the image sample set of K-1 categories provided by an embodiment of the present invention;

[0058] FIG7(2) is another flowchart of selecting at least one image sample as the second image sample from the image sample set of K-1 categories provided by an embodiment of the present invention;

[0059] Figure 8 is a schematic diagram of selecting samples for fusion based on soft labels in an image data enhancement method provided by an embodiment of the present invention;

[0060] Figure 9 Schematic diagram of an image data enhancement device provided by an embodiment of the present invention.

[0061] Figure 10 The figure is a schematic diagram of the hardware structure of a device for implementing an image data enhancement method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0062] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics.

[0063] The solution provided in the embodiments of this application involves technologies such as deep learning (DL) of artificial intelligence and computer vision (CV).

[0064] Deep learning is a new research direction in the field of machine learning (ML). It was introduced to bring ML closer to its original goal: artificial intelligence. Deep learning studies the inherent patterns and representational hierarchies of sample data. The information gained from this learning process is highly helpful in interpreting data such as text, images, and sound. Its ultimate goal is to enable machines to acquire human-like analytical learning capabilities and recognize data such as text, images, and sound.

[0065] Computer vision is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying, tracking, and measuring objects. Further image processing is performed to transform the computer-generated images into images more suitable for human observation or transmission to instrumentation. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0066] For image classification networks and image training samples, data augmentation methods for mixed samples, such as Mixup, Cutout, and CutMix, have been proposed. All three methods improve the generalization of deep learning classification networks by generating difficult examples. However, these methods synthesize new difficult examples by randomly extracting two examples from the training set and fusing them. This lacks consideration of the correlation between categories, making it difficult to measure and control the difficulty of training the network model with the synthesized samples.

[0067] In order to control the difficulty of synthesizing samples and improve the generalization ability of the image classification network, the embodiments of the present invention provide an image data enhancement method, device, medium and equipment. The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. Examples of the embodiments are shown in the drawings, where the same or similar numbers throughout represent the same or similar elements or elements with the same or similar functions.

[0068] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0069] To facilitate understanding of the technical solutions and the technical effects produced by the embodiments of the present invention, the following explanations are given for the relevant professional terms involved in the embodiments of the present invention:

[0070] CNN: Convolutional Neural Network, a deep learning network widely used in image classification tasks. The network includes convolutional layers, pooling layers, and fully connected layers.

[0071] Soft-label: Compared with hard-label, which is either 0 or 1, soft label has the advantage of being able to depict the relationship between categories.

[0072] Generalization refers to a machine learning algorithm's ability to adapt to new samples. The goal of learning is to learn the patterns underlying the data. This ability allows a trained network to produce appropriate outputs for data outside of the learning set that exhibits the same patterns.

[0073] See also Figure 2 , which is a schematic diagram of an implementation environment of an image data enhancement method provided in an embodiment of the present application, such as Figure 2 As shown, the implementation environment may include at least a client 01 and a server 02 .

[0074] Specifically, the client 01 may include devices such as smartphones, desktop computers, tablet computers, laptops, digital assistants, smart wearable devices, monitoring devices, and voice interaction devices. It may also include software running on the device, such as web pages provided by service providers to users, or applications provided by these service providers to users. Specifically, the client 01 may be used to display the image training set or the first image sample, as well as the second image sample and the fused target image sample sent by the server 02.

[0075] Specifically, the server 02 may include a standalone server, a distributed server, or a server cluster consisting of multiple servers. The server 02 may include a network communication unit, a processor, and a memory. Specifically, the server 02 may be used to train a classification network based on an image training set to obtain a soft label generator, perform feature mapping on image samples, select image samples that meet a preset correlation, and fuse them to generate new target image samples.

[0076] The following describes an image data enhancement method provided by the present invention. Figure 3 It is a flowchart of an image data enhancement method provided by an embodiment of the present invention. This specification provides the method operation steps as described in the embodiment or flowchart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many steps, and does not represent the only execution order. When the actual system or server product is executed, it can be executed in sequence or in parallel according to the method shown in the embodiment or the accompanying drawings (for example, in a parallel processor or multi-threaded processing environment). Please refer to Figure 3 , an image data enhancement method provided in an embodiment of this specification includes the following steps:

[0077] S101: Train a deep learning network according to an image training set to obtain a soft label generator, wherein the image training set includes image sample sets of K categories.

[0078] It is understandable that when constructing image classification network models, recognition network models, etc., a large amount of sample data will be required for pre-training in order to adjust the parameters of the network model to improve its performance. It can be seen that the characteristics and quantity of training data are the most important factors that determine the performance of a network model. Specifically, for image classification tasks, insufficient training sample data will cause overfitting of the model, that is, the classification of training sample data can achieve good results, but the performance on test data is not good. In order to alleviate the problem caused by insufficient training data, the training data set can be expanded through data enhancement. Commonly, geometric transformations, color transformations, pixel transformations and other operations can be performed on each image in the original image space, or image features can be extracted and transformed in the feature space of the image. New samples can also be synthesized by generating models, which is a mixed sample data enhancement method.

[0079] In the embodiments of this specification, the present invention provides an image data enhancement method that performs data enhancement by extracting image samples from an image training set to synthesize new samples, and controls the difficulty of synthesizing samples to better serve the training of image classification models.

[0080] In the embodiments of this specification, the public ImageNet (a large visual database for visual object recognition software research), CIFAR-100 and / or CIFAR-10 are used as the image training set. For example, the CIFAR-10 dataset consists of 60,000 32x32 color images of 10 classes, with 6,000 images per class, divided into 50,000 training images and 10,000 test images. In a feasible embodiment, as Figure 4 As shown in the figure, a convolutional neural network is trained using all image samples in the training set to generate a soft label generator. The entire training process is a 1-of-K classification training, where K is the number of categories in the training set. The cross-entropy loss function is used to supervise the training and optimization of the soft label generator. The image classification network can be any network model that can be used for image classification, such as the VGGNet model, AlexNet model, GoogLeNet model, and ResNet model.

[0081] S103: Acquire a first image sample, and acquire a soft label and a category of the first image sample according to the soft label generator, where the soft label represents a category association between the first image sample and the K categories.

[0082] In a feasible implementation, specifically, step S103 may include the following steps:

[0083] S301: Extracting image samples from the sample sets of the K categories as the first image samples.

[0084] In this embodiment, data augmentation, also known as data expansion, is performed on an existing image training set using a mixed sample approach. A random image sample or an image sample of a specified type is selected from the original image training set as one of the fusion objects. In the original image training set, the image samples are divided into K categories, each with a corresponding category label.

[0085] S303: Obtain a soft label for the first image sample according to the soft label generator.

[0086] Specifically, such as Figure 8 As shown, the weights of the trained soft label generator are frozen. When the first image sample is input into the soft label generator, we can obtain the soft label of the image.

[0087] It can be understood that, numerically, soft labels represent the probability of the first image sample being classified into each of the K categories. For example, for a three-category classification task, a hard label might describe the category of an image as (1, 0, 0), with the category corresponding to 1 selected as the classification result. A soft label might describe the category of the same image as (0.6, 0.35, 0.05), with the category corresponding to the maximum probability of 0.6 selected as the classification result. The classification results are the same, but compared to hard labels, soft labels contain the probability of the input data being classified into each category, preserving more information from the original data. They also capture the relationship between categories. For example, in (0.6, 0.35, 0.05), the correlation between the categories 0.6 and 0.35 is high, indicating that the classification of these two categories is difficult. For example, in a picture of a dog, the soft label value for the cat category is often higher, while the value (i.e., correlation) for the bicycle category is lower.

[0088] In this specification, the image training set used includes K categories (K is a positive integer). Therefore, the soft label of the first image sample obtained by the soft label generator can be represented by a K-dimensional vector. In other embodiments, based on the sum of probability values ​​being 1, it can also be represented by a K-1-dimensional vector. This specification uses a K-dimensional vector as an example for the subsequent description.

[0089] S305: Calculate and determine the category of the first image sample according to the soft label of the first image sample, where the category of the first image sample is any one of the K categories.

[0090] In this embodiment, soft labels are used to represent the degree of association / similarity between the image sample's category and other categories. Therefore, if we need to generate image samples that are difficult for the classification network to classify, we can select other categories with higher probabilities from the soft labels of the first image sample and select samples from the image sample set of that category. Conversely, selecting samples from other categories with lower probabilities can generate easier samples. Compared to randomly selecting samples from the image training set for fusion, this embodiment selects fusion sample categories based on soft labels, allowing us to control the difficulty of image samples.

[0091] S105: Obtaining, according to the soft label generator, category representativeness of image samples in an image sample set of K-1 categories, where the K-1 categories do not include the category of the first image sample.

[0092] It can be understood that the category representativeness of the image sample can characterize the category feature strength of the image sample, and can also characterize the difficulty of the classification network model to classify the image sample. If the category representativeness of the image sample is high, the model can more easily classify it into this category. If the category representativeness of the image sample point is low, although the model can also classify it correctly, it is more difficult.

[0093] In the embodiments of this specification, in order to control the classification difficulty of the fusion samples, in addition to selecting the category of the fusion samples, image samples can be further selected from the image sample set of the category to better control the difficulty of training.

[0094] In a feasible embodiment, specifically, as Figure 5 As shown, the step of obtaining the category representativeness of image samples in the image sample set of K-1 categories according to the soft label generator may include the following steps:

[0095] S501: Mapping the image samples in the image sample sets of the K-1 categories to the same feature space by the soft label generator.

[0096] In one embodiment of this specification, a feature space and a spatial vector are used to represent image samples and thereby represent the category representativeness of the image samples. The soft label generator is essentially a classification network that can map image samples to a feature space based on feature extraction and represent the image samples in the form of a spatial vector.

[0097] It can be understood that in the method provided in this specification, during the sample selection process, the selection range is limited to image samples in the image sample set of K-1 categories, and the category of the first image sample is not included. This is because when selecting fused samples in the image sample set of the first image sample category, the difficulty of the new image sample after the fusion operation is not easy to define and control, which may strengthen or weaken the characteristic performance of the category, and less new information is generated.

[0098] S503: Obtain the image sample space vector sets of the K-1 categories in the feature space.

[0099] In a feasible implementation, after the image samples are mapped to the feature space, a fixed-length space vector is generated for each image sample, and the space vectors of image samples of the same category are taken as a set for subsequent calculations.

[0100] S505: Perform calculations based on the image sample space vector sets of the K-1 categories to obtain the center vectors of the K-1 categories in the feature space.

[0101] In one feasible implementation, for a set of image sample space vectors of a category, the center vector of the category is calculated by averaging all the image sample space vectors in the set. The center vector is the representation of the category in the feature space and does not correspond to a specific image sample.

[0102] S507: Calculate the category representativeness of the image samples in the image sample set of the K-1 categories according to the center vectors of the K-1 categories.

[0103] In a feasible embodiment, specifically, as Figure 6 As shown, the calculating of the category representativeness of the image samples in the image sample set of the K-1 categories may include the following steps:

[0104] S601: Determine the center vector and image sample space vector set of the target category in the K-1 categories.

[0105] It is understandable that any category among the K-1 categories is used as the target category for illustration here, and the category representativeness of image samples of other categories can be calculated based on the same method.

[0106] S603: Based on a vector similarity measurement algorithm, calculate the similarity between each image sample space vector in the image sample space vector set of the target category and the central vector of the target category.

[0107] In one embodiment of the present specification, the Earth Mover's distance (EMD) is used to measure the similarity between an image sample N of a target category and a center vector of the target category, and the category representativeness of the image sample is characterized according to the similarity.

[0108] The specific calculation formula of EMD can be shown as follows:

[0109]

[0110] Among them, F C represents the center vector of the target category, F N Represents the spatial vector of image samples in the target category image sample set. This equation contains the following constraints:

[0111]

[0112] S605: Determine the category representativeness of the image samples in the image sample set of the target category according to the similarity.

[0113] In one embodiment of the present specification, by d N Describes the category representativeness of the image sample, dN The smaller it is, the smaller the distance between the spatial vector of the image sample N and the center vector of the category is, the higher the similarity is, and the more representative the image sample N is of the category, that is, there is more information that can represent the category. On the contrary, d N The larger the value is, the more difficult the image sample is in this category and has less feature information of this category.

[0114] In other embodiments of this specification, the similarity between the spatial vector and the center vector of the image sample can also be calculated using other vector similarity algorithms such as Euclidean distance. In addition to using feature spaces and spatial vectors, the embodiments of this specification can also use other methods to calculate and characterize the category representativeness of image samples, and this invention does not limit this comparison.

[0115] S107: Select at least one image sample from the image sample sets of the K-1 categories as a second image sample according to the soft label of the first image sample and the category representativeness of the image sample.

[0116] In a feasible implementation, specifically, as shown in FIG7(1), step S107 may include the following steps:

[0117] S701: According to the soft label of the first image sample, select a category that meets a first correlation degree from K-1 categories as an optional category.

[0118] In terms of category selection, the category of the fused sample is selected based on the soft label. Specifically, if a difficult example needs to be generated, samples from categories that are highly correlated with the classification category of the first image sample are selected for fusion. Conversely, samples from categories with a low degree of correlation are selected to generate easy examples.

[0119] S703: Acquire the image sample set of the optional category and the category representativeness of the image samples in the image sample set of the optional category.

[0120] S705: Select at least one image sample from the set of image samples of the optional category as a second image sample, where the category representativeness of the at least one image sample satisfies a second degree of association.

[0121] In the embodiments of this specification, one image sample may be selected to be fused with the first image sample, or multiple image samples may be selected to be fused with the first image sample. This can be achieved based on the same method and principle and is not limited here.

[0122] The methods provided in the embodiments of this specification also control the difficulty of sample generation by selecting image samples for fusion based on their category representativeness. Specifically, if difficult examples are required, image samples with low category representativeness are selected for fusion from the set of image samples with the same category as the selectable category. Conversely, image samples with high category representativeness are selected to generate easy examples.

[0123] It is understandable that the two-step selection method based on categories and samples can be applied to both difficult examples with higher difficulty and easy examples with lower difficulty. For example, from the category with the highest degree of category correlation, the image sample with the lowest category representativeness is selected and fused with the first image sample to generate a difficult example, or from the category with the lowest degree of category correlation, the image sample with the highest category representativeness is selected and fused with the first image sample to generate an easy example.

[0124] In another feasible implementation, specifically, as shown in FIG7(2), step S107 may further include the following steps:

[0125] S702: Sort the image samples among classes according to the soft labels of the first image samples to obtain a first sequence.

[0126] S704: Sort the image samples within a category according to their category representativeness to obtain a second sequence.

[0127] S706: Obtain a comprehensive sequence of the image samples according to the first sequence and the second sequence.

[0128] In the embodiment of this specification, all image samples in the image sample set of K-1 categories are sorted, and a first sequence R is obtained according to the degree of category association. l , and the second sequence R is obtained based on the category representativeness d , and then get the comprehensive sequence R jnt , the comprehensive sequence can be represented by the following formula:

[0129] R jnt =f ind (↓R l )*f ind (↓R d )

[0130] Among them, f ind It is the position retrieval function in two sequences, and ↓ means sorting from large to small.

[0131] S708: Select at least one image sample as a second image sample according to the comprehensive ranking of the image samples, where the sequence position of the second image sample in the comprehensive sequence meets a preset threshold.

[0132] From the above formula, we can see that the selection of the second image sample is based on R jnt It can be divided into the following situations:

[0133] (1) High R l , high R d ->High R jnt : difficult examples with higher difficulty;

[0134] (2) Low R l , low R d -> Low R jnt : Easy examples with lower difficulty;

[0135] (3) High R l , low R d or low R l , high R d -> Middle R jnt : Examples of moderate difficulty can be sampled more frequently.

[0136] The embodiments of this specification list two selection steps, and other selection principles can also be selected. The present invention does not limit this, and the selection can be made according to actual needs during application.

[0137] S109: Fusing the first image sample and the second image sample to obtain a target image sample, thereby completing image data enhancement of the image training set.

[0138] In the embodiment provided by the present invention, specifically, the fusing of the first image sample and the second image sample may include the following steps:

[0139] S901: Fusing the first image sample and the second image sample by interpolation or filling to obtain a target image sample.

[0140] Specifically, the second image sample is determined by controlling the difficulty of the sample, and the first image sample and the second image sample are fused. Taking the second image sample as an example, the target image sample is obtained by performing a pixel-level weighted average of the two image samples using interpolation (Mixup); or the two image samples are fused using filling (CutMix).

[0141] S903: Obtain a soft label for the second image sample through the soft label generator.

[0142] It is understandable that, after the target image samples are obtained by fusion as training image samples, the labels of the target image samples need to be determined.

[0143] S905: Calculate the soft label of the first image sample and the soft label of the second image sample to obtain the label of the target image sample.

[0144] Specifically, depending on the fusion method used, an appropriate label calculation method is used to determine the label of the target image sample. For example, if a Mixup fusion method is used, the soft labels of the first image sample and the second image sample are weighted averaged to further determine the label of the target image sample to update the image training set.

[0145] The image data enhancement method provided in the embodiment of this specification is mainly divided into two steps. The first step can be as follows: Figure 4 As shown, a classification network is trained as a soft label generator; the second step can be as follows Figure 8 As shown in the figure, the trained network weights are frozen, an image sample is input, and image samples from other categories are selected for fusion based on its soft label. During the sample selection process, the difficulty of synthesizing samples is controlled based on the category relevance and category representativeness to improve the generalization ability of the classification model.

[0146] The embodiment of the present invention also provides an image data enhancement device, such as Figure 9 As shown, the device may include:

[0147] The training module 910 is used to train the deep learning network according to the image training set to obtain a soft label generator, wherein the image training set includes image sample sets of K categories.

[0148] The first acquisition module 920 is configured to acquire a first image sample and obtain a soft label and a category of the first image sample according to the soft label generator, where the soft label represents a category association between the first image sample and the K categories.

[0149] The second acquisition module 930 acquires, according to the soft label generator, category representativeness of image samples in a set of K-1 categories of image samples, where the K-1 categories do not include the category of the first image sample.

[0150] The selection module 940 is configured to select at least one image sample from the image sample set of the K-1 categories as a second image sample according to the soft label of the first image sample and the category representativeness of the image sample.

[0151] The fusion module 950 is used to fuse the first image sample and the second image sample to obtain a target image sample, thereby completing image data enhancement of the image training set.

[0152] In one embodiment of the present specification, the second acquisition module 930 may include:

[0153] The feature mapping unit 931 is configured to map the image samples in the image sample sets of the K-1 categories to the same feature space through the soft label generator.

[0154] The first acquiring unit 932 is configured to acquire the image sample space vector sets of the K-1 categories in the feature space.

[0155] The first calculation unit 933 is configured to perform calculation based on the image sample space vector sets of the K-1 categories to obtain the center vectors of the K-1 categories in the feature space.

[0156] The second calculation unit 934 is configured to calculate the category representativeness of the image samples in the image sample set of the K-1 categories according to the center vectors of the K-1 categories.

[0157] In one embodiment of the present specification, the selection module 940 may include:

[0158] The category selection unit 941 is configured to select a category satisfying a first correlation degree from K-1 categories according to the soft label of the first image sample as a selectable category.

[0159] The second acquiring unit 943 is configured to acquire the image sample set of the optional category and the category representativeness of the image samples in the image sample set of the optional category.

[0160] The first selection unit 945 is configured to select at least one image sample from the set of image samples of the optional category as a second image sample, where the category representativeness of the at least one image sample satisfies a second degree of association.

[0161] In another embodiment of the present specification, the selection module 940 may further include:

[0162] The first sequence unit 942 is configured to perform inter-class sorting on the image samples according to the soft labels of the first image samples to obtain a first sequence.

[0163] The second sequence unit 944 is configured to sort the image samples within a class according to their class representativeness to obtain a second sequence.

[0164] The integrated sequence unit 946 is configured to obtain an integrated sequence of the image samples according to the first sequence and the second sequence.

[0165] The second selection unit 948 is configured to select at least one image sample as a second image sample according to the comprehensive ranking of the image samples, wherein the sequence position of the second image sample in the comprehensive sequence meets a preset threshold.

[0166] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0167] An embodiment of the present invention provides a computer device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement an image data enhancement method provided in the above method embodiment.

[0168] Figure 10 A schematic diagram of the hardware structure of a device for implementing the method provided in the embodiment of the present application is shown, and the device may participate in or include the apparatus or system provided in the embodiment of the present application. Figure 10 As shown, the device 10 may include one or more (illustrated as 1002a, 1002b, ..., 1002n in the figure) processors 1002 (the processor 1002 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 1004 for storing data, and a transmission device 1006 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply and / or a camera. It will be understood by those skilled in the art that Figure 10 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 10 More or fewer components than shown, or with Figure 10 Different configurations shown.

[0169] It should be noted that the one or more processors 1002 and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components of the device 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0170] The memory 1004 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the methods described in the embodiments of the present application. The processor 1002 executes various functional applications and data processing by running the software programs and modules stored in the memory 1004, thereby implementing the above-mentioned image data enhancement method. The memory 1004 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 1004 may further include a memory remotely located relative to the processor 1002, and these remote memories may be connected to the device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0171] The transmission device 1006 is used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by the communications provider of the device 10. In one embodiment, the transmission device 1006 may include a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In one embodiment, the transmission device 1006 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0172] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of device 10 (or mobile device).

[0173] An embodiment of the present invention also provides a computer-readable storage medium, which can be set in a server to store at least one instruction or at least one program related to implementing an image data enhancement method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement an image data enhancement method provided in the above method embodiment.

[0174] Optionally, in this embodiment, the storage medium may be located in at least one of a plurality of network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0175] It can be seen from the embodiments of the image data enhancement method, device, medium and equipment provided by the present invention that:

[0176] (1) The solution provided by the present invention performs data enhancement of mixed samples based on the image dataset to generate more training samples, solve the problem of insufficient training data, and reduce overfitting caused by model training;

[0177] (2) The solution provided by the present invention constructs soft labels to describe the category relationship of image data and uses them as the basis for selecting fusion samples to synthesize training samples with controllable difficulty, thereby adjusting the difficulty of classification network training and improving the generalization ability of deep learning classification networks;

[0178] (3) The present invention can better determine the direction of model training by controlling the difficulty of training samples, reduce the time and computing power consumption caused by too many simple samples in model training, and enable the deep network model to converge better and faster.

[0179] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments. The above description is of specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0180] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, equipment, and storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simplified. For relevant portions, refer to the descriptions of the method embodiments.

[0181] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0182] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for image data enhancement, characterized in that: The method comprises: Training a deep learning network based on an image training set to obtain a soft label generator, wherein the image training set includes a set of image samples of K categories; Obtain a first image sample, and obtain a soft label and a category of the first image sample according to the soft label generator, wherein the soft label represents a category association degree between the first image sample and the K categories; Obtaining, according to the soft label generator, category representativeness of image samples in an image sample set of K-1 categories, wherein the K-1 categories do not include the category of the first image sample; Selecting at least one image sample from the K-1 categories of image sample sets as a second image sample based on the soft label of the first image sample and the category representativeness of the image sample; The first image sample and the second image sample are fused to obtain a target image sample, thereby completing image data enhancement of the image training set.

2. The image data enhancement method according to claim 1, wherein: The category representativeness of the image samples in the image sample set of K-1 categories obtained by the soft label generator includes: Mapping the image samples in the K-1 categories of image sample sets to the same feature space by the soft label generator; Obtaining the image sample space vector sets of the K-1 categories in the feature space; Calculating based on the image sample space vector sets of the K-1 categories to obtain the center vectors of the K-1 categories in the feature space; The category representativeness of the image samples in the image sample sets of the K-1 categories is calculated according to the center vectors of the K-1 categories.

3. The image data enhancement method according to claim 2, wherein: Calculating the category representativeness of the image samples in the image sample set of the K-1 categories according to the center vectors of the K-1 categories includes: Determine the center vector and image sample space vector set of the target category in the K-1 categories; Based on a vector similarity measurement algorithm, for each image sample space vector in the image sample space vector set of the target category, calculating the similarity with the central vector of the target category; Determine the category representativeness of the image samples in the image sample set of the target category according to the similarity.

4. The image data enhancement method according to claim 1, wherein: The selecting, according to the soft label of the first image sample and the category representativeness of the image sample, at least one image sample from the image sample set of the K-1 categories as the second image sample comprises: According to the soft label of the first image sample, select a category that satisfies a first correlation degree from K-1 categories as an optional category; Acquire an image sample set of the optional category and the category representativeness of the image samples in the image sample set of the optional category; At least one image sample is selected from the set of image samples of the optional category as the second image sample, and the category representativeness of the at least one image sample satisfies the second degree of association.

5. The image data enhancement method according to claim 1, wherein: The selecting at least one image sample as the second image sample from the image sample set of the K-1 categories according to the soft label of the first image sample and the category representativeness of the image sample further includes: sorting the image samples among classes according to the soft labels of the first image samples to obtain a first sequence; sorting the image samples within a category according to their category representativeness to obtain a second sequence; Obtaining a comprehensive sequence of image samples according to the first sequence and the second sequence; At least one image sample is selected as a second image sample according to the comprehensive ranking of the image samples, and a sequence position of the second image sample in the comprehensive sequence meets a preset threshold.

6. An image data enhancement method according to any one of claims 4 or 5, characterized in that: The fusing the first image sample and the second image sample to obtain a target image sample includes: fusing the first image sample and the second image sample by interpolation or filling to obtain a target image sample; Acquire a soft label of the second image sample by the soft label generator; The soft label of the first image sample and the soft label of the second image sample are calculated to obtain a label of the target image sample.

7. The image data enhancement method according to claim 1, characterized in that: The acquiring of the first image sample and the acquiring of the soft label and category of the first image sample according to the soft label generator include: Extracting an image sample from the sample sets of the K categories as the first image sample; Acquire a soft label for the first image sample according to the soft label generator; The category of the first image sample is calculated and determined according to the soft label of the first image sample, where the category of the first image sample is any one of the K categories.

8. An image data enhancement device, characterized in that: The device comprises: A training module is used to train the deep learning network based on an image training set to obtain a soft label generator, wherein the image training set includes a set of image samples of K categories; a first acquisition module, configured to acquire a first image sample, and acquire a soft label and a category of the first image sample according to the soft label generator, wherein the soft label represents a degree of association between the first image sample and the K categories; A second acquisition module is configured to acquire, according to the soft label generator, category representativeness of image samples in a set of K-1 categories of image samples, wherein the K-1 categories do not include the category of the first image sample; a selection module, configured to select at least one image sample as a second image sample from the image sample set of the K-1 categories based on the soft label of the first image sample and the category representativeness of the image sample; The fusion module is used to fuse the first image sample and the second image sample to obtain a target image sample, thereby completing image data enhancement of the image training set.

9. A computer storage medium, characterized in that The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement an image data enhancement method as described in any one of claims 1 to 7.

10. A computer device, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded by the processor and executed by an image data enhancement method according to any one of claims 1 to 7.

Images