Picture processing method and apparatus

Abstract

The application provides a picture processing method and device, and belongs to the technical field of artificial intelligence. The picture processing method comprises the following steps: inputting a picture to be processed into a feature extractor of a neural network classification model to obtain features of the picture; inputting the features of the picture into a semantic direction prediction network model to obtain a first semantic direction with meaning of the picture; enhancing the features of the picture along the first semantic direction to obtain enhanced features; and inputting the enhanced features into a classifier of the neural network classification model to obtain a classification result of the picture. The technical scheme of the application can avoid destroying classification information when data enhancement is performed in the image domain.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an image processing method and apparatus. Background Technology

[0002] Data augmentation techniques are widely used in the training process of deep neural networks. By randomly performing data augmentation operations such as masking, cropping, flipping, geometric transformation, and brightness adjustment on training images, the generalization ability of the model on the test set can be effectively improved.

[0003] However, data augmentation methods performed in the image domain can randomly destroy the information of the training images in the image domain, thus causing confusion for the network during training. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an image processing method and apparatus that can avoid destroying classification information when performing data augmentation in the image domain.

[0005] On one hand, embodiments of the present invention provide an image processing method, including:

[0006] The image to be processed is input into the feature extractor of the neural network classification model to obtain the features of the image;

[0007] The features of the image are input into a semantic direction prediction network model to obtain the first semantic direction of the image that has meaning.

[0008] The features of the image are enhanced along the first semantic direction to obtain the enhanced features;

[0009] The enhanced features are input into the classifier of the neural network classification model to obtain the classification result of the image.

[0010] In some embodiments, enhancing the features of the image along the first semantic direction to obtain enhanced features includes:

[0011] From normal distribution M random samples are performed to obtain the features after M data augmentations, and the normal distribution is... Based on feature a of the image i The mean is used to represent the semantic direction Σ. i The covariance matrix is ​​the variance, and M is a positive integer.

[0012] In some embodiments, after obtaining the classification result of the image, the method further includes:

[0013] Calculate the loss of the neural network classification model on the training data;

[0014] The parameters of the neural network classification model are updated for the first time using the loss on the training data.

[0015] In some embodiments, after performing a first update on the parameters of the neural network classification model using the loss on the training data, the method further includes:

[0016] The cross-entropy loss of the metadata of the semantic orientation prediction network model is calculated using the updated parameters of the neural network classification model.

[0017] The semantic direction prediction network model is updated based on the cross-entropy loss and the learning rate of the semantic direction prediction network at the current time step.

[0018] In some embodiments, after updating the semantic direction prediction network model based on the cross-entropy loss and the current learning rate of the semantic direction prediction network, the method further includes:

[0019] The features of the image are input into the updated semantic direction prediction network model to obtain the second semantic direction of the image.

[0020] The loss of the neural network classification model on the training data is calculated based on the second semantic direction;

[0021] The parameters of the neural network classification model are updated a second time using the loss on the training data.

[0022] This invention also provides an image processing apparatus, comprising:

[0023] The first processing module is used to input the image to be processed into the feature extractor of the neural network classification model to obtain the features of the image;

[0024] The second processing module is used to input the features of the image into the semantic direction prediction network model to obtain the first semantic direction of the image that has meaning.

[0025] The enhancement module is used to enhance the features of the image along the first semantic direction to obtain enhanced features;

[0026] The classification module is used to input the enhanced features into the classifier of the neural network classification model to obtain the classification result of the image.

[0027] In some embodiments, the enhancement module is specifically used to obtain data from a normal distribution. M random samples are performed to obtain the features after M data augmentations, and the normal distribution is... Based on feature a of the image iThe mean is used to represent the semantic direction Σ. i The covariance matrix is ​​the variance, and M is a positive integer.

[0028] In some embodiments, the apparatus further includes:

[0029] The first update module is used to calculate the loss on the training data of the neural network classification model; and to update the parameters of the neural network classification model using the loss on the training data.

[0030] In some embodiments, the apparatus further includes:

[0031] The second update module is used to calculate the cross-entropy loss of the metadata of the semantic direction prediction network model using the updated parameters of the neural network classification model; and to update the semantic direction prediction network model according to the cross-entropy loss and the learning rate of the semantic direction prediction network at the current time.

[0032] In some embodiments, the apparatus further includes:

[0033] The third update module is used to input the features of the image into the updated semantic direction prediction network model to obtain the second semantic direction of the image; calculate the loss on the training data of the neural network classification model based on the second semantic direction; and use the loss on the training data to perform a second update on the parameters of the neural network classification model.

[0034] This invention also provides an image processing apparatus, including a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the program, it implements the image processing method described above.

[0035] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the image processing method described above.

[0036] The embodiments of the present invention have the following beneficial effects:

[0037] In the above scheme, the features of the image to be processed are obtained, and these features are enhanced along a first semantic direction to obtain enhanced features. The image is then classified based on these enhanced features. This embodiment performs data augmentation in the feature space, which avoids destroying classification information when performing data augmentation in the image domain. Attached Figure Description

[0038] Figure 1 This is a flowchart illustrating the image processing method according to an embodiment of the present invention;

[0039] Figure 2This is a schematic diagram of the image processing device according to an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the composition of the image processing device according to an embodiment of the present invention. Detailed Implementation

[0041] To make the technical problems, technical solutions and advantages of the embodiments of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0042] This invention provides an image processing method and apparatus that can avoid destroying classification information when performing data augmentation in the image domain.

[0043] Embodiments of the present invention provide an image processing method, such as... Figure 1 As shown, it includes:

[0044] Step 101: Input the image to be processed into the feature extractor of the neural network classification model to obtain the features of the image;

[0045] Step 102: Input the features of the image into the semantic direction prediction network model to obtain the first semantic direction of the image that has meaning;

[0046] Step 103: Enhance the features of the image along the first semantic direction to obtain enhanced features;

[0047] Step 104: Input the enhanced features into the classifier of the neural network classification model to obtain the classification result of the image.

[0048] In this embodiment, the features of the image to be processed are obtained, and these features are enhanced along a first semantic direction to obtain enhanced features. The image is then classified based on these enhanced features. This embodiment performs data augmentation in the feature space, which avoids the destruction of classification information when performing data augmentation in the image domain. This embodiment can be applied to various fine-grained image classification scenarios and can significantly improve the classification performance of various mainstream neural network models.

[0049] Data augmentation methods applied to the image domain can randomly corrupt the image information of the training images. For fine-grained classification tasks, information with classification features exists only in a small region of the image. Data augmentation methods applied to the image domain are likely to corrupt this information, thus confusing the network during training.

[0050] Feature-dimensional data augmentation methods avoid the aforementioned problems by directly augmenting data in the image's feature space. However, current feature-dimensional data augmentation methods estimate a set of possible semantic augmentation directions for each class in classification problems. For fine-grained classification problems, the intra-class variance of training samples is large, while the inter-class variance is small. Using a uniform set of semantic directions for data augmentation for each class can cause samples to be mistakenly placed into the feature space of other samples after augmentation.

[0051] To avoid the above problems, in this embodiment, the neural network classification model Θ f A feature extractor extracts the features a of an image. i The semantic direction prediction network model utilizes the parameters at the current time (time t). For the image feature a i Predict its meaningful semantic direction Σ i In this embodiment, a set of possible semantic enhancement directions are predicted for each image, which can prevent the enhanced image from accidentally entering the feature space of other images.

[0052] In some embodiments, enhancing the features of the image along the first semantic direction to obtain enhanced features includes:

[0053] From normal distribution M random samples are performed to obtain the features after M data augmentations, and the normal distribution is... Based on feature a of the image i The mean is used to represent the semantic direction Σ. i The covariance matrix is ​​the variance, and M is a positive integer.

[0054] In this embodiment, feature a of the image i Based on its meaningful semantic direction Σ i From a normal distribution with its own mean as the mean and the covariance matrix representing the semantic direction as the variance. The data is augmented M times through random sampling, as shown in formula (1).

[0055]

[0056] In some embodiments, after obtaining the classification result of the image, the method further includes:

[0057] Calculate the loss of the neural network classification model on the training data;

[0058] The parameters of the neural network classification model are updated for the first time using the loss on the training data.

[0059] Specifically, the features enhanced by M data iterations can be input into the classifier of a neural network classification model, and the cross-entropy loss l can be calculated. M As shown in formula (2). After data augmentation, one sample becomes M samples. Calculate the cross-entropy loss of the M samples, and then take the average as the cross-entropy loss l. M .

[0060]

[0061] In formula (2), C is the number of categories in the classification problem, [w1, w2, ..., w C ] T and [b1,b2,…,b C ] T These are the learnable parameters of the linear classifier in a neural network classification model.

[0062] When M approaches infinity, the above cross-entropy loss can be written in the form of expectation, as shown in Equation (3).

[0063]

[0064] Using Jensen's inequality, an upper bound for the loss in the above expected form can be obtained, as shown in formula (4).

[0065]

[0066] The loss L on the training data can be calculated using formula (4). train The aforementioned upper bound is used as the loss on the training data.

[0067] Using the loss L on the training data train The parameters of the neural network classification model can be pseudo-updated (i.e., the first update), as shown in formula (5).

[0068]

[0069] Where, α t It is the learning rate of the neural network classification model at the current time (time t).

[0070] In some embodiments, after performing a first update on the parameters of the neural network classification model using the loss on the training data, the method further includes:

[0071] The cross-entropy loss of the metadata of the semantic orientation prediction network model is calculated using the updated parameters of the neural network classification model.

[0072] The semantic direction prediction network model is updated based on the cross-entropy loss and the learning rate of the semantic direction prediction network at the current time step.

[0073] Specifically, in this embodiment, the semantic direction prediction network can be based on the parameters of the updated neural network classification model. Calculate the cross-entropy loss of metadata And based on cross-entropy loss The parameters of the semantic direction prediction network model are updated as shown in Equation (6).

[0074]

[0075] Where, β t It is the learning rate of the semantic direction prediction network at the current moment.

[0076] In some embodiments, after updating the semantic direction prediction network model based on the cross-entropy loss and the current learning rate of the semantic direction prediction network, the method further includes:

[0077] The features of the image are input into the updated semantic direction prediction network model to obtain the second semantic direction of the image.

[0078] The loss of the neural network classification model on the training data is calculated based on the second semantic direction;

[0079] The parameters of the neural network classification model are updated a second time using the loss on the training data.

[0080] After updating the parameters of the semantic orientation prediction network model, the parameters of the neural network classification model can be further updated, and the updated semantic orientation prediction network can be used. For the feature a of the sample i The meaningful semantic direction is predicted, and the loss L on the training data can be calculated using formula (4) based on the newly predicted semantic direction. train Then utilize the loss L from the training data train The parameters of the neural network classification model are further updated, which is the real update (i.e., the second update), as shown in formula (7).

[0081]

[0082] This embodiment can repeat the above process. After further updating the parameters of the neural network classification model, the updated neural network classification model is used to extract features from the image (i.e., the sample), and the semantic direction prediction network is updated. Then, the updated semantic direction prediction network L is used... trainThe parameters of the neural network classification model are further updated. By continuously updating the neural network classification model and the semantic direction prediction network, the classification accuracy of the neural network classification model and the prediction accuracy of the semantic direction prediction network can be improved.

[0083] This invention also provides an image processing apparatus, such as... Figure 2 As shown, it includes:

[0084] The first processing module 21 is used to input the image to be processed into the feature extractor of the neural network classification model to obtain the features of the image;

[0085] The second processing module 22 is used to input the features of the image into the semantic direction prediction network model to obtain the first semantic direction of the image that has meaning.

[0086] Enhancement module 23 is used to enhance the features of the image along the first semantic direction to obtain enhanced features;

[0087] The classification module 24 is used to input the enhanced features into the classifier of the neural network classification model to obtain the classification result of the image.

[0088] In this embodiment, the features of the image to be processed are obtained, and these features are enhanced along a first semantic direction to obtain enhanced features. The image is then classified based on these enhanced features. This embodiment performs data augmentation in the feature space, which avoids the destruction of classification information when performing data augmentation in the image domain. This embodiment can be applied to various fine-grained image classification scenarios and can significantly improve the classification performance of various mainstream neural network models.

[0089] Data augmentation methods applied to the image domain can randomly corrupt the image information of the training images. For fine-grained classification tasks, information with classification features exists only in a small region of the image. Data augmentation methods applied to the image domain are likely to corrupt this information, thus confusing the network during training.

[0090] Feature-dimensional data augmentation methods avoid the aforementioned problems by directly augmenting data in the image's feature space. However, current feature-dimensional data augmentation methods estimate a set of possible semantic augmentation directions for each class in classification problems. For fine-grained classification problems, the intra-class variance of training samples is large, while the inter-class variance is small. Using a uniform set of semantic directions for data augmentation for each class can cause samples to be mistakenly placed into the feature space of other samples after augmentation.

[0091] In some embodiments, the enhancement module 23 is specifically used to obtain data from a normal distribution. M random samples are performed to obtain the features after M data augmentations, and the normal distribution is... Based on feature a of the image i The mean is used to represent the semantic direction Σ. i The covariance matrix is ​​the variance, and M is a positive integer.

[0092] In some embodiments, the apparatus further includes:

[0093] The first update module is used to calculate the loss on the training data of the neural network classification model; and to update the parameters of the neural network classification model using the loss on the training data.

[0094] In some embodiments, the apparatus further includes:

[0095] The second update module is used to calculate the cross-entropy loss of the metadata of the semantic direction prediction network model using the updated parameters of the neural network classification model; and to update the semantic direction prediction network model according to the cross-entropy loss and the learning rate of the semantic direction prediction network at the current time.

[0096] In some embodiments, the apparatus further includes:

[0097] The third update module is used to input the features of the image into the updated semantic direction prediction network model to obtain the second semantic direction of the image; calculate the loss on the training data of the neural network classification model based on the second semantic direction; and use the loss on the training data to perform a second update on the parameters of the neural network classification model.

[0098] This invention also provides an image processing apparatus, such as... Figure 3 As shown, it includes a memory 31, a processor 32, and a computer program stored in the memory 31 and executable on the processor 32; when the processor 32 executes the program, it implements the image processing method described above, which will not be repeated here.

[0099] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps in the image processing method described above.

[0100] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage, or any other non-transferable medium that can be used to store information accessible to the computer-readable terminal device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0101] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A picture processing method, characterized by, The method comprises: inputting a picture to be processed into a feature extractor of a neural network classification model to obtain features of the picture; inputting the features of the picture into a semantic direction prediction network model to obtain a first semantic direction of the picture having meaning; enhancing the features of the picture along the first semantic direction to obtain enhanced features; inputting the enhanced features into a classifier of the neural network classification model to obtain a classification result of the picture.

2. The picture processing method of claim 1, wherein, The method further comprises: M times of random sampling is performed from a normal distribution to obtain M times of data enhanced features, the normal distribution with the feature a i of the picture as the mean value, with the covariance matrix representing the semantic direction Σ i as the variance, and M is a positive integer.

3. The picture processing method of claim 1, wherein, after obtaining the classification result of the picture, the method further comprises: calculating a loss on training data of the neural network classification model; updating parameters of the neural network classification model using the loss on the training data.

4. The picture processing method of claim 3, wherein, The method further comprises: after updating the parameters of the neural network classification model, the method further comprises: calculating a cross-entropy loss of metadata of the semantic direction prediction network model using the updated parameters of the neural network classification model; 5. The picture processing method of claim 4, wherein, updating the semantic direction prediction network model according to the cross-entropy loss and a learning rate of the semantic direction prediction network at the current moment. The method further comprises: after updating the semantic direction prediction network model, the method further comprises: inputting the features of the picture into the updated semantic direction prediction network model to obtain a second semantic direction of the picture; 6. An image processing apparatus, characterized by comprising: calculating a loss on training data of the neural network classification model according to the second semantic direction; updating parameters of the neural network classification model using the loss on the training data. The method comprises: a first processing module configured to input a picture to be processed into a feature extractor of a neural network classification model to obtain features of the picture; a second processing module configured to input the features of the picture into a semantic direction prediction network model to obtain a first semantic direction of the picture having meaning; 7. The picture processing apparatus of claim 6, wherein, The enhancement module is specifically used for M times of random sampling from a normal distribution to obtain M times of features after data enhancement, the normal distribution with the feature a i of the picture as a mean value, with a covariance matrix representing the semantic direction Σ i as a variance, and M being a positive integer.

8. The picture processing apparatus of claim 6, wherein, an enhancement module configured to enhance the features of the picture along the first semantic direction to obtain enhanced features; a classification module configured to input the enhanced features into a classifier of the neural network classification model to obtain a classification result of the picture.

9. The picture processing apparatus of claim 8, wherein, The apparatus further comprises: a first updating module configured to calculate a loss on training data of the neural network classification model and update parameters of the neural network classification model using the loss on the training data.

10. The picture processing apparatus of claim 9, wherein, The apparatus further comprises: a second updating module configured to calculate a cross-entropy loss of metadata of the semantic direction prediction network model using the updated parameters of the neural network classification model and update the semantic direction prediction network model according to the cross-entropy loss and a learning rate of the semantic direction prediction network at the current moment. The apparatus further comprises: A third updating module configured to input a feature of the picture into the updated semantic direction prediction network model to obtain a second semantic direction of the picture, calculate a loss on training data of the neural network classification model according to the second semantic direction, and perform a second update on parameters of the neural network classification model by using the loss on the training data.

11. An image processing apparatus comprising a memory, a processor, and a computer program stored on the memory and executable on the processor; characterized in that, The processor implements the picture processing method of any one of claims 1-5 when executing the program.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the steps in the picture processing method of any one of claims 1-5.

Images