Deep learning model training data enhancement method

Abstract

The invention relates to the technical field of deep learning, and discloses a deep learning model training data enhancement method, which comprises the following steps: data preprocessing: preprocessing original training data, including image size adjustment, normalization and other operations, to ensure the consistency and stability of input data; gAN data enhancement: using a pre-trained generative adversarial network to generate an image sample similar to original data distribution; enhancing neural style conversion data: combining the content of the original image with images of different styles by using a neural style conversion technology to generate an image sample with a new style; and mixed data enhancement: mixing the image generated by the GAN and the image generated by the neural style conversion to form a final enhanced data set. The deep learning model training data enhancement method aims to combine the generation capability of the GAN and the style migration characteristic of neural style conversion, and can generate enhanced data with diversity and richness.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method for enhancing deep learning model training data. Background Art

[0002] In the training of deep learning models, the quality and diversity of data play a vital role. They are directly related to the final performance of the model. High-quality data means less noise in the data and accurate annotations, which can guide the model to learn more accurate feature representations. The diversity of data ensures that the model can generalize to various scenarios and situations, avoid overfitting, and improve the robustness and generalization ability of the model. If the data quality is not high, the model may learn noise, resulting in performance degradation; if the data lacks diversity, the model may only be able to handle specific situations and cannot adapt to new, unseen data. Therefore, when preparing training data, special attention should be paid to the quality and diversity of the data. Through data cleaning, enhancement, expansion and other means, the quality of the data can be continuously improved and the diversity of the data can be enriched, thereby providing a solid foundation for the training of deep learning models.

[0003] Although current data augmentation technology has alleviated the problem of insufficient data in the field of deep learning to a certain extent, it still faces some significant defects in its application. Specifically, some image augmentation methods, such as excessive rotation, distortion, or adding irrelevant textures, may introduce unnatural image features. These features not only fail to reflect the diversity of the real world, but may mislead the model to learn wrong patterns, thereby reducing the accuracy and reliability of the model. In addition, the difference between the distribution of data after data augmentation and the original data is also a major challenge. If there is a significant difference in distribution between the enhanced data and the original data, the model may find it difficult to extract useful features from the enhanced data, and may even cause the model's performance to decline in real scenarios. This distribution difference may be due to improper application of augmentation methods, such as excessive or insufficient augmentation, and improper selection of augmentation parameters. Summary of the invention

[0004] The purpose of the present invention is to combine the generation ability of GAN and the style transfer characteristics of neural style conversion to generate enhanced data with diversity and richness, and propose a deep learning model training data enhancement method.

[0005] The technical solution of the present invention to solve the above technical problems is as follows:

[0006] A method for enhancing data for deep learning model training, the method comprising the following steps:

[0007] S10, data preprocessing: preprocess the original training data, including image resizing, normalization and other operations to ensure the consistency and stability of the input data;

[0008] S20, GAN data enhancement: Use a pre-trained generative adversarial network to generate image samples similar to the original data distribution;

[0009] S30, Neural Style Transfer Data Enhancement: Using neural style transfer technology, the content of the original image is combined with images of different styles to generate image samples with new styles;

[0010] S40, hybrid data enhancement: Mix the images generated by GAN and the images generated by neural style transfer to form the final enhanced dataset;

[0011] S50, Model training: Use the enhanced dataset to train the deep learning model.

[0012] Based on the above technical solution, the present invention can also be improved as follows.

[0013] Furthermore, in the S10 data preprocessing step, the Min-Max normalization method is used, which scales the pixel values ​​of the image to the range of [0,1]. The specific formula is as follows:

[0014]

[0015] Among them, I is the pixel value of the original image, I min is the minimum value of all pixel values ​​in the image, I max is the maximum value of all pixel values ​​in the image, I normalliced is the normalized image pixel value. In addition, to ensure data consistency, the image should be resized to meet the input requirements of the deep learning model.

[0016] Furthermore, in step S20, a pre-trained deep convolutional generative adversarial network model is used to generate images. The adversarial network model uses convolutional neural networks in both the generator and the discriminator to generate higher quality images. In the specific steps, the noise vector z i Sampling from the standard normal distribution, the generator G uses a multi-layer deconvolution network to map the noise vector to the image space and generate a new image I fale =G(z i ), the discriminator D uses a multi-layer convolutional network to extract image features and outputs a scalar value to represent the probability that the image is real. Through continuous iterative training, the generator G can generate more and more realistic images with higher diversity and details.

[0017] Furthermore, in step S30, the VGG19 network is used as a pre-trained convolutional neural network to extract the content features of the content image and the style features of the style image, and the content features and the style features are combined by the Adam optimizer to generate a new stylized image I style ,The optimization objective function includes two parts: content loss and style loss. By adjusting the weights of the two, different stylized images can be generated.

[0018] Furthermore, in step S40, the mixing strategy not only includes random selection and proportional mixing, but also introduces a mixing method based on image similarity. This method first calculates the similarity between each pair of images using cosine similarity or Euclidean distance, and then mixes them according to the similarity. Images with higher similarity are easier to fuse, thereby generating a more natural enhanced image. In addition, mixing can also be performed according to the label or category of the image to generate an enhanced image with specific category characteristics.

[0019] Furthermore, in step S50, when the enhanced data set is used to train the deep learning model, an early stopping strategy is introduced to prevent the model from overfitting. The early stopping strategy stops training when the performance on the validation set begins to decline, thereby retaining the best model. In addition, a learning rate decay strategy can be used to dynamically adjust the learning rate to accelerate the convergence of the model and improve the performance of the model. During the training process, the data enhancement strategy can also be dynamically adjusted according to the feedback of the model, such as increasing or decreasing the proportion of a certain data enhancement, to further improve the generalization ability of the model.

[0020] Furthermore, the GAN model uses a deep convolutional generative adversarial network structure with residual connections. The adversarial network introduces residual connections based on DCGAN to alleviate the gradient vanishing problem in the deep network, thereby improving the quality and stability of the generated image. In the adversarial network, both the generator G and the discriminator D use multi-layer convolutional networks, and residual connections are added between each layer. The input of the generator G is a noise vector z i , after multi-layer deconvolution network and residual connection, the generated image I is output fake , the input of the discriminator D is the real image I real Or the generated image I falke , after passing through multiple layers of convolutional networks and residual connections, a scalar value is output to represent the probability that the image is real.

[0021] Furthermore, in step S40, the VGG19 network is used as a pre-trained convolutional neural network to extract the content features of the content image and the style features of the style image. In the neural style transfer, the content loss function L contentThe mean square error is used to measure the difference between the content image and the generated image in a certain layer of the VGG19 network feature map; the style loss function L style The Gram matrix is ​​used to measure the statistical difference between the style image and the generated image in the multi-layer feature map of the VGG19 network, and the objective function L = αLcontent + βL is optimized. style , where α and β are weight coefficients, which can generate images with specified style and content.

[0022] Furthermore, in order to further improve the effect of hybrid data enhancement in step S40, image fusion technology can be introduced. The image fusion technology fuses the information of multiple images together to generate a new image. The information of multiple images is fused together using methods such as weighted averaging and Laplace pyramid fusion, thereby generating a more natural and rich enhanced image.

[0023] Furthermore, during the training process, strategies such as learning rate decay and early stopping are used to prevent model overfitting. Regularization terms and dropout techniques are introduced to improve model performance. The regularization term limits the complexity of the model by adding a penalty term related to the model parameters to the loss function; the dropout technology randomly discards a part of the neuron connections during the training process to reduce the model's dependence on specific features.

[0024] Compared with the prior art, the technical solution of this application has the following beneficial technical effects:

[0025] In the data preprocessing stage, the present invention ensures the consistency and stability of the input data by performing operations such as image resizing and normalization on the original training data. It also introduces GAN data enhancement. GAN can learn the distribution of the original data and generate image samples similar to the distribution of the original data. This technical feature effectively solves the problem that unnatural image features may be introduced in the existing data enhancement methods. The image samples generated by GAN not only maintain the distribution characteristics similar to the original data, but also increase the diversity of the data, which helps to improve the generalization ability of the model. In addition, the method also uses neural style transfer data enhancement. The neural style transfer technology can combine the content of the original image with images of different styles to generate image samples with new styles. This step not only enriches the diversity of the data, but also makes the enhanced data set closer to the diversity of the real world. , which helps the model learn more generalized feature representations. In the hybrid data enhancement stage, this method mixes the images generated by GAN and the images generated by neural style conversion to form the final enhanced data set. This step further improves the diversity and richness of the data, and provides more comprehensive data support for the training of deep learning models. Finally, in the model training stage, the enhanced data set is used for training. Since the enhanced data set has been significantly improved in quality and diversity, the trained deep learning model will also be significantly improved in performance and generalization ability. In summary, by combining multiple steps such as data preprocessing, GAN data enhancement, neural style conversion data enhancement and hybrid data enhancement, the defects in the existing data enhancement technology are effectively solved, the quality and diversity of the data are improved, and the training effect of the deep learning model is optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a method flow chart of a deep learning model training data enhancement method of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] A deep learning model training data enhancement method of the present invention comprises the following steps:

[0029] S10, data preprocessing: preprocess the original training data, including image resizing, normalization and other operations to ensure the consistency and stability of the input data;

[0030] S20, GAN data enhancement: Use a pre-trained generative adversarial network to generate image samples similar to the original data distribution;

[0031] S30, Neural Style Transfer Data Enhancement: Using neural style transfer technology, the content of the original image is combined with images of different styles to generate image samples with new styles;

[0032] S40, hybrid data enhancement: Mix the images generated by GAN and the images generated by neural style transfer to form the final enhanced dataset;

[0033] S50, Model training: Use the enhanced dataset to train the deep learning model.

[0034] In a preferred embodiment, the present invention can be further configured as follows: In the step of S10 data preprocessing, a Min-Max normalization method is used, which scales the pixel values ​​of the image to the range of [0, 1]. The specific formula is as follows:

[0035]

[0036] Among them, I is the pixel value of the original image, I min is the minimum value of all pixel values ​​in the image, I max is the maximum value of all pixel values ​​in the image, I normalliced is the normalized image pixel value. In addition, to ensure data consistency, the image should be resized to meet the input requirements of the deep learning model. In addition to normalization, the image is also resized. Since the input layer of a deep learning model usually requires an image of a fixed size, the original image needs to be resized to a size that meets the model input requirements. This step can be achieved by scaling, cropping, or padding. The specific method depends on the characteristics of the image and the requirements of the model. By resizing, the consistency of the input data is ensured, providing a stable data foundation for subsequent enhancement and training steps.

[0037] The application of the Min-Max normalization method eliminates the brightness differences between different images, making it easier for the model to learn the features in the image, improving the accuracy and generalization ability of the model. At the same time, the image resizing step ensures the consistency of the input data, avoiding model training failure or performance degradation caused by inconsistent image sizes.

[0038] In a preferred embodiment, the present invention can be further configured as follows: in step S20, a pre-trained deep convolutional generative adversarial network model is used to generate an image, and the adversarial network model uses convolutional neural networks in both the generator and the discriminator to generate a higher quality image. In the specific steps, the noise vector z iSampling from the standard normal distribution, the generator G uses a multi-layer deconvolution network to map the noise vector to the image space and generate a new image I fale =G(z i ), the discriminator D uses a multi-layer convolutional network to extract image features and outputs a scalar value to represent the probability that the image is real. Through continuous iterative training, the generator G can generate more and more realistic images, and the generated images have higher diversity and details. The specific implementation method of using the pre-trained deep convolutional generative adversarial network (DCGAN) model for image generation is detailed. First, regarding the generation of noise vectors, it is clearly pointed out that they come from the standard normal distribution. This is to ensure the randomness and diversity of noise vectors and provide rich input information for the generator. In the design of the generator, a multi-layer deconvolution network (also known as a transposed convolutional network) is used. These network layers gradually map low-dimensional noise vectors to high-dimensional image space through layer-by-layer upsampling and convolution operations, and finally generate new images. In this process, technologies such as batch normalization and ReLU activation function may also be introduced to enhance the stability and nonlinear expression ability of the network. In the design of the discriminator, a multi-layer convolutional network structure is also used. These network layers extract feature information in the image through layer-by-layer downsampling and convolution operations. The final output of the discriminator is a scalar value, which indicates the probability that the input image is a real image. In order to improve the discriminative ability of the discriminator, a leaky ReLU activation function may be introduced after the convolution layer, and a sigmoid function may be used in the output layer to limit the output value to the range [0,1]. In addition, the loss function design during the training process will be described in detail, including the loss function of the generator (usually based on the probability that the generated image is misclassified as a real image by the discriminator) and the loss function of the discriminator (based on the probability that the real image and the generated image are correctly classified). Through continuous iterative training, the generator G can gradually learn how to generate more and more realistic images while keeping the generated images highly diverse and rich in details.

[0039] By introducing a pre-trained deep convolutional generative adversarial network (DCGAN) model, the quality and efficiency of image generation are significantly improved. Specifically, the use of a multi-layer deconvolutional network as a generator enables the generator to capture more complex image features, thereby generating more realistic and diverse images. At the same time, the multi-layer convolutional network as a discriminator can more effectively extract feature information from the image and improve the discriminative ability of the discriminator. This adversarial training between the generator and the discriminator enables the entire model to be continuously optimized during the iteration process, and the quality of the images generated by the generator is continuously improved, while maintaining a high degree of diversity and rich details.

[0040] In a preferred embodiment, the present invention can be further configured as follows: in step S30, the VGG19 network is used as a pre-trained convolutional neural network to extract the content features of the content image and the style features of the style image, and the content features and the style features are combined by the Adam optimizer to generate a new stylized image I style , the optimization objective function includes two parts, content loss and style loss. Different stylized images can be generated by adjusting the weights of the two. In step S30, the specific process of using the VGG19 network as a pre-trained convolutional neural network to extract the content features of the content image and the style features of the style image is elaborated in detail, and the method of combining these features through the Adam optimizer to generate a new stylized image is further clarified. First, the first few layers (usually convolutional layers) of the VGG19 network are selected to extract the content features of the content image. These layers can capture low-level and mid-level visual features in the image, such as edges, textures, and local structures. At the same time, the higher layers of the VGG19 network (usually convolutional layers before fully connected layers) are selected to extract the style features of the style image. These layers can capture the global statistical information of the image, such as color distribution, texture patterns, etc. Next, an optimization objective function is defined, which includes two parts, content loss and style loss. The content loss measures the difference between the generated image and the content image in content features, while the style loss measures the difference between the generated image and the style image in style features. In order to balance these two parts of loss, a weight parameter is introduced. By adjusting this parameter, the degree to which the generated image incorporates different style features of the style image while maintaining the main content of the content image can be controlled. During the optimization process, the Adam optimizer is used to iteratively update the pixel values ​​of the generated image to minimize the optimization objective function. The Adam optimizer has the characteristics of adaptive learning rate adjustment, which can speed up training and improve training stability. Through continuous iterative optimization, stylized images with different style features can be generated.

[0041] By using the VGG19 network as a pre-trained convolutional neural network to extract the features of content images and style images, and combining it with the Adam optimizer to generate new stylized images, efficient and high-quality image style transfer is achieved. First of all, as a classic convolutional neural network structure, the VGG19 network has a strong feature extraction capability and can capture rich information in the image. By selecting features from different layers for extraction, the performance of the generated image in terms of content and style can be flexibly controlled. Secondly, the use of the Adam optimizer improves the efficiency and stability of the optimization process, which significantly improves the quality of the generated images. Finally, by adjusting the weight parameters of the content loss and style loss, stylized images with different style features can be generated to meet the diverse needs of users.

[0042] In a preferred embodiment, the present invention can be further configured as follows: in step S40, the mixing strategy not only includes random selection and proportional mixing, but also introduces a mixing method based on image similarity. The method first calculates the similarity between each pair of images, using cosine similarity or Euclidean distance, and then mixes according to the similarity. Images with higher similarity are easier to merge, thereby generating a more natural enhanced image. In addition, the image can be mixed according to the label or category of the image to generate an enhanced image with specific category features. In step S40, the mixing strategy is further refined, not only retaining the original random selection and proportional mixing methods, but also introducing a mixing method based on image similarity and a mixing method based on image labels or categories. For the mixing method based on image similarity, a similarity calculation function is first defined, and the function can select cosine similarity or Euclidean distance as a calculation standard. Cosine similarity measures the similarity of two vectors by calculating the cosine value of the angle between them, and is suitable for measuring the directional similarity between image feature vectors; while Euclidean distance measures the difference between two vectors by calculating the straight-line distance between them, and is suitable for measuring the absolute distance between image feature vectors. After calculating the similarity between each pair of images, images are mixed according to the similarity. Specifically, image pairs with higher similarity are easier to fuse because they have higher similarity in content or style, which makes the mixed image more natural and coherent visually. To achieve this, weighted averaging or other fusion algorithms can be used to adjust the weight of each image in the mixed image according to the similarity. In addition, in order to increase the diversity of mixed images, a certain degree of randomness can be introduced, such as randomly selecting a similarity threshold or randomly adjusting the fusion weight. In addition to the mixing method based on image similarity, a mixing method based on image labels or categories is also proposed. This method first groups images according to their labels or category information, and then mixes between images in the same group. The advantage of this is that it can generate enhanced images with specific category characteristics, such as images of animal categories may have more obvious texture and morphological features, while images of landscape categories may have richer colors and composition elements. In this way, more targeted enhancement effects can be generated for images of different categories, improving the practicality and application value of image enhancement technology.

[0043] By introducing a hybrid method based on image similarity and image labels or categories, the effect and flexibility of image enhancement are significantly improved. The hybrid method based on image similarity can intelligently fuse images according to the content or style similarity between images to generate more natural and coherent enhanced images, avoiding image breakage or splicing traces that may occur in traditional hybrid methods. At the same time, this method can also adjust the fusion weight according to the similarity to achieve a more refined image mixing effect. The hybrid method based on image labels or categories further improves the pertinence and practicality of image enhancement. By grouping and mixing images according to their labels or categories, enhancement effects with specific characteristics can be generated for images of different categories to meet the diverse needs of users. In addition, this method can also promote the promotion and application of image enhancement technology in specific fields or application scenarios, such as medical image analysis and art restoration.

[0044] In a preferred embodiment, the present invention can be further configured as follows: in step S50, when using the enhanced data set to train the deep learning model, an early stopping strategy is also introduced to prevent the model from overfitting. When the performance of the early stopping strategy on the validation set begins to decline, the training is stopped, thereby retaining the best model. In addition, a learning rate decay strategy can be used to dynamically adjust the learning rate to accelerate the convergence of the model and improve the performance of the model. During the training process, the data enhancement strategy can also be dynamically adjusted according to the feedback of the model, such as increasing or decreasing the proportion of a certain data enhancement to further improve the generalization ability of the model. When using the enhanced data set to train the deep learning model, how to introduce an early stopping strategy, a learning rate decay strategy, and a dynamic adjustment of the data enhancement strategy to optimize the training process. First, with respect to the early stopping strategy, a validation set is set, which is independent of the training set and is used to monitor the performance of the model during the training process. In the early stage of training, as the number of iterations increases, the performance of the model on the validation set usually gradually improves. However, when the model begins to overfit, its performance on the validation set will no longer improve, but may begin to decline. At this point, the early stopping strategy will stop training immediately and retain the model parameters that perform best on the validation set, thereby effectively preventing the model from overfitting. Secondly, a learning rate decay strategy is introduced to dynamically adjust the learning rate. The learning rate is a key hyperparameter in deep learning model training, which determines the step size of model parameter updates. In the early stage of training, a larger learning rate can accelerate the convergence of the model. However, as the training progresses, the model gradually approaches the optimal solution. At this time, an excessively large learning rate may cause the model to oscillate near the optimal solution and fail to converge stably. Therefore, a learning rate decay strategy is adopted to dynamically adjust the learning rate according to the training progress or the performance on the validation set. For example, a fixed decay rate can be set and the learning rate can be multiplied by the decay rate after each epoch; or the learning rate can be reduced when the performance no longer improves according to the performance changes on the validation set. In this way, the model can converge to the optimal solution more stably during the training process. Finally, a method for dynamically adjusting the data augmentation strategy based on model feedback is proposed. During the training process, the performance of the model on the validation set and the classification accuracy of each category of data are continuously monitored. If it is found that the model has a low classification accuracy for a certain type of data, the enhancement ratio of this type of data can be appropriately increased to improve the model's ability to identify this type of data. On the contrary, if the model has a high classification accuracy for a certain type of data, the enhancement ratio of this type of data can be appropriately reduced to avoid excessive enhancement leading to a decrease in model performance. In this way, the data enhancement strategy can be dynamically adjusted according to the actual situation of the model to further improve the generalization ability of the model.

[0045] By introducing the early stopping strategy, the learning rate decay strategy and the dynamic adjustment of the data enhancement strategy, the training process of the deep learning model is significantly optimized. The early stopping strategy effectively prevents the model from overfitting and improves the generalization ability of the model; the learning rate decay strategy accelerates the convergence speed of the model and makes the model converge to the optimal solution more stably; the dynamic adjustment of the data enhancement strategy flexibly adjusts the data enhancement strategy according to the actual situation of the model, further improving the performance and generalization ability of the model. In addition, the combined use of these strategies also brings additional beneficial effects. For example, the combined effect of the early stopping strategy and the learning rate decay strategy can enable the model to use computing resources more efficiently during the training process, reducing unnecessary training time and computing costs. At the same time, the dynamic adjustment of the data enhancement strategy can also help discover the possible category imbalance problem in the data set, and balance the contribution of each category of data by adjusting the data enhancement ratio, thereby improving the fairness and robustness of the model. In summary, it provides a more comprehensive and effective optimization method for the training of deep learning models, which helps to improve the performance and practical value of the model. .

[0046] In a preferred embodiment, the present invention can be further configured as follows: a deep convolutional generative adversarial network structure with residual connections is used in the GAN model. The adversarial network introduces residual connections on the basis of DCGAN to alleviate the gradient vanishing problem in the deep network, thereby improving the quality and stability of the generated image. In the adversarial network, both the generator G and the discriminator D use multi-layer convolutional networks, and residual connections are added between each layer. The input of the generator G is a noise vector z i , after multi-layer deconvolution network and residual connection, the generated image I is output fake , the input of the discriminator D is the real image I real Or the generated image I falke , after passing through multiple layers of convolutional networks and residual connections, a scalar value is output to represent the probability that the image is real.

[0047] In addition, to further improve the quality of generated images, the idea of ​​Progressive Generative Adversarial Network (ProgressiveGAN) can be introduced to gradually increase the number of layers of the generator and discriminator, and gradually generate images from low resolution to high resolution. Specifically, in the early stage of training, shallower generators and discriminators are used to generate low-resolution images; as the training progresses, the number of layers of the generator and discriminator is gradually increased to generate higher-resolution images. This method can gradually capture the detailed features of the image, thereby generating more realistic images. Based on the deep convolutional generative adversarial network structure with residual connections in the GAN model, the specific construction method of the generator and discriminator is further described in detail, and the idea of ​​Progressive Generative Adversarial Network (ProgressiveGAN) is introduced to optimize the image generation process. First, regarding the design of the generator, its input is a random noise vector, which is gradually upsampled through a multi-layer deconvolution network, and the resolution of the generated image is gradually improved. Between each layer of the deconvolution network, residual connections are added to alleviate the gradient vanishing problem in the deep network. These residual connections allow the network to directly pass the feature information of the previous layer to the next layer, thereby maintaining the integrity of the features and improving the quality and stability of the generated image. The final output of the generator is a high-resolution generated image. The design of the discriminator is opposite to that of the generator. Its input is a real image or a generated image, which is gradually downsampled through a multi-layer convolutional network to extract the features of the image. Similarly, residual connections are added between each layer of the convolutional network to enhance the feature extraction ability of the network. The final output of the discriminator is a scalar value, which represents the probability that the input image is a real image. In addition, in order to further improve the quality of the generated image, the idea of ​​progressive generative adversarial network (ProgressiveGAN) is introduced. In the early stage of training, shallower generators and discriminators are used to generate low-resolution images. As the training progresses, the number of layers of the generator and discriminator is gradually increased to generate higher-resolution images. This method can gradually capture the detailed features of the image and gradually generate realistic images from low resolution to high resolution. Specifically, a resolution growth plan can be preset, such as from 4x4 to 8x8, then to 16x16, and so on, until the target resolution is reached. At each resolution level, the generator and discriminator are fully trained until they reach a stable state at that resolution, and then the number of layers is increased to enter the next resolution level.

[0048] By introducing the deep convolutional generative adversarial network structure with residual connections and the idea of ​​progressive generative adversarial networks, the quality and stability of generated images are significantly improved. Residual connections effectively alleviate the gradient vanishing problem in deep networks, allowing the network to learn the features of images more deeply and generate more realistic images. At the same time, the idea of ​​progressive generative adversarial networks enables the generator to gradually capture the detailed features of the image, and gradually generate images from low resolution to high resolution, further improving the quality of generated images. It also has good flexibility and scalability. By adjusting the number of layers of the generator and discriminator and the number of residual connections, GAN models of different complexities can be customized according to actual needs. At the same time, the idea of ​​progressive generative adversarial networks also makes it possible to gradually increase the resolution of generated images as needed to meet the needs of different application scenarios.

[0049] In a preferred embodiment of the present invention, the following configuration can be further performed: in step S40, the VGG19 network is used as a pre-trained convolutional neural network to extract the content features of the content image and the style features of the style image. In the neural style conversion, the content loss function L content The mean square error is used to measure the difference between the content image and the generated image in a certain layer of the VGG19 network feature map; the style loss function L style The Gram matrix is ​​used to measure the statistical difference between the style image and the generated image in the multi-layer feature map of the VGG19 network, and the objective function L = αLcontent + βL is optimized. style , where α and β are weight coefficients, which can generate images with specified style and content.

[0050] In addition, in order to further improve the efficiency and effect of neural style transfer, the adaptive instance normalization (AdaIN) technology can also be introduced. AdaIN calculates the mean and standard deviation of the feature maps of the content image and the style image at a certain layer of the VGG19 network, and uses them to generate the corresponding layer of the image, thereby achieving style transfer. This method can retain the structural information of the content image while giving it the color and texture features of the style image. In step S40, the VGG19 network is used as a pre-trained convolutional neural network to extract the content features of the content image and the style features of the style image. In the process of neural style transfer, the calculation methods of the content loss function and the style loss function are defined in detail. The content loss function uses the mean square error (MSE) to measure the difference between the feature maps of the content image and the generated image at a certain layer of the VGG19 network (usually a higher layer, such as relu_4_1), ensuring that the generated image is consistent with the original content image in terms of content. The style loss function uses the Gram Matrix to measure the statistical difference between the feature maps of the style image and the generated image in multiple layers of the VGG19 network (usually multiple layers from lower layers to higher layers, such as relu_1_1, relu_2_1, relu_3_1, relu_4_1, relu_5_1). This difference is mainly reflected in texture and color, thereby ensuring that the generated image is similar in style to the original style image. In order to further improve the efficiency and effect of neural style transfer, the adaptive instance normalization (AdaIN) technology is introduced. The core idea of ​​the AdaIN technology is to use the mean and standard deviation of the feature maps of the content image and style image in a certain layer of the VGG19 network (usually a layer with richer feature representation, such as relu_4_1) to normalize the corresponding layer of the generated image. Specifically, first calculate the mean and standard deviation of the feature map of the content image and the mean and standard deviation of the feature map of the style image, and then normalize the feature map of the generated image so that its mean and standard deviation match the corresponding values ​​of the content image and style image, respectively. In this way, the generated image can effectively absorb the color and texture features of the style image while retaining the structural information of the content image. In actual operation, the AdaIN technology can be combined with the original content loss function and style loss function to jointly guide the optimization process of the generated image. Specifically, in each iteration, the VGG19 network is first used to extract the feature maps of the content image and the style image, and then the content loss and style loss are calculated. Then, the AdaIN technology is used to normalize the generated image, and finally the parameters of the generator are updated to minimize the objective function (the weighted sum of the content loss and the style loss). In this way, the generated image can be gradually optimized so that it has both the specified content and the specified style.

[0051] By introducing the adaptive instance normalization (AdaIN) technology, the efficiency and effect of neural style transfer are significantly improved. First, the AdaIN technology can effectively transfer the color and texture features of the style image to the generated image while retaining the structural information of the content image, thereby generating a more natural and realistic stylized image. Secondly, since the AdaIN technology directly normalizes the feature map, it avoids the complex iterative optimization process, and thus can significantly improve the speed of neural style transfer. In addition, by combining with the original content loss function and style loss function, it can more accurately capture the features of the content image and style image, thereby generating a stylized image that better meets user expectations.

[0052] In a preferred embodiment of the present invention, the present invention can be further configured as follows: in step S40, the mixing strategy not only includes random selection and proportional mixing, but also introduces a mixing method based on image similarity and category information. The mixing method based on image similarity first calculates the similarity between each pair of images, and then mixes according to the similarity. Images with higher similarity are easier to fuse, thereby generating a more natural enhanced image. The mixing method based on category information mixes according to the label or category of the image to generate an enhanced image with specific category features. This method can further improve the diversity and richness of the data set while retaining the category information of the image. In order to further improve the effect of hybrid data enhancement, image fusion technology can also be introduced. The image fusion technology fuses the information of multiple images together to generate a new image. The information of multiple images is fused together using methods such as weighted averaging and Laplace pyramid fusion, thereby generating a more natural and rich enhanced image. The mixing strategy is further refined and expanded, and not only includes traditional random selection and proportional mixing methods, but also introduces a mixing method based on image similarity and category information, as well as image fusion technology to generate a more natural and rich enhanced image. The hybrid method based on image similarity first calculates the similarity between each pair of images using a similarity measurement algorithm (such as cosine similarity, Euclidean distance, etc.). When calculating the similarity, a feature space of the image (such as color space, texture space, or depth feature space) can be selected for calculation. Then, based on the calculated similarity, select the image pair with higher similarity for hybridization. When hybridizing, weighted averaging, linear interpolation, and other methods can be used to fuse the information of the two images. Since images with higher similarity are closer in visual features, the hybrid image will be more natural visually and will not easily produce abrupt splicing effects. The hybrid method based on category information is hybridized according to the label or category of the image. First, the image needs to be annotated to obtain its category information. Then, when hybridizing, select images with the same or similar category labels for hybridization. This method can ensure that the hybrid images are consistent in category features, thereby generating enhanced images with specific category features. For example, in the vehicle recognition task, vehicle images of different angles and models but belonging to the same category can be selected for hybridization to generate more diverse vehicle image data. In order to further improve the effect of hybrid data enhancement, image fusion technology is also introduced. Image fusion technology fuses the information of multiple images together to generate a new image. In the fusion process, weighted averaging, Laplacian pyramid fusion, multi-scale fusion and other methods can be used. The weighted averaging method is simple and intuitive, but it may not be able to preserve the details of the image well. The Laplacian pyramid fusion method can well preserve the details and texture information of the image, but the computational complexity is high. The multi-scale fusion method combines image information of different scales to generate more natural and rich enhanced images.In practical applications, the appropriate fusion method can be selected according to specific needs.

[0053] By introducing a hybrid method based on image similarity and category information, as well as image fusion technology, the effect of hybrid data enhancement is significantly improved. The hybrid method based on image similarity can ensure that the hybrid image is more natural visually and is not prone to abrupt splicing effects; the hybrid method based on category information can generate enhanced images with specific category characteristics, thereby further improving the diversity and richness of the data set. At the same time, both methods retain the category information of the image, which helps to maintain the label consistency of the data set. The introduction of image fusion technology further enhances the ability of hybrid data enhancement. By fusing the information of multiple images, more natural and rich enhanced images can be generated. These images not only contain the feature information of the original image, but also fuse the information of other images, thereby improving the complexity and diversity of the image. This is of great significance for improving the generalization ability and robustness of deep learning models. In addition, image fusion technology can also provide more abundant tools and methods for image editing, image synthesis and other fields.

[0054] In a preferred embodiment, the present invention can be further configured as follows: during the training process, strategies such as learning rate decay and early stopping are adopted to prevent model overfitting, and regularization terms and dropout techniques are introduced to improve the performance of the model. The regularization term limits the complexity of the model by adding a penalty term related to the model parameters to the loss function; the dropout technology randomly discards a part of the neuron connections during the training process to reduce the model's dependence on specific features.

[0055] In addition, to further improve the efficiency and effect of model training, batch normalization and gradient clipping can also be introduced. Batch normalization accelerates the convergence of the model by normalizing each small batch of data; gradient clipping prevents the gradient explosion problem by limiting the norm of the gradient. These technologies can further improve the stability and efficiency of model training. In the process of training deep learning models, in order to effectively prevent model overfitting, improve model performance, and further improve training efficiency and effect, in addition to adopting strategies such as learning rate decay and early stopping, regularization terms and dropout technology are also introduced. On this basis, batch normalization and gradient clipping are introduced. Specifically, the regularization term limits the complexity of the model by adding a penalty term related to the model parameters to the loss function, thereby preventing the model from overfitting on the training data. This penalty term is usually proportional to the sum of squares or absolute values ​​of the model parameters. By adjusting the coefficient of the regularization term, the balance between model complexity and training error can be controlled. At the same time, the dropout technology is used to randomly discard a part of the neuron connections during the training process, which helps to reduce the model's dependence on specific features and enhance the model's generalization ability. In order to further improve the efficiency and effect of model training, batch normalization technology is introduced. Batch normalization normalizes each small batch of data so that each batch of data has the same mean and variance, thereby accelerating the convergence speed of the model and improving the stability of training. In addition, gradient clipping technology is introduced to prevent the gradient explosion problem by limiting the norm of the gradient. During the training process, if the norm of the gradient exceeds the preset threshold, it will be clipped to within the threshold, which can avoid the problem of unstable model update or training failure caused by excessive gradient.

[0056] By introducing regularization terms, dropout technology, batch normalization and gradient clipping, the training efficiency and effect of deep learning models are significantly improved. The combination of regularization terms and dropout technology effectively prevents the model from overfitting on the training data and improves the generalization ability of the model. The introduction of batch normalization technology accelerates the convergence speed of the model, improves the stability of training, and enables the model to achieve better performance in a shorter time. The use of gradient clipping technology effectively prevents the gradient explosion problem and avoids the instability and failure risk during model training. In addition, the combined use of these technologies further improves the flexibility and robustness of model training. In practical applications, the parameters and combination methods of these technologies can be flexibly adjusted according to the characteristics of specific tasks and data sets to achieve the best training effect and performance. Therefore, it provides an efficient, stable and easy-to-implement solution for the training of deep learning models, which helps to promote the widespread application and development of deep learning technology.

[0057] This deep learning model training data enhancement method aims to significantly improve the training efficiency and performance of deep learning models through a series of carefully designed steps and strategies. First, in the data preprocessing stage, the original training data is processed as necessary, including adjusting the image size to a uniform size and using the Min-Max normalization method to scale the image pixel values ​​to the range of [0,1] to ensure the consistency and stability of the input data. This step lays a solid foundation for subsequent data enhancement and model training.

[0058] Next, in the GAN data augmentation stage, a pre-trained deep convolutional generative adversarial network (GAN) model is used to generate image samples with a distribution similar to the original data. These generated images not only increase the diversity of the dataset, but also help improve the model's generalization ability for unseen data. In particular, a deep convolutional generative adversarial network structure with residual connections is adopted to alleviate the gradient vanishing problem in deep networks, and the idea of ​​progressive generative adversarial networks is introduced to gradually increase the number of layers of the generator and discriminator, and gradually generate images from low resolution to high resolution, thereby generating more realistic images.

[0059] In the data enhancement stage of neural style transfer, the VGG19 network is used as a pre-trained convolutional neural network to extract the content features of the content image and the style features of the style image. By optimizing the objective function, including the content loss and the style loss, and adjusting their weights, an image with a specified style and content can be generated. In order to further improve the efficiency and effect of neural style transfer, the adaptive instance normalization (AdaIN) technology is also introduced, which can retain the structural information of the content image while giving it the color and texture characteristics of the style image.

[0060] Then, in the hybrid data enhancement stage, the images generated by GAN and the images generated by neural style transfer are mixed to form the final enhanced dataset. The hybrid strategy includes not only random selection and proportional mixing, but also the introduction of hybrid methods based on image similarity and category information. The hybrid method based on image similarity can ensure that images with high similarity are easier to fuse, thereby generating more natural enhanced images; while the hybrid method based on category information can generate enhanced images with specific category features, further improving the diversity and richness of the dataset. In addition, image fusion technology is introduced to fuse the information of multiple images together to generate more natural and rich enhanced images.

[0061] Finally, in the model training phase, the enhanced dataset is used to train the deep learning model. In order to prevent the model from overfitting, strategies such as learning rate decay and early stopping are adopted, and regularization terms and dropout techniques are introduced to improve the performance of the model. The regularization term limits the complexity of the model by adding a penalty term related to the model parameters to the loss function; while the dropout technique randomly discards a part of the neuron connections during the training process to reduce the model's dependence on specific features. In addition, techniques such as batch normalization and gradient clipping are introduced to further improve the stability and efficiency of model training.

[0062] In summary, this deep learning model training data enhancement method effectively improves the training efficiency and performance of deep learning models through a series of carefully designed steps and strategies. By combining multiple data enhancement techniques and model training strategies, more diverse and rich training data can be generated, thereby training a more robust and generalized deep learning model.

[0063] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0064] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for enhancing data for deep learning model training, characterized in that: The method comprises the following steps: S10, data preprocessing: preprocess the original training data, including image resizing, normalization and other operations to ensure the consistency and stability of the input data; S20, GAN data enhancement: Use a pre-trained generative adversarial network to generate image samples similar to the original data distribution; S30, Neural Style Transfer Data Enhancement: Using neural style transfer technology, the content of the original image is combined with images of different styles to generate image samples with new styles; S40, hybrid data enhancement: Mix the images generated by GAN and the images generated by neural style transfer to form the final enhanced dataset; S50, Model training: Use the enhanced dataset to train the deep learning model.

2. A deep learning model training data enhancement method according to claim 1, characterized in that: In the S10 data preprocessing step, the Min-Max normalization method is used, which scales the pixel values ​​of the image to the range of [0,1]. The specific formula is as follows: Among them, I is the pixel value of the original image, I min is the minimum value of all pixel values ​​in the image, I max is the maximum value of all pixel values ​​in the image, I normalliced is the normalized image pixel value. In addition, to ensure data consistency, the image should be resized to meet the input requirements of the deep learning model.

3. A deep learning model training data enhancement method according to claim 1, characterized in that: In step S20, the pre-trained deep convolutional generative adversarial network model is used to generate images. The adversarial network model uses convolutional neural networks in both the generator and the discriminator to generate higher quality images. In the specific steps, the noise vector z i Sampling from the standard normal distribution, the generator G uses a multi-layer deconvolution network to map the noise vector to the image space and generate a new image I fale =G(z i ), the discriminator D uses a multi-layer convolutional network to extract image features and outputs a scalar value to represent the probability that the image is real. Through continuous iterative training, the generator G can generate more and more realistic images with higher diversity and details.

4. A deep learning model training data enhancement method according to claim 1, characterized in that: In step S30, the VGG19 network is used as a pre-trained convolutional neural network to extract the content features of the content image and the style features of the style image, and the content features and the style features are combined by the Adam optimizer to generate a new stylized image I style ,The optimization objective function includes two parts: content loss and style loss. By adjusting the weights of the two, different stylized images can be generated.

5. The method for enhancing deep learning model training data according to claim 1, characterized in that: In step S40, the mixing strategy not only includes random selection and proportional mixing, but also introduces a mixing method based on image similarity. This method first calculates the similarity between each pair of images using cosine similarity or Euclidean distance, and then mixes them according to the similarity. Images with higher similarity are easier to fuse, thereby generating a more natural enhanced image. In addition, mixing can also be performed according to the label or category of the image to generate an enhanced image with specific category characteristics.

6. A method for enhancing deep learning model training data according to claim 1, characterized in that: In step S50, when the enhanced data set is used to train the deep learning model, an early stopping strategy is introduced to prevent the model from overfitting. The early stopping strategy stops training when the performance on the validation set begins to decline, thereby retaining the best model. In addition, a learning rate decay strategy can be used to dynamically adjust the learning rate to accelerate the convergence of the model and improve the performance of the model. During the training process, the data enhancement strategy can also be dynamically adjusted according to the feedback of the model, such as increasing or decreasing the proportion of a certain data enhancement, to further improve the generalization ability of the model.

7. A method for enhancing deep learning model training data according to claim 3, characterized in that: The GAN model uses a deep convolutional generative adversarial network structure with residual connections. The adversarial network introduces residual connections based on DCGAN to alleviate the gradient vanishing problem in the deep network, thereby improving the quality and stability of the generated image. In the adversarial network, both the generator G and the discriminator D use multi-layer convolutional networks, and residual connections are added between each layer. The input of the generator G is a noise vector z i , after multi-layer deconvolution network and residual connection, the generated image I is output fake , the input of the discriminator D is the real image I real Or the generated image I falke , after passing through multiple layers of convolutional networks and residual connections, a scalar value is output to represent the probability that the image is real.

8. A method for enhancing deep learning model training data according to claim 4, characterized in that: In step S40, the VGG19 network is used as a pre-trained convolutional neural network to extract the content features of the content image and the style features of the style image. In the neural style conversion, the content loss function L content The mean square error is used to measure the difference between the content image and the generated image in a certain layer of the VGG19 network feature map; the style loss function L style The Gram matrix is ​​used to measure the statistical difference between the style image and the generated image in the multi-layer feature map of the VGG19 network, and the objective function L = αLcontent + βL is optimized. style , where α and β are weight coefficients, which can generate images with specified style and content.

9. A method for enhancing deep learning model training data according to claim 5, characterized in that: In order to further improve the effect of hybrid data enhancement in step S40, image fusion technology can also be introduced. The image fusion technology fuses the information of multiple images together to generate a new image. The information of multiple images is fused together using methods such as weighted averaging and Laplace pyramid fusion, thereby generating a more natural and rich enhanced image.

10. A method for enhancing deep learning model training data according to claim 6, characterized in that: During the training process, strategies such as learning rate decay and early stopping are used to prevent model overfitting. Regularization terms and dropout techniques are introduced to improve model performance. The regularization term limits the complexity of the model by adding a penalty term related to the model parameters to the loss function; the dropout technology randomly discards a part of the neuron connections during the training process to reduce the model's dependence on specific features.

Images