Data enhancement method, terminal and storage medium based on multi-environmental features
By extracting and utilizing time, lighting and weather features, establishing a feature library and using Encoder-Decoder technology, the problems of insufficient data and low model accuracy in real scene image processing in the existing technology are solved, and the model is improved under diversified conditions Performance.
Patent Information
- Application Number
- CN202111340841.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-12
- Publication Date
- 2024-12-31
- Estimated Expiration
- 2041-11-12
AI Technical Summary
The prior art has insufficient data and low model accuracy when processing images of real scenes, especially when simulating weather, time and lighting changes.
Using a data enhancement method based on multi-environmental features, a feature library is established by collecting and extracting time, lighting and weather features, and using Encoder-Decoder technology to generate more diverse image data, avoiding the problem of difficulty in labeling.
The model accuracy of image processing is improved, making the generated data closer to the actual situation, and the processing capability is enhanced under different times, lighting and weather conditions.
Smart Images
Figure CN114067131B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of image processing technology, and more specifically, to a data enhancement method, terminal and storage medium based on multiple environmental features. Background Art
[0002] With the continuous development of artificial intelligence, computer vision and hardware technology, video image processing technology has been widely used in intelligent systems. In recent years, with the promotion of deep learning, the demand for data, especially high-quality labeled data, has risen sharply, while the cost of data collection and annotation has greatly restricted the quantity and quality of data.
[0003] The significance of data enhancement is to reduce the cost of data collection and annotation as much as possible, so that the model can achieve better results based on the existing collected and labeled data, thereby enabling artificial intelligence to be more widely used in a larger range and solving the data bottleneck problem that currently exists in the application of artificial intelligence.
[0004] Among them, common data enhancement methods include supervised data enhancement and unsupervised data enhancement methods. Among them, supervised data enhancement includes those based on geometric transformation and color transformation. Geometric transformation refers to geometric transformation of images, including flipping, rotation, cropping, deformation, scaling and other operations. Geometric transformation operations do not change the content of the image itself. It may select a part of the image or redistribute the pixels. If the content of the image itself is to be changed, it belongs to the color transformation data enhancement. Common ones include noise, blur, color transformation, erasure, filling and so on. Unsupervised data enhancement learns the distribution of data through the model and randomly generates pictures that are consistent with the distribution of the training data set. The representative method is GAN. The above common data enhancement methods, whether geometric transformation, color transformation, or GAN generation, are poorly applied to weather, time, lighting and other real scenes, because geometric transformation and color transformation methods are difficult to simulate or restore the impact of weather, time and lighting on the target in real scenes, while GAN requires a large amount of labeled weather, time and lighting data to be added to the training. For example, when the sun shines on a person's face or the person's face is in a backlit environment, the model training effect is poor and the accuracy is greatly reduced because there is only a small amount of relevant data in the collected data set or the enhanced data set. Therefore, it is necessary to deal with such common problems specifically for real scenes. Summary of the invention
[0005] The purpose of the embodiments of the present application is to provide a data enhancement method, terminal and storage medium based on multiple environmental features to solve the technical problems of insufficient data and low model accuracy in the image processing process in real scenes in the prior art.
[0006] To achieve the above purpose, the technical solution adopted in this application is: to provide a data enhancement method based on multiple environmental features, comprising the following steps:
[0007] Collect a number of first target images with time, light and weather characteristics;
[0008] Input the first target image to the encoders of time, light and weather respectively, extract the features of time, light and weather respectively, and output the first feature map or vector of time, light and weather respectively;
[0009] Save a first feature map or vector of time, illumination and weather of the first target image to a feature library;
[0010] Input the first feature map or vector of time, light and weather of the first target image into the decoder of time, light and weather, and output it as a second target image;
[0011] By calculating the distance between the first target image and the second target image as the loss, the weights of the Encoder and Decoder and the first feature map or vector of time, light and weather in the feature library are returned to update;
[0012] Input the third target image to the updated encoder of time, light and weather, extract the features of time, light and weather, and output the second feature map or vector of time, light and weather;
[0013] Replace the second feature map or vector of time, light and weather with the first feature map or vector representing different time, light and weather in the updated feature library;
[0014] Output image data with different time, lighting and weather.
[0015] Preferably, the first target image is any image without label data of time, lighting and weather.
[0016] Preferably, the Encoder includes a time Encoder, a light Encoder and a weather Encoder, the time Encoder is used to extract time features of the first target image, the light Encoder is used to extract light features of the first target image, and the weather Encoder is used to extract weather features of the first target image.
[0017] Preferably, the feature library includes a time feature library, a lighting feature library and a weather feature library. The time feature library is used to maintain a first feature map or vector of the time of the first target image, the lighting feature library is used to maintain a first feature map or vector of the lighting of the first target image, and the weather feature library is used to maintain a first feature map or vector of the weather of the first target image.
[0018] Preferably, the data enhancement method based on multiple environmental features further comprises the steps of:
[0019] After calculating the distance between the first target image and the second target image as the loss, when the Loss is less than the rated value, the input of the first target image ends, the training of the Encoder and the Decoder ends, and the updating of the feature map or vector in the time, illumination, and weather feature library ends.
[0020] Preferably, the data enhancement method based on multiple environmental features further comprises the steps of:
[0021] After extracting the second feature map or vector, the second feature map or vector of time, illumination and weather of the third target image is saved in the feature library.
[0022] The present application also provides a terminal, which includes a processor, and the processor is used to run the data enhancement method based on multiple environmental features as described above.
[0023] The present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the facial attribute recognition method described above are implemented.
[0024] Compared with the prior art, the data enhancement method, terminal and storage medium based on multi-environment features provided by the present application fully utilize and extract the features of dimensions such as time, light and weather in the training images, and establish a feature library based on this. At the same time, due to the use of Encoder-Decoder technology, the core problem of the difficulty of time and light annotation in the training images is avoided, the diversity of images is greatly improved, and the accuracy of the model obtained by training with generated data is improved. By enhancing any input image under different time, light and weather conditions in a manner close to the actual situation. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0026] Figure 1 A schematic diagram of the three-dimensional structure of a data enhancement method based on multiple environmental features provided in an embodiment of the present application;
[0027] Figure 2 A schematic diagram of a terminal and a storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION
[0028] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0029] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.
[0030] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.
[0031] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.
[0032] Please also read Figure 1 Now, the data enhancement method based on multiple environmental features provided in the embodiment of the present application is described. The data enhancement method based on multiple environmental features includes:
[0033] Step S1, collecting a number of first target images with time, light and weather characteristics.
[0034] Step S2: Input the first target image into the encoders of time, light and weather respectively, extract the features of time, light and weather accordingly, and output the first feature map or vector of time, light and weather accordingly.
[0035] Step S3, saving the first feature map or vector of time, lighting and weather of the first target image to the feature library.
[0036] Step S4, input the first feature map or vector of time, light and weather of the first target image into the decoder of time, light and weather, and output it as a second target image.
[0037] Step S5, by calculating the distance between the first target image and the second target image as the loss, returning to update the weights of the Encoder and Decoder and the first feature map or vector of time, light and weather in the feature library.
[0038] Step S6, input the third target image to the updated encoder of time, light and weather, extract the features of time, light and weather accordingly, and output the second feature map or vector of time, light and weather accordingly.
[0039] Step S7, replacing the second feature graph or vector of time, light and weather with the first feature graph or vector representing different time, light and weather in the updated feature library.
[0040] Step S8, outputting picture data with different time, light and weather.
[0041] It is understandable that the first target image is any image without label data of time, light and weather. The main purpose of steps S1 to S5 is to train the Encoder and Decoder and update the feature library at the same time. Each time a first target image is input, the Encoder and Decoder can be trained once through steps S1 to S5, and the feature library can be updated once.
[0042] First, the Encoder is used to extract the feature maps or vectors corresponding to time, light and weather and store them in the feature library. Then, the feature maps or vectors output by the Encoder are used as the input of the Decoder in the next step to restore the image, and the weights are updated using the Decoder output and the loss of the initial input image. Finally, the required time, light and weather are extracted from the feature library to replace the target time, light and weather, generating more diverse image data.
[0043] It is worth mentioning that the Encoder is used to extract the time, lighting and weather features in the image, while the Decoder takes a certain feature (such as 10 o'clock in the morning) as input to output a picture with this corresponding feature; therefore, the Encoder is actually a stack of multiple ordinary convolutional layers Conv, and the Decoder is a stack of multiple ordinary deconvolution DeConv, which can be designed according to specific needs.
[0044] Compared with the prior art, the data enhancement method based on multi-environment features provided in the present application fully utilizes and extracts the features of dimensions such as time, lighting and weather in the training images, and establishes a feature library based on this. At the same time, due to the use of Encoder-Decoder technology, the core problem of the difficulty of labeling time and lighting in training images is avoided, and the diversity of images is greatly improved, so that the accuracy of the model obtained by training with generated data is improved. By enhancing any input image under different time, lighting and weather conditions in a manner close to the actual situation.
[0045] In another embodiment of the present application, the first target image is any image without label data of time, lighting and weather.
[0046] It is understandable that in order to ensure the diversity of feature maps or vectors in the feature library and avoid data imbalance, the first target image should be acquired randomly, which is conducive to improving model accuracy.
[0047] In another embodiment of the present application, the Encoder includes a time Encoder, a light Encoder and a weather Encoder, the time Encoder is used to extract the time characteristics of the first target image, the light Encoder is used to extract the light characteristics of the first target image, and the weather Encoder is used to extract the weather characteristics of the first target image.
[0048] It can be understood that the time, light and weather in the first target image can be extracted as independent factors as features through the time encoder, light encoder and weather encoder; the time encoder, light encoder and weather encoder can run simultaneously and independently of each other, thereby improving the running speed.
[0049] In another embodiment of the present application, the feature library includes a time feature library, a lighting feature library, and a weather feature library. The time feature library is used to maintain a first feature map or vector of the time of the first target image, the lighting feature library is used to maintain a first feature map or vector of the lighting of the first target image, and the weather feature library is used to maintain a first feature map or vector of the weather of the first target image.
[0050] It can be understood that the first feature graph or vector of time, illumination and weather of the first target image is independently stored in three independent feature libraries, that is, the time feature library, the illumination feature library and the weather feature library can be updated independently.
[0051] In another embodiment of the present application, the data enhancement method based on multiple environmental features further includes:
[0052] Step S5.1, when the Loss is less than the rated value, the first target image input is completed, the Encoder and Decoder training is completed, and the updating of the feature map or vector in the time, light and weather feature library is completed.
[0053] It is understandable that the purpose of updating the Encoder and Decoder is because initially, the weights of both are random. The Encoder is poor at extracting the time, lighting, and weather features of the picture, and the Decoder is poor at restoring the features to the picture. Therefore, training is required, that is, updating the weights of the Encoder and Decoder to achieve the best possible effect, that is, minimize the loss.
[0054] In another embodiment of the present application, the data enhancement method based on multiple environmental features further includes:
[0055] Step S6.1, save the second feature map or vector of time, lighting and weather of the third target image to the feature library.
[0056] It is understandable that by continuously updating the feature graphs or vectors in the feature library, the diversity of the feature library can be continuously improved. Even after the encoder and decoder are trained through steps S1 to S5, the feature library can be further enriched during use.
[0057] See also Figure 2 The present application also provides a terminal, which includes a processor, and the processor is used to run the data enhancement method based on multiple environmental features as described above.
[0058] It is understandable that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal, and various interfaces and lines are used to connect various parts of the entire terminal.
[0059] If the module / unit integrated in the terminal is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0060] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.
Claims
1. A data enhancement method based on multiple environment features, characterized in that: The following steps are involved: Collect a number of first target images with time, light and weather characteristics; Input the first target image to the encoders of time, light and weather respectively, extract the features of time, light and weather respectively, and output the first feature map or vector of time, light and weather respectively; Save a first feature map or vector of time, illumination and weather of the first target image to a feature library; Input the first feature map or vector of time, light and weather of the first target image into the decoder of time, light and weather, and output it as a second target image; By calculating the distance between the first target image and the second target image as the loss, the weights of the Encoder and Decoder and the first feature map or vector of time, light and weather in the feature library are returned to update; Input the third target image to the updated encoder of time, light and weather, extract the features of time, light and weather, and output the second feature map or vector of time, light and weather; Replace the second feature map or vector of time, light and weather with the first feature map or vector representing different time, light and weather in the updated feature library; Output image data with different time, lighting and weather.
2. The data enhancement method based on multiple environmental features as claimed in claim 1, characterized in that: The first target image is any image without label data of time, light and weather.
3. The data enhancement method based on multiple environmental features as claimed in claim 1, characterized in that: The Encoder includes a time Encoder, a light Encoder and a weather Encoder. The time Encoder is used to extract the time features of the first target image, the light Encoder is used to extract the light features of the first target image, and the weather Encoder is used to extract the weather features of the first target image.
4. The data enhancement method based on multiple environmental features as claimed in claim 3, characterized in that: The feature library includes a time feature library, a lighting feature library and a weather feature library. The time feature library is used to maintain a first feature map or vector of the time of the first target image, the lighting feature library is used to maintain a first feature map or vector of the lighting of the first target image, and the weather feature library is used to maintain a first feature map or vector of the weather of the first target image.
5. The data enhancement method based on multiple environmental features as claimed in claim 1, characterized in that: Also includes the steps: After calculating the distance between the first target image and the second target image as the loss, when the Loss is less than the rated value, the input of the first target image ends, the training of the Encoder and the Decoder ends, and the updating of the feature map or vector in the time, illumination, and weather feature library ends.
6. The data enhancement method based on multiple environmental features as claimed in claim 1, characterized in that: Also includes the steps: After extracting the second feature map or vector, the second feature map or vector of time, illumination and weather of the third target image is saved in the feature library.
7. A terminal, characterized in that: The terminal includes a processor, and the processor is used to run the data enhancement method based on multiple environmental features as described in any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the steps of the data enhancement method based on multiple environmental features as described in any one of claims 1 to 6 are implemented.