Image augmentation method and device, and storage medium
Images with different lighting conditions are generated through color gamut transformation and saturation transformation, and fusion is solved, which makes the existing data augmentation solution more space in the data set, and the rapid and effective data augmentation is achieved, and the training accuracy and generalization ability of deep learning models are improved.
Patent Information
- Application Number
- CN202311246297.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-25
- Publication Date
- 2025-03-28
AI Technical Summary
When the existing data augmentation scheme improves the richness of data sets and generalization capabilities, it will cause the image data set to occupy ROM space significantly, extend training time, and the solution based on adversarial generation networks will take a long time.
Images of different lighting conditions are generated through color gamut transformation and saturation transformation, and these images are fused into one image to generate target images that fuse all lighting conditions, so as to quickly and effectively expand data without increasing the capacity of the data set.
It realizes rapid and efficient data augmentation without increasing the capacity of the data set, improves the precision of deep learning training and model generalization capabilities, and extends the service life of the computer.
Smart Images
Figure CN119722501A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an image augmentation method, device and storage medium based on color gamut transformation and image fusion. Background Art
[0002] At present, digital image processing methods based on deep learning have shined in related fields such as image classification, target tracking, edge detection, image segmentation, etc., and are the mainstream solutions for handling the above tasks. However, most deep learning algorithms require the collection and annotation of large image data sets, which is very time-consuming and labor-intensive. For this reason, researchers have proposed the concept of data augmentation, which is to perform digital image processing operations such as random cropping and flipping on the collected images, so as to obtain more data sets using fewer actual captured images, improve the richness of the images contained in the data set, and reduce the occurrence of overfitting of deep learning models.
[0003] However, there are some common problems with current data augmentation solutions. For example, if data augmentation is used to improve the richness of the dataset and the generalization ability of the trained model, multiple operations need to be performed to obtain multiple similar images. However, this will cause the image dataset to occupy a significantly larger ROM space. Under the premise that the computer performance remains unchanged and the number of images in the original dataset meets the training requirements, the training time will be significantly extended, which will aggravate the aging of the computer. In addition, there are also some data augmentation solutions based on generative adversarial networks (GANs). Although such solutions can provide a large number of high-quality data augmentation images, they require training and are time-consuming.
[0004] Therefore, in order to solve the above problems, the present invention provides a data augmentation method that can be performed quickly and effectively without increasing the capacity of the data set, which is of great significance for improving the accuracy of deep learning training and the generalization ability of the model, and extending the service life of the computer. Summary of the invention
[0005] In view of the above problems, the present invention proposes an image augmentation method, device, and storage medium, which can fuse images under different lighting conditions into one image, so as to quickly and effectively perform data augmentation without increasing the capacity of the data set.
[0006] In order to achieve the above object, the present invention provides an image augmentation method, comprising:
[0007] Collect the required RGB images and generate original images;
[0008] Performing color gamut transformation and saturation transformation on the original image to generate a plurality of first images, each of the first images corresponding to a different lighting condition;
[0009] fusing each of the first images with the original image to generate a plurality of second images;
[0010] The second images are fused to generate a target image that integrates all lighting conditions.
[0011] Optionally, fusing each of the first images with the original image to generate a plurality of second images includes:
[0012] According to a first fusion direction, selecting at least one of the first images to be fused with the original image to generate a first fused image;
[0013] fusing the remaining first images with the original images according to a second fusion direction to generate a second fused image, wherein the second fusion direction is opposite to the first fusion direction;
[0014] The second image includes the first fused image and a second fused image.
[0015] Optionally, fusing the second images to generate a target image fusing all lighting conditions includes:
[0016] The first fused image is fused with the second fused image according to the first fusion direction to generate the target image.
[0017] Optionally, before generating the original image, the method further includes:
[0018] The collected RGB images are preprocessed by image clarity determination, image filtering and denoising, and image size conversion.
[0019] Optionally, before generating the original image, the method further includes:
[0020] Label the task targets in the collected RGB images.
[0021] Optionally, after generating the target image that integrates all lighting conditions, it also includes:
[0022] If the resolution of the target image is different from the resolution of the original image,
[0023] The target image is resized so that the resolution of the target image is the same as the resolution of the original image.
[0024] Optionally, after generating the target image that integrates all lighting conditions, it also includes:
[0025] If the resolution of the target image is the same as the resolution of the original image,
[0026] The target image is overwritten with the original image using the file path and file name of the original image, making it compatible with the annotated labels.
[0027] Optionally, performing color gamut transformation and saturation transformation on the original image to generate multiple first images includes:
[0028] Convert the original image from the RGB color model to the LAB color model;
[0029] The values of the three channels of the LAB color model and the image saturation are adjusted to generate a plurality of the first images.
[0030] Another aspect of the present invention further provides an image augmentation device, which adopts the above-mentioned image augmentation method and at least comprises:
[0031] The acquisition module acquires the required RGB images and generates the original images;
[0032] An image transformation module, used for performing color gamut transformation and saturation transformation on the original image to generate a plurality of first images, each of which corresponds to a different illumination condition;
[0033] an image fusion module, used for fusing each of the first images with the original image to generate a plurality of second images; and
[0034] The second images are fused to generate a target image that integrates all lighting conditions.
[0035] Another aspect of the present invention further provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the above-mentioned image augmentation method are implemented.
[0036] It can be seen from the above scheme that the advantages of the present invention are:
[0037] The image augmentation method provided by the present invention realizes image augmentation based on color gamut transformation and image fusion technology. Specifically, the original image is generated by acquiring the required RGB image; the original image is subjected to color gamut transformation and saturation transformation to generate multiple first images with different corresponding lighting conditions; then, each of the first images is fused with the original image to generate multiple second images; the second images are fused, and after multiple fusion processes, a target image with all lighting conditions is generated. The method fuses images under different lighting conditions into one image to quickly and effectively perform data augmentation without increasing the capacity of the data set. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 The overall flow chart of the image augmentation method provided by the embodiment of the present invention is shown;
[0039] Figure 2 A comparison diagram of the original image and the generated first image under different lighting conditions is shown;
[0040] Figure 3 Shows Figure 1 Specific flow diagram of step S3;
[0041] Figure 4 An example diagram of the effect of the initial fusion is shown;
[0042] Figure 5 An example diagram of the effect of re-fusion is shown;
[0043] Figure 6 The figure shows an architecture diagram of an image augmentation device.
[0044] in,
[0045] 400- Image augmentation device;
[0046] 401-collection module;
[0047] 402-image transformation module;
[0048] 403-image fusion module;
[0049] S1-S4, S301-S302: steps. DETAILED DESCRIPTION
[0050] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.
[0051] The present invention provides an image augmentation method, which is based on color gamut transformation and image fusion technology. By performing color gamut transformation and saturation transformation operations, multiple images with different lighting conditions are generated from the original image. Then, based on the different transparency rates of the image and the weighted fusion principle of the image, multiple images with different lighting conditions are generated from the original image for fusion, and multiple fused images with different lighting conditions are generated. These fused images are seamlessly spliced and fused according to a rule to obtain an image containing multiple lighting conditions. It fuses images under different lighting scenarios into one image, so as to quickly and effectively perform data augmentation without increasing the capacity of the data set.
[0052] The image augmentation method provided by the present invention is described in detail below.
[0053] like Figure 1 As shown in Figure 1 The overall flow chart of the image augmentation method is shown.
[0054] An image augmentation method, comprising:
[0055] S1. Collect the required RGB image and generate the original image.
[0056] In a specific implementation, a monocular RGB camera can be used to capture an RGB image containing the target scene of the required data set, such as in daytime with good lighting conditions.
[0057] In addition, in the specific implementation, the phenomenon of image blur and image noise pollution caused by external reasons such as device jitter during shooting will significantly affect the training accuracy of the deep learning model. Therefore, the collected RGB image cannot be used directly, and the image needs to be preprocessed to improve the overall quality of the image data set. Therefore, in this embodiment, before generating the original image, the collected RGB image needs to be further preprocessed. Specifically, an image processing program can be used to preprocess the collected RGB image, first remove the unqualified image data with blurred images, and then unify the image resolution and filter denoising. Then, after completing the image preprocessing, use Labelme and other annotation software to annotate the task target in the RGB image, and obtain an annotation file containing the task target annotation name, annotation coordinates, and the name is the same as the image name, and the file suffix is .json, .txt or .xml.
[0058] In the specific implementation, preprocessing usually includes operations such as image clarity determination, image filtering and denoising, and image size transformation. Among them: the purpose of image clarity determination is to remove extremely blurred RGB images that do not meet the requirements. The clarity of all images is judged by the Tenengrad gradient method. If the calculated clarity is greater than the set threshold, the image is retained, otherwise it is eliminated. The purpose of image filtering and denoising is to remove the noise contained in the image that affects the training accuracy and improve the quality of the image data set. Specifically, the mean filter and median filter methods can be used to perform global screening on the image to remove the noise. The image size transformation is to unify the resolution and solve the problem that the resolution size of the images taken in some data sets is inconsistent, and the image resolution size changes after data augmentation, resulting in the original label being inapplicable. The specific idea is to unify the resolution of all images before fusion, read the resolution of the original image after image data augmentation, and then resize the corresponding image after augmentation.
[0059] Specifically, for the image clarity determination method, the Tenengrad gradient method is used. This method is based on calculating the horizontal and vertical gradients of the Sobel operator image, and then calculating the average of the gradients in the two directions. The higher the average gradient, the clearer the image. Specifically, the structure of the Sobel operator is an odd-order matrix, which is 3×3 here, and its form is as follows:
[0060]
[0061] (a) Horizontal Sobel operator matrix b) Vertical Sobel operator matrix
[0062] To calculate the Sobel operator calculation result of the image, the RGB image needs to be converted into a single-channel grayscale image, and then the grayscale image is converted into a binary image. If the pixel values contained in the converted image can be expressed by the matrix of formula (2), the calculation formula of the Tenengrad gradient method is shown in formula (3).
[0063]
[0064] If the size of the image is larger than the order of the Sobel operator, the operator needs to be slid on the image from left to right and from top to bottom. The sliding step size can be set as needed. Finally, the average value of formula (3) is calculated for each sliding. If the result is greater than the preset clarity threshold T, it is judged as a clear image and retained. Otherwise, it is removed from the image dataset.
[0065] For the image filtering denoising method, a combination of mean filtering and median filtering is specifically used. Among them, mean filtering replaces all pixel values in the image area with the mean value of the pixels in the image area, and median filtering replaces all pixel values in the image area with the median value of the pixels in the image area. In order to ensure the clarity of the image, the image area used for filtering is generally small, such as 3×3. After filtering is completed in one area, it will move to the next position on the image for filtering according to the preset step size, and the moving direction is as described above. The calculation formulas are shown in equations (4) and (5) respectively.
[0066]
[0067] Median filtering = Md(a,b,c,d,e,f,g,h,i)(5)
[0068] Among them, a, b, c, d, e, f, g, h, i, and k all represent parameters.
[0069] For the image resizing method, in order to smoothly integrate the programming logic of the program and ensure that the image labeling area remains unchanged after data augmentation, it is necessary to resize the original image and the augmented image of the image data set constructed based on different resolutions. In order to improve the processing speed, the nearest neighbor interpolation method can be used to enlarge the image size, and the mean method can be used to reduce the image size. Among them, the principle of the nearest neighbor interpolation is to replace the value of the target pixel in the image with the gray value of the input pixel closest to it to achieve the principle of magnification. The mean method reduces the image by first dividing the original image into several regions with the number of rows and columns as the reduced size, and then averaging the pixel values in each region, and using the mean as the pixel value of each region.
[0070] S2. Performing color gamut transformation and saturation transformation on the original image to generate a plurality of first images, each of the first images corresponding to a different lighting condition.
[0071] In this embodiment, after the original image is generated in step S1, the original image is further subjected to color gamut transformation and saturation transformation to generate multiple first images corresponding to different lighting conditions. Figure 2 As shown in Figure 2 The comparison diagram of the original image and the generated first image under different lighting conditions is shown. By adjusting the color gamut and saturation of the original image, images under three different lighting conditions, namely, the first image, are generated, such as early morning, noon, and dusk. Figure 2 Figure (a) shows the original image, Figure (b) shows the first image that looks like early morning, Figure (c) shows the first image that looks like noon, and Figure (d) shows the first image that looks like dusk.
[0072] In the specific implementation, through the lighting environment transformation algorithm, first, the original image is transformed from the RGB color model to the LAB color model, where the L component represents the image pixel brightness, and takes values in [0,100], that is, from pure black to pure white; A and B represent the range from red to green and from yellow to blue, respectively, and both have a value range of [127,-128]. Then, the values of the three channels of the LAB color model and the image saturation are adjusted to generate multiple first images corresponding to different lighting conditions. Experimental verification shows that keeping the values of all elements of the L and A channels unchanged and changing the values of all elements of the B channel to -28 can obtain a lighting effect similar to the early morning; when keeping L unchanged and changing the values of the A and B channels to 32 and 122 respectively, a dusk-like lighting effect will be obtained; and keeping the values of all channel elements unchanged and adjusting the image saturation to the lowest, that is, fusing a white image into the image, a strong lighting effect similar to midday light can be obtained.
[0073] S3, fusing each of the first images with the original image to generate multiple second images. In this embodiment, after generating multiple first images with different lighting conditions through step S2, further based on the different transparency of the images and the weighted fusion principle of the images, the multiple first images generated in step S2 are naturally fused with the original image to generate multiple second images, and then the second images are seamlessly spliced and fused again. After multiple fusion processes, a target image is finally generated, so that the target image contains a variety of lighting environments, thereby effectively enriching the data set without increasing the number of images or additionally annotating the images.
[0074] In a specific implementation, for the initial fusion of the original image and the first image, as Figure 3 As shown in Figure 3 The specific flow chart of step S3 is shown. When the images are fused, it is necessary to select the image fusion direction and transition speed. The image fusion direction determines the left-right arrangement of the fused materials in the fused image, and the transition speed determines the transition smoothness of the fused image. Therefore, in this embodiment, first, the first fusion direction and fusion speed are set, and at least one first image is selected to be fused with the original image to generate a first fused image (S301). Then, the other remaining first images are fused with the original images according to the second fusion direction to generate a second fused image, wherein the second fusion direction is opposite to the first fusion direction (S302). Therefore, after this initial fusion process, the generated second image contains two types of images: the first fused image and the second fused image.
[0075] For the initial fusion process, Figure 4 As shown in Figure 4 The effect example of the initial fusion is shown. The first image that looks like the morning is selected to be fused with the original image. The first fusion direction is set to be from left to right. The final fusion result will transition from the first image that looks like the morning to the original image from left to right. If the transition speed is set to fast, the algorithm will generate 100 increasing transparency rates in the transparency rate interval of [0,1], and then generate 100 fusion images with fusion materials as materials according to the transparency rate, and divide the 100 images into 100 equal-sized areas from left to right. In the order of transparency from small to large, a corresponding image area is selected for each image, and finally spliced, so as to obtain the first fusion image that transitions from the first image that looks like the morning to the original image with a natural transition. If the transition speed is set to slow, the above operation will be repeated in the transparency rate interval of [0,0.6]. Then, for the other two images except the first image that looks like the morning, the second fusion direction opposite to the first fusion direction is selected to fuse with the original image, that is, from right to left to generate the second fusion image. Figure 4Figure (a) shows the fusion of the first image that seems to be in the early morning and the original image, the first fusion direction is set to be from left to right, and the fusion speed is fast; Figure (b) shows the fusion of the first image that seems to be in the noon and the original image, the second fusion direction is set to be from right to left, and the fusion speed is fast; Figure (c) shows the fusion of the first image that seems to be in the dusk and the original image, the second fusion direction is set to be from right to left, and the fusion speed is fast.
[0076] In addition, the transparency of the image changes according to the different fusion stages, so the transparency of the image after the initial fusion is:
[0077]
[0078] The transparency of the first image generated in formula (6) is an increasing decimal in the range of [0, 1], while the transparency of the original image is a decreasing decimal in the range of [1, 0]. The sum of the transparency of the two is always 1. When the calculated result of the transparency is infinite, it means that the fused image at this time is the original image.
[0079] S4. Fusing the second images to generate a target image that fuses all lighting conditions.
[0080] In this embodiment, after the initial fusion in step S1, the second image generated after the fusion is further fused to improve the information richness contained in the image and generate a target image that integrates all lighting conditions. Specifically, according to the first fusion direction, the fusion speed is set, and the first fused image and the second fused image are fused according to the principle of step S3 to generate the target image. In addition, the transparency after the second fusion is:
[0081]
[0082] For the re-fusion process, Figure 5 As shown in Figure 5 An example of the effect of re-fusion is shown. Figure 4 The multiple images generated in the first fusion direction are fused again to finally generate a target image after data augmentation.
[0083] In addition, in this embodiment, if different image resolutions are used to capture images, image distortion may occur after transformation. Therefore, it is necessary to use the image resizing algorithm of the image processing program to resize the target image after data augmentation so that it is consistent with the resolution size of the original image to ensure that the label annotation area of the original image is correct.
[0084] In this regard, after generating a target image that integrates all lighting conditions, this embodiment further determines whether the resolution of the target image is the same as the resolution of the original image. If the resolution of the target image is different from that of the original image, it is necessary to first resize the target image so that the resolution of the target image is the same as that of the original image; then, in order to make the augmented target image compatible with the labels in the original image, the file path and file name of the original image are used to overwrite the original image with the target image, making it compatible with the labeled labels. If the resolution of the target image is the same as that of the original image, the file path and file name of the original image can be directly used to overwrite the original image with the target image, making the augmented target image compatible with the labels in the original image.
[0085] In summary, the image augmentation method provided in this embodiment realizes image augmentation based on color gamut transformation and image fusion technology. Specifically, the original image is generated by acquiring the required RGB image; and the original image is subjected to color gamut transformation and saturation transformation to generate multiple first images with different corresponding lighting conditions; then, each of the first images is fused with the original image to generate multiple second images; the second images are fused, and after multiple fusion processes, a target image with all lighting conditions is generated. This method fuses images under different lighting conditions into one image to quickly and effectively perform data augmentation without increasing the capacity of the data set.
[0086] In addition, the above embodiment of the present invention can be a terminal device with image augmentation method function, and the terminal device can include a personal terminal and a host terminal, etc., which is not limited by the embodiment of the present invention. The terminal can support operating systems such as Windows, Android, IOS, Windows Phone, etc.
[0087] Reference Figure 6 , Figure 6 An image augmentation device 400 is shown. Based on the image augmentation method, it can be applied to a personal terminal and a host terminal device, which can be realized by Figure 1 , Figure 3 The image augmentation method shown, the image augmentation device provided in the embodiment of the present application can implement each process implemented by the above-mentioned image augmentation method.
[0088] An image augmentation device 400, using the above-mentioned image augmentation method, at least comprises:
[0089] The acquisition module 401 acquires the required RGB image and generates the original image;
[0090] An image transformation module 402 is used to perform color gamut transformation and saturation transformation on the original image to generate a plurality of first images, each of which corresponds to a different illumination condition;
[0091] An image fusion module 403, configured to fuse each of the first images with the original image to generate a plurality of second images; and
[0092] The second images are fused to generate a target image that integrates all lighting conditions.
[0093] In addition, it should be understood that in the image augmentation device 400 according to the embodiment of the present application, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the image augmentation device 400 can be divided into functional modules different from the modules illustrated above to complete all or part of the functions described above.
[0094] In addition, an embodiment of the present application also provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, the steps of the above-mentioned image augmentation method are implemented and the same technical effect can be achieved.
[0095] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0096] In addition, an embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the above-mentioned image augmentation method are implemented and the same technical effect can be achieved.
[0097] It should be noted that, in this article, the term "comprises", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device containing a series of elements not only contains those elements, but also contains other elements that are not explicitly listed, or also contains elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method, article or device containing the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be applied, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0098] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. An image augmentation method, characterized in that: Include: Collect the required RGB images and generate original images; Performing color gamut transformation and saturation transformation on the original image to generate a plurality of first images, each of the first images corresponding to a different lighting condition; fusing each of the first images with the original image to generate a plurality of second images; The second images are fused to generate a target image that integrates all lighting conditions.
2. The method according to claim 1, characterized in that Fusion of each of the first images with the original image to generate a plurality of second images comprises: According to a first fusion direction, selecting at least one of the first images to be fused with the original image to generate a first fused image; fusing the remaining first images with the original images according to a second fusion direction to generate a second fused image, wherein the second fusion direction is opposite to the first fusion direction; The second image includes the first fused image and a second fused image.
3. The method according to claim 2, characterized in that The second image is fused to generate a target image fused with all illumination conditions, including: The first fused image is fused with the second fused image according to the first fusion direction to generate the target image.
4. The method according to claim 1, characterized in that: Before generating the original image, the method further includes: The collected RGB images are preprocessed by image clarity determination, image filtering and denoising, and image size conversion.
5. The method according to claim 1, characterized in that Before generating the original image, the method further includes: Label the task targets in the collected RGB images.
6. The method according to claim 1, characterized in that After generating the target image that integrates all lighting conditions, it also includes: If the resolution of the target image is different from the resolution of the original image, The target image is resized so that the resolution of the target image is the same as the resolution of the original image.
7. The method according to claim 5 or 6, characterized in that: After generating the target image that integrates all lighting conditions, it also includes: If the resolution of the target image is the same as the resolution of the original image, The target image is overwritten with the original image using the file path and file name of the original image, making it compatible with the annotated labels.
8. The method according to claim 1, characterized in that The original image is subjected to color gamut transformation and saturation transformation to generate a plurality of first images, comprising: Convert the original image from the RGB color model to the LAB color model; The values of the three channels of the LAB color model and the image saturation are adjusted to generate a plurality of the first images.
9. An image augmentation device, characterized in that: The image augmentation method according to any one of claims 1 to 7 comprises at least: The acquisition module acquires the required RGB images and generates the original images; An image transformation module, used for performing color gamut transformation and saturation transformation on the original image to generate a plurality of first images, each of which corresponds to a different illumination condition; An image fusion module, used for fusing each of the first images with the original image to generate a plurality of second images; and The second images are fused to generate a target image that integrates all lighting conditions.
10. A readable storage medium having a program or instruction stored thereon, characterized in that: When the program or instruction is executed by a processor, the steps of the image augmentation method according to any one of claims 1 to 8 are implemented.