A data enhancement method based on image curve adjustment and color redistribution

Abstract

A data augmentation algorithm based on image curve adjustment and color redistribution is proposed. This algorithm adjusts the pose of the image using trigonometric functions and adjusts the color information of pedestrians using an improved flood filling algorithm. The newly generated image data is then applied to the training process of a deep learning network. By amplifying the pose and appearance information of the images respectively, the algorithm achieves a significant augmentation of the dataset samples, enabling the deep learning model to acquire more data features for training and thus achieving a higher pedestrian re-identification accuracy. This method is based on traditional algorithms and mathematical trigonometric functions, is easy to implement, and provides a significant performance improvement. It can be easily embedded as a data preprocessing module into other deep learning networks.

Description

[Technical field]

[0001] The present invention relates to the fields of image processing, computer vision, and in particular to a data enhancement method based on image curve adjustment and color redistribution. [Background technology]

[0002] There are several major issues in person re-identification research, including small datasets, severe occlusion, uncontrollable lighting conditions, variable angles, and variable pedestrian postures. In particular, the variable environment, postures, and small datasets make it easier for the training results of deep learning network models to overfit. Under such circumstances, exploring a more effective and low-cost data augmentation method has become a very important research direction.

[0003] Existing and commonly used methods include flipping, scaling, rotating, etc., which improve the training results of deep learning network models. However, these methods do not increase the feature quantity of samples, making the improvement effect on the model very limited. Some methods that use generative adversarial networks to generate fake samples attempt to increase the feature quantity of samples, and have achieved relatively obvious improvement effects. However, the quality of fake samples generated by these methods is relatively poor, and they require relatively high costs for self-training and parameter correction, making them difficult to apply to other deep learning network models. [Summary of the invention]

[0004] The purpose of the present invention is to propose a data enhancement method based on image curvilinear adjustment and color redistribution, which can overcome the shortcomings of the prior art, make random adjustments in the posture and appearance of the image, enhance the data set, and more significantly improve the recognition accuracy of the deep learning network model compared to the prior art. It has great practical value in pedestrian re-identification and also has good reference significance for other computer vision fields that require image enhancement. The algorithm is simple and easy to implement.

[0005] A data enhancement method based on image curve adjustment and color redistribution, characterized in that it includes two parts: curve adjustment and image color redistribution processing, specifically including the following steps:

[0006] As attached Figure 1 As shown, the batch of image samples will be processed by two parallel operation branches, image curve adjustment and image color redistribution, and then the batch image results processed by the two branches are directly put together. The number of merged images is twice the number of batch image samples. The operation steps of the two branches will be explained in detail below, where each image sample in the batch image sample is regarded as the original image.

[0007] (1) Treat each image in the batch of image samples as the original image to be processed, and perform image curve adjustment on the original image:

[0008] (1.1) Use the trigonometric function shown in formula (1-1) to adjust the original image into an image curve:

[0009] d=αW sin(ωh+φ) (1-1)

[0010] Where W is the width of the original image, α is the horizontal offset distance scaling factor; h is the value of the original image in the height direction, ω is the period adjustment factor; φ is the offset parameter of the trigonometric function; d is the calculation result of the formula, and the image height h position will move horizontally according to the value of d;

[0011] The lateral offset distance scaling factor α in the step (1.1) is the main factor affecting the curvature degree of the original image, that is, the larger the α value, the greater the lateral curvature degree, that is, the greater the curve amplitude.

[0012] The period adjustment factor ω in the step (1.1) is the main factor affecting the degree of longitudinal curvature of the original image, that is, the larger the ω value, the smaller the degree of longitudinal curvature.

[0013] The offset parameter φ of the trigonometric function in the step (1.1) is used to dynamically adjust the position of the origin of the curve of the original image. By randomly assigning a value to φ, the randomness of the curve adjustment is increased.

[0014] (1.2) The original image is regarded as a two-dimensional coordinate space, and the height h of the original image is taken as the independent variable, that is, as the input of formula (1-1), and the horizontal offset distance d of the original image at the height h is taken as the dependent variable, that is, the output of formula (1-1); in the two cases where d is greater than 0 or less than 0, the direction of the horizontal offset is only required to be opposite; the resulting (h, d) is the two-dimensional space coordinate of the image, and the vertical direction is recorded as the independent variable and the horizontal direction as the dependent variable, thereby completing the image curve adjustment of the original image;

[0015] (2) Use the improved flood filling algorithm to redistribute the image color of the original image:

[0016] The existing flood filling algorithm is a widely used image processing algorithm, which assigns a fixed target color to a selected color block (the range of a color block in the flood filling algorithm is determined by the selected starting pixel point and threshold), that is, directly replacing all colors in the color block with the seed color; the present invention improves the flood filling algorithm, calculates a new color value according to the original distribution characteristics of the replaced color, and retains the distribution characteristics of the original color block color, so that the sample data can be amplified while retaining the contour information and color distribution information of the original sample, which is specifically achieved through the following steps:

[0017] (2.1) Given an original image, denoted as original image I, randomly select pixel point C on the original image I (r，c) As the starting pixel point, (r, c) is the corresponding coordinate of the pixel point in the two-dimensional space, where r represents the vertical position of the pixel point on the original image I, and c represents the horizontal position of the pixel point on the original image I; C (r，c) Represents the pixel value of the pixel point with coordinates (r, c) on the original image I;

[0018] (2.2) Setting a color threshold T. When the absolute value of the difference between two adjacent colors is less than the color threshold T, the two adjacent colors are considered to be connected. The color threshold T is an important parameter in the flood filling algorithm. The specific threshold calculation method of the present invention is described as follows:

[0019] Calculate the pixel C (r，c) The four adjacent points above, below, left and right of the center point are respectively connected to the pixel C (r，c) The pixel difference is calculated by summing up the absolute value of the pixel difference and dividing it by 2. The result is the value of the color threshold T. The calculation formula is as follows (2-1):

[0020] T=(C (r+1，c) +C (r-1，c) +C (r，c+1) +C (r，c-1) -4×C (r，c) ) / 2 (2-1)

[0021] (2.3) Randomly select an existing color value C on any other original image new , as the seed color to be rendered on the original image I;

[0022] (2.4) Since pixel C (r，c) As the starting pixel, we can directly use the seed color C selected in step (2.3) new Replace C (r，c) , and record the new color value at the pixel coordinate (r, c) as N (r，c) ;

[0023] (2.5) The four adjacent pixels above, below, left and right of the coordinate point (r, c) are respectively (r，c) Calculate a pixel difference dis and take its absolute value; and modify the pixel value of the corresponding adjacent pixel point according to the relationship between the difference dis and the color threshold T;

[0024] In the step (2.5), the pixel value of the corresponding adjacent pixel point is modified according to the relationship between the difference value dis and the color threshold T, specifically: if an adjacent pixel point is (r，c) If the difference dis is less than the threshold T, the new color value N is used. (r，c)Add the value of dis, and the result is the new pixel value of the adjacent pixel, and the newly assigned pixel is recorded as the processed pixel; if the difference dis is not less than the threshold T, the adjacent pixel will no longer be processed and will be regarded as the unprocessed pixel.

[0025] (2.6) After completing the assignment operation of new pixel values ​​of the four upper, lower, left, and right adjacent pixels with the pixel coordinate (r, c) as the center point in steps (2.4) and (2.5), the processed four upper, lower, left, and right adjacent pixels will be used as the center point in turn, and the unprocessed pixel will not be used as the center point. The calculation will continue according to steps (2.4) and (2.5) until a new center point is found; each processed pixel needs to be used as a center point to process its four upper, lower, left, and right adjacent pixels, and each pixel can only be processed once and cannot be assigned a value or used as a center point twice;

[0026] (2.7) After the original image I goes through steps (2.1) to (2.6), the colors of the original image I are re-rendered to obtain a picture with a new appearance.

[0027] Working principle of the present invention: The conventional flood filling method assigns a fixed target color to the selected color block, while the improved flood filling algorithm calculates the new color value according to the original distribution characteristics of the replaced color. During the training process, all pictures will be processed in real time by the image color redistribution module. In the image redistribution module, we have implemented an algorithm that can calculate the new color value according to the color distribution difference of the original image, and use the breadth-first algorithm to achieve the work of changing the color of the picture. The picture generated by the color redistribution algorithm can keep the original color distribution law of the picture, and let the picture show new appearance characteristics, so that the discriminant network can be forced to identify the picture through the color distribution law, rather than just relying on the color value itself.

[0028] This method is the first lightweight traditional algorithm that can generate good quality fake samples without relying on deep learning network models. Different from the existing methods based on generative adversarial networks, this method is based on simple trigonometric functions and an improved flood filling algorithm, making adjustments in posture and appearance respectively, and does not need to rely on deep learning network models, and can directly generate high-quality fake samples.

[0029] In addition, the method of the present invention allows the original features to have new feature states after adjustment. Among them, the image curve module can adjust the posture of the pedestrian in a reasonable range, so that the original posture of the pedestrian evolves into more possible postures, thereby providing more mining potential for the model; and the image color redistribution module renders a certain color block into a new color under the premise of maintaining the original color distribution characteristics of the image, thus adding more appearance states for each pedestrian. Through the above two methods, that is, using trigonometric functions to adjust the posture of the image and using the improved flood filling algorithm to adjust the color information of the pedestrian, the data set is augmented in posture and appearance respectively, and the newly generated image data is applied to the training process of the deep learning network, so that more features are fed into the deep network learning model, which can effectively reduce the overfitting of the model and improve the generalization ability of the model.

[0030] The advantages of the present invention: the curve processing and color redistribution methods greatly enrich the characteristics of the data. The model can propose more robust features on the data set; the curve processing module adjusts the degree of image distortion to allow pedestrian pictures with a single posture to have variable postures. The color redistribution method adds more appearance characteristics to pedestrian pictures while maintaining the original color distribution characteristics; the combination of the two methods allows the data set to be expanded in terms of both posture and appearance, which can greatly improve the generalization ability of the model and provide ideas for other fields that are limited by small data sets; compared with existing methods, this application is lightweight, easy to implement, easy to expand, and plug-and-play, which is convenient for various deep learning network models to choose and use, and has great practical value.

Brief Description of the Drawings

[0031] Figure 1 It is a flow chart of a data enhancement method based on image curve adjustment and color redistribution in combination with a deep learning network model according to the present invention.

[0032] Figure 2 It is a schematic diagram of the effect of fixing ω and φ and comparing the influence of two different α values ​​on the generated results in a specific embodiment of the present invention.

[0033] Figure 3 It is a schematic diagram of the effect of fixing α and φ and comparing the influence of two different ω values ​​on the generated results in a specific embodiment of the present invention.

[0034] Figure 4 It is a schematic diagram of the effect of fixing ω and α and comparing the influence of two different φ values ​​on the generated results in a specific embodiment of the present invention.

[0035] Figure 5 It is a diagram showing the ablation experiment effect of the image curve adjustment module in a specific embodiment of the present invention.

[0036] Figure 6 It is a rendering of the image color redistribution method in a specific embodiment of the present invention. [Specific implementation method]

[0037] Embodiment: A data enhancement method based on image curve adjustment and color redistribution, the framework diagram is shown in the attached figure Figure 1 As shown, its features include the following instructions and steps:

[0038] (1) As shown in the attached figure Figure 1 As shown in the figure, the batch of image samples will be processed by two parallel operation branches: image curve adjustment and image color redistribution. Then the processing results of the two branches are directly put together, so that the number of merged images is twice the number of batch image samples. The operation steps of the two branches will be explained in detail below. Each image sample in the batch of image samples is called the original image in the following step description.

[0039] (2) Image Curve Adjustment Process

[0040] (2.1) Image curve adjustment is to process the original image using the trigonometric function shown in formula (2-1):

[0041] d=αW sin(ωh+φ) (2-1)

[0042] Where W is the width of the original image, α is the horizontal offset distance scaling factor; h is the value in the height direction of the original image, ω is the period adjustment factor; φ is the offset parameter of the trigonometric function;

[0043] The lateral offset distance scaling factor α in step (2.1) is the main factor affecting the degree of curvature of the original image, that is, the larger the α value, the greater the degree of lateral curvature, that is, the greater the curve amplitude, and its effect is shown in the figure. Figure 2 shown.

[0044] The period adjustment factor ω in step (2.1) is the main factor affecting the degree of longitudinal curvature of the original image, that is, the larger the ω value, the smaller the degree of longitudinal curvature. The effect is shown in the figure. Figure 3 shown.

[0045] The offset parameter φ of the trigonometric function in step (2.1) is used to dynamically adjust the position of the origin of the curve of the original image. By randomly assigning φ, the randomness of the curve adjustment is increased. The effect is shown in the figure. Figure 4 shown.

[0046] Embodiment description: In step (2.1), φ can be dynamically and randomly assigned an integer during program execution, so this value does not have a fixed value method; α and ω respectively affect the horizontal and vertical curvatures of the curve of the original image. When α is too large (it can be assumed here that α>1), the amplitude of the trigonometric function will exceed the width, causing the image to be severely curved, and thus the meaning of image curvature will be lost; the value of α is related to the original image and the content of the image. The image cited in the present invention is sample data for pedestrian re-identification. Through the clamping method, the value of α is adjusted multiple times and the generated data is actually verified by the deep network model of pedestrian re-identification. When α=0.04, the effect is best; the process of determining the ω value is the same as that of α. Through the clamping method, the value is set multiple times and then confirmed through deep model training. When ω=0.05, the result is best.

[0047] (2.2) The original image is regarded as a two-dimensional coordinate space, and the height h of the original image is taken as the independent variable, that is, as the input of formula (2-1), and the horizontal offset distance d of the original image at the height h is taken as the dependent variable, that is, the output of formula (2-1); in the two cases where d is greater than 0 or less than 0, the direction of the horizontal offset is only required to be opposite; the resulting (h, d) is the two-dimensional space coordinate of the image, but the vertical direction is the independent variable and the horizontal direction is the dependent variable;

[0048] (3) Image color redistribution process

[0049] The improved flood filling algorithm is used to perform color redistribution processing on the image; the existing flood filling algorithm assigns a fixed target color to the selected color block, that is, directly replaces all colors in the color block with the seed color; the present invention improves the flood filling algorithm, calculates the new color value according to the original distribution characteristics of the replaced color, that is, retains the distribution characteristics of the original color block color, so that the sample data can be amplified while retaining the original sample's outline information and color distribution information, which is specifically achieved through the following steps:

[0050] (3.1) Given an original image, denoted as original image I, randomly select pixel point C on the original image I (r，c) As the starting pixel point, (r, c) is the corresponding coordinate of the pixel point in the two-dimensional space, where r represents the vertical position of the pixel point on the original image I, and c represents the horizontal position of the pixel point on the original image I; C (r，c) Represents the pixel value of the pixel point with coordinates (r, c) on the original image I;

[0051] (3.2) Setting a color threshold T. When the absolute value of the difference between two adjacent colors is less than the color threshold T, the two adjacent colors are considered to be connected. The color threshold T is an important parameter in the flood filling algorithm. The specific threshold calculation method of the present invention is described as follows:

[0052] Calculate the pixel C (r，c) The four adjacent points above, below, left and right of the center point are respectively connected to the pixel C (r，c) The pixel difference is calculated by summing up the absolute value of the pixel difference and dividing it by 2. The result is the value of the color threshold T. The calculation formula is as follows (3-1):

[0053] T=(C (r+1，c) +C (r-1，c) +C (r，c+1) +C (r，c-1) -4×C (r，c) ) / 2 (3-1)

[0054] (3.3) Randomly select an existing color value C on any other original image new , as the seed color to be rendered on the original image I;

[0055] (3.4) Since pixel C (r，c) As the starting pixel, we can directly use the seed color C selected in step (3.3) new Replace C (r，c) , and record the new color value at the pixel coordinate (r, c) as N (r，c) ;

[0056] (3.5) The four adjacent pixels above, below, left and right, centered at coordinate (r, c) are respectively (r，c) Calculate a pixel difference dis (take the absolute value); and modify the pixel value of the corresponding adjacent pixel point according to the relationship between the difference dis and the color threshold T;

[0057] In the step (3.5), the pixel value of the corresponding adjacent pixel point is modified according to the relationship between the difference value dis and the color threshold T, specifically: if an adjacent pixel point is (r，c) If the difference dis is less than the threshold T, the new color value N is used. (r，c) Add the value of dis, and the result is the new pixel value of the adjacent pixel; if the difference dis is not less than the threshold T, the adjacent pixel will no longer be processed.

[0058] (3.6) After completing the assignment of new pixel values ​​to the four adjacent pixels above, below, left, and right with the pixel coordinates (r, c) as the center in steps (3.4) and (3.5), move the center to the four adjacent pixels above, below, left, and right that have been processed (if a certain adjacent pixel has not been processed, it will not be used as the center point), and continue to calculate according to steps (3.4) and (3.5) until a new center point is found. Each processed point needs to be used as a center point to process its four adjacent pixels above, below, left, and right, and each pixel can only be processed once and cannot be assigned a value or used as a center point twice.

[0059] (3.7) After the original image I goes through steps (3.1) to (3.6), the colors of the original image I are re-rendered to obtain a picture with a new appearance.

[0060] As shown in the attached figure Figure 6 As shown in FIG. 1 , after the original image is processed by image color redistribution, a pedestrian image with the pants color and part of the background color changed appears.

[0061] Since this example needs to process a large number of images, it is a repetitive and cumbersome task, so it needs to be implemented in C++ language with a faster processing speed, and then a link library can be generated, which is convenient for popular python or java to call. This not only meets the needs of using a deep network model implemented in a high-level language to conveniently call the method of the present invention, but also ensures the speed of image processing. The main innovations are as follows:

[0062] 1. The trigonometric function is applied to image posture adjustment, which is simple and easy to implement, and the experimental effect is excellent.

[0063] 2. The flood filling method is improved to retain the color block distribution characteristics of the original image, avoiding the problem of previous crude direct color replacement, so that the newly generated image has a new appearance while ensuring that the original color distribution characteristics are inherited.

[0064] 3. The implementation method is traditional algorithm, avoiding the use of heavy deep network learning models, and is easy to understand and use.

[0065] The present invention proposes a data enhancement algorithm based on image curve adjustment and color redistribution, which mainly includes the following key modules:

[0066] In the present invention, steps (2.1) and (2.2) determine the method of adjusting the image by using the trigonometric function formula. The ablation experiment effect is shown in the accompanying figure. Figure 5 shown. Figure 5In the first row, ω and φ are set to fixed values, and α is gradually increased. It can be seen that the amplitude of the trigonometric function curve on the image is getting larger and larger, which leads to an increasing degree of distortion of the image. Figure 5 In the second row, α and φ are set to fixed values, and ω is gradually increased, which results in a smaller period of the trigonometric function curve and too many distortion points in the image, which is increasingly unreasonable. α and ω are set to fixed values, and φ is gradually increased, which results in a change in the origin of the trigonometric function, resulting in a variety of curve effects in the image, but the degree of distortion remains unchanged.

[0067] From the perspective of trigonometric mathematical principles, the value of φ will only periodically affect the origin of the trigonometric function curve, and in the present invention, this variable is only used to increase the randomness of the curve, so it can be randomly selected; and the values ​​of α and ω will directly affect the degree of distortion of the image, and when the values ​​are too large, the effect of reasonable curve adjustment of the image will be seriously affected. Therefore, in actual data enhancement needs, it is recommended to use the clamping method to find the most suitable parameter value. In this case, the dataset for pedestrian re-identification was enhanced, and α=0.04 and ω=0.05 were determined by the clamping method.

[0068] Further in step 3, the implementation method of the improved flood filling algorithm of the present invention is determined. Figure 6 The effect diagram of color redistribution is given. The original image obtains two results after two color redistribution processes. It can be seen that it not only changes the appearance but also retains the color distribution characteristics of the original color blocks.

Claims

1. A data enhancement method based on image curve adjustment and color redistribution, characterized in that It includes two parts: curve adjustment and image color redistribution. The batch of image samples will be processed by two parallel operation branches, image curve adjustment and image color redistribution. Then the batch image results processed by the two branches are directly put together. The number of merged images is twice the number of batch image samples. Each image sample in the batch image sample is regarded as the original image. It specifically includes the following steps: (1) Treat each image in the batch of image samples as the original image to be processed, and perform image curve adjustment on the original image: (1.1) Use the trigonometric function shown in formula (1-1) to adjust the original image into an image curve: d=αWsin(ωh+φ) (1-1) Where W is the width of the original image, α is the horizontal offset distance scaling factor; h is the value of the original image in the height direction, ω is the period adjustment factor; φ is the offset parameter of the trigonometric function; d is the calculation result of the formula, and the image height h position will move horizontally according to the value of d; (1.2) The original image is regarded as a two-dimensional coordinate space, and the height h of the original image is taken as the independent variable, that is, as the input of formula (1-1), and the horizontal offset distance d of the original image at the height h is taken as the dependent variable, that is, the output of formula (1-1); in the two cases where d is greater than 0 or less than 0, the direction of the horizontal offset is only required to be opposite; the resulting (h, d) is the two-dimensional space coordinate of the image, and the vertical direction is recorded as the independent variable and the horizontal direction as the dependent variable, thereby completing the image curve adjustment of the original image; (2) Use the improved flood filling algorithm to redistribute the image color of the original image: (2.1) Given an original image, denoted as original image I, randomly select pixel point C on the original image I (r，c) As the starting pixel point, (r, c) is the corresponding coordinate of the pixel point in the two-dimensional space, where r represents the vertical position of the pixel point on the original image I, and c represents the horizontal position of the pixel point on the original image I; C (r，c) Represents the pixel value of the pixel point with coordinates (r, c) on the original image I; (2.2) Set the color threshold T. When the absolute value of the difference between two adjacent colors is less than the color threshold T, the two adjacent colors are considered to be connected. The color threshold T is an important parameter in the flood filling algorithm. The specific threshold calculation method is: Calculate the pixel C (r，c) The four adjacent points above, below, left and right of the center point are respectively connected to the pixel C (r，c) The pixel difference is calculated by summing up the absolute value of the pixel difference and dividing it by 2. The result is the value of the color threshold T. The calculation formula is as follows (2-1): T=(C (r+1，c) +C (r-1，c) +C (r，c+1) +C (r，c-1) -4×C (r，c) ) / 2 (2-1) (2.3) Randomly select an existing color value C on any other original image new , as the seed color to be rendered on the original image I; (2.4) Since pixel C (r，c) As the starting pixel, therefore, directly use the seed color C selected in step (2.3) new Replace C (r，c) , and record the new color value at the pixel coordinate (r, c) as N (r，c) ; (2.5) The four adjacent pixels above, below, left and right of the coordinate point (r, c) are respectively (r，c) Calculate a pixel difference dis and take its absolute value; and modify the pixel value of the corresponding adjacent pixel point according to the relationship between the difference dis and the color threshold T; Modifying the pixel value of the corresponding adjacent pixel point according to the relationship between the difference value dis and the color threshold T specifically means: if an adjacent pixel point is (r，c) If the difference dis is less than the threshold T, the new color value N is used. (r，c) Add the value of dis, and the result is the new pixel value of the adjacent pixel, and the newly assigned pixel is recorded as the processed pixel; if the difference dis is not less than the threshold T, the adjacent pixel is no longer processed and is recorded as an unprocessed pixel; (2.6) After completing the assignment operation of new pixel values ​​of the four upper, lower, left, and right adjacent pixels with the pixel coordinate (r, c) as the center point in steps (2.4) and (2.5), the processed four upper, lower, left, and right adjacent pixels will be used as the center point in turn, and the unprocessed pixel will not be used as the center point. The calculation will continue according to steps (2.4) and (2.5) until a new center point is found; each processed pixel needs to be used as a center point to process its four upper, lower, left, and right adjacent pixels, and each pixel can only be processed once and cannot be assigned a value or used as a center point twice; (2.7) After the original image I goes through steps (2.1) to (2.6), the colors of the original image I are re-rendered to obtain a picture with a new appearance.

2. The data enhancement method based on image curve adjustment and color redistribution according to claim 1, characterized in that The lateral offset distance scaling factor α in the step (1.1) is the main factor affecting the curvature degree of the original image, that is, the larger the α value, the greater the lateral curvature degree, that is, the greater the curve amplitude.

3. The data enhancement method based on image curve adjustment and color redistribution according to claim 1, characterized in that The period adjustment factor ω in the step (1.1) is the main factor affecting the degree of longitudinal curvature of the original image, that is, the larger the ω value, the smaller the degree of longitudinal curvature.

4. The data enhancement method based on image curve adjustment and color redistribution according to claim 1, characterized in that The offset parameter φ of the trigonometric function in the step (1.1) is used to dynamically adjust the position of the origin of the curve of the original image. By randomly assigning a value to φ, the randomness of the curve adjustment is increased.

Images