Data enhancement image reannotation method based on logistic regression

Abstract

The invention discloses a data enhancement image relabeling method based on logistic regression, which comprises the following steps of: intercepting a sub-graph in a labeled data set by constructing a sliding window, constructing a training sample set by taking color values of pixel points in each frame of the sub-graph as feature data and taking color values of a central pixel point as labels, and carrying out image relabeling on the trained sample set; training the established first prediction model by adopting the training sample set; the method comprises the following steps: acquiring an original size and a first annotation box of a to-be-processed picture, performing image enhancement on the to-be-processed picture to obtain a target picture and a second annotation box, filling a missing position in the target picture by adopting a first prediction model, acquiring a target contour in the second annotation box, and updating the second annotation box with a new target contour to complete re-annotation of the picture. The workload of picture relabeling after image enhancement is effectively reduced, so that the training efficiency of the model is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a data enhancement image re-annotation method based on logistic regression. Background Art

[0002] Image annotation refers to adding labels, bounding boxes, key points and other annotation information to target objects or areas in an image manually, semi-automatically or automatically to help computers understand and identify image content, thereby providing training data for subsequent image classification, target detection, semantic segmentation and other tasks. Annotation types can be divided into classification annotation, target detection annotation, semantic segmentation annotation, instance segmentation annotation and key point annotation. Among them, target detection annotation refers to using bounding boxes to select target objects in an image and mark the category of the object. For example, in a picture containing a car and a pedestrian, different bounding boxes are used to frame the car and pedestrian, and labels such as car and pedestrian are marked. Usually, the coordinate information of the bounding box needs to be marked, such as the coordinates of the upper left corner and the lower right corner.

[0003] In order to solve the problem of insufficient data or increase data diversity during model training, it is usually necessary to perform image enhancement processing on sample images, such as rotation, flipping, scaling, cropping, brightness adjustment, etc., to generate more training samples and expand the size of the data set. However, when performing data enhancement processing on images with labeled information, the shape and boundaries of the target in the image after rotation and enhancement often change, and the original rectangular annotation box may not fit the target perfectly, which will cause annotation deviation. Using such sample training models will have an adverse effect on the training results. In addition, if you choose to re-label the target position of the image after image enhancement, it will significantly increase the workload, thereby reducing the training efficiency of the model. Summary of the invention

[0004] In view of this, the present invention provides a data-enhanced image re-annotation method based on logistic regression, which realizes efficient re-annotation of annotated pictures after image enhancement.

[0005] The present invention provides a data enhancement image re-annotation method based on logistic regression, which specifically includes the following steps:

[0006] Step 1: Determine the sliding window according to the ratio of the size of the annotation box of the image in the annotated data set to the size of the image, use the sliding window to slide and intercept the sub-image in the image, obtain the color value of the central pixel of the sub-image and the pixel points in each frame, and use the color value of the pixel points in each frame as the feature data and the color value of the central pixel point as the label to construct a training sample set; establish a first prediction model based on the logistic regression model, and use the training sample set to train the first prediction model;

[0007] Step 2: obtain the original size, the first annotation frame and the first sliding window of the image to be processed, obtain the target image and the second annotation frame after image enhancement of the image to be processed, take the missing position in the target image as the pixel to be filled, take the pixel to be filled as the center pixel of the first sliding window to obtain the color value of the pixel in each frame, input the color value into the first prediction model to obtain the predicted color value, and use the predicted color value to fill the pixel to be filled;

[0008] Step 3, repair the out-of-bounds pixels of the second annotation box, use the first sliding window to intercept a sub-image in the picture determined by the second annotation box, obtain the pixel intensity and pixel direction from the average color of the pixels in the sub-image border, determine the target contour pixels according to the pixel intensity and pixel direction, and use the maximum and minimum values ​​of the horizontal and vertical coordinates to correct the four sides of the second rectangular box to form a rectangular box as a new annotation box to complete the re-annotation.

[0009] Furthermore, the method of using a sliding window to slide and capture a sub-image in the picture in step 1 is: sliding and capturing a sub-image in the picture with a set step size.

[0010] Furthermore, the set step size is 1 pixel.

[0011] Furthermore, the training sample set in step 1 is normalized data.

[0012] Furthermore, the sliding window is expressed as:

[0013]

[0014] in, T is the ratio of the annotation box size to the image size.

[0015] Furthermore, the second annotation box is calculated by obtaining the coordinates of the four vertices of the first annotation box after image enhancement, subtracting the horizontal and vertical coordinates of the four vertices in pairs and taking the absolute values ​​to form four new vertex coordinates, and the horizontal rectangle determined by the four new vertex coordinates is the second annotation box.

[0016] Furthermore, the method of repairing the out-of-bounds pixels of the second annotation box in step 3 is: if there are out-of-bounds pixels, the vertex coordinates of the second annotation box are modified to zero when there are negative values, and modified to the maximum coordinate value when they are greater than the maximum coordinate value.

[0017] Furthermore, the maximum coordinate value is determined by the original size.

[0018] Furthermore, the method for obtaining the pixel intensity and pixel direction from the color average value of the pixels within the sub-image border in step 3 is: summing the square of the horizontal pixel mean difference and the square of the vertical pixel mean difference, and then taking the square root of the sum, and the calculation result is the pixel intensity; the arc tangent angle of the ratio of the absolute value of the horizontal pixel mean change to the absolute value of the vertical pixel mean change is the pixel direction. Beneficial Effects

[0019] The present invention constructs a sliding window to intercept a sub-image in a labeled data set, uses the color values ​​of pixels in each frame of the sub-image as feature data and the color value of the central pixel as a label to construct a training sample set, and uses the training sample set to train the established first prediction model; obtains the original size of the image to be processed and the first annotation frame, then performs image enhancement on the image to be processed to obtain a target image and a second annotation frame, uses the first prediction model to fill in the missing position in the target image, then obtains the target contour in the second annotation frame, and updates the second annotation frame with the new target contour to complete the re-annotation of the image, effectively reducing the workload of re-annotation of the image after image enhancement, thereby improving the training efficiency of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic flow chart of a data enhancement image re-annotation method based on logistic regression provided by the present invention. DETAILED DESCRIPTION

[0021] The present invention is described in detail below with reference to the accompanying drawings and with reference to the embodiments.

[0022] The present invention provides a data enhancement image re-annotation method based on logistic regression, the core idea of ​​which is: intercepting a sub-image in an annotated data set by constructing a sliding window, constructing a training sample set with the color values ​​of the pixels in each frame of the sub-image as feature data and the color value of the central pixel as a label, and using the first prediction model established by training the training sample set; obtaining the original size of the image to be processed and the first annotation frame, then performing image enhancement on the image to be processed to obtain a target image and a second annotation frame, using the first prediction model to fill in the missing position in the target image, and then obtaining the target contour in the second annotation frame, and updating the second annotation frame with the new target contour to complete the re-annotation of the image.

[0023] The present invention provides a data enhancement image re-annotation method based on logistic regression, the specific process is as follows Figure 1 As shown, the specific steps include:

[0024] Step 1: For the images in the labeled data set, the sliding window is determined according to the ratio of the annotation box size to the image size. The sliding window is used to slide and capture the sub-image in the image with a set value as the step size, and the color value of the central pixel of the sub-image and the pixel points in each border are obtained. The color value of the pixel points in each border is used as the feature data, and the color value of the central pixel point is used as the label to construct a training sample set.

[0025] In order to further improve the model training effect, the color values ​​in the training sample set are normalized. In order to increase the amount of feature data, the present invention sets the value to 1 pixel and uses it as the step size to move the sliding window.

[0026] The ratio of the size of the annotation frame to the size of the picture in the present invention T The calculation method is:

[0027]

[0028] Among them, Width is the width of the image, Height is the height of the image, xmax and ymax are the coordinates of the vertex of the lower right corner of the annotation box, and xmin and ymin are the coordinates of the vertex of the upper left corner of the annotation box.

[0029] One way to determine the sliding window based on the ratio of the annotation box size to the image size is:

[0030]

[0031] Specifically, if the threshold T If the threshold is not less than 20, it means that the marked target size is small and the amount of feature data is small, so a smaller sliding window is needed to extract more information and generate a 3*3 sliding window. T If the value of is between 5 and 20, it means that the marked target size is moderate and the feature data volume is moderate. Therefore, a standard size sliding window can be used to extract information and generate a 6*6 sliding window. T If it is not greater than 5, it means that the marked target size is large and the amount of feature data is large. Therefore, a larger sliding window can be used to extract feature data to improve calculation efficiency and generate a 9*9 sliding window.

[0032] In addition, the present invention can also adjust the size of the sliding window according to actual needs.

[0033] Step 2: Establish a first prediction model based on the logistic regression model, and predict the color value of the central pixel according to the color value of the pixel in the frame; and use the training sample set to complete the training of the first prediction model.

[0034] Step 3, take the image with target detection annotation as the image to be processed, obtain the original size of the image to be processed, the annotated target category and the coordinates of the first annotation box; perform image enhancement on the image to be processed to obtain the target image, form the second annotation box after image enhancement, take the position where the pixel is missing or the abnormal hole appears in the target image as the pixel to be filled, determine the sliding window of the target image according to the original size and the first annotation box as the first sliding window, take the pixel to be filled as the center pixel of the first sliding window to obtain the color value of the pixel in each frame, input the color values ​​of these pixel points into the first prediction model to obtain the predicted color value of the pixel to be filled, and use the predicted color value to fill the pixel to be filled.

[0035] Among them, the second annotation box formed after the first annotation box is enhanced by the image is calculated as follows: obtain the coordinates of the four vertices of the first annotation box after the image enhancement, subtract the horizontal and vertical coordinates of the four vertices from each other and take the absolute values ​​to form four new vertex coordinates, and the horizontal rectangle determined by the four new vertex coordinates is the second annotation box.

[0036] Step 4, determine whether the second annotation box is out of bounds. If so, modify the vertex coordinates of the second annotation box to zero when there are negative values, and modify them to the maximum coordinate value when they are greater than the maximum coordinate value, thereby obtaining the modified second annotation box; if not, keep the second annotation box unchanged.

[0037] Among them, the maximum coordinate value is the maximum horizontal and vertical coordinate value corresponding to the original size.

[0038] Step 5: Slide and intercept a sub-image on the picture in the second annotation frame with a first sliding window with a step size of 1 pixel and a pixel point in the second annotation frame as the center pixel point, calculate the color average value of the pixel points in each frame of the sub-image to form the upper mean value, bottom mean value, right mean value and left mean value of the sub-image, and then calculate the pixel intensity and pixel direction of the central pixel point of the sub-image according to the upper mean value, bottom mean value, right mean value and left mean value of the sub-image, and complete the calculation of the pixel intensity and pixel direction of all pixels in the second annotation frame; traverse the pixels in the second annotation frame to obtain the surrounding adjacent pixels, compare the pixel intensity and pixel direction of the pixel point with the adjacent pixels, when there are continuous pixels whose pixel intensity exceeds the number threshold and is greater than the pixel intensity of the pixel point, and the difference in the pixel direction is less than the direction difference threshold, these continuous pixels are the target contour pixels; form a target contour pixel point set from the target contour pixels of all pixels, and obtain the maximum and minimum values ​​of the horizontal and vertical coordinates of all pixels in the target contour pixel point set.

[0039] The pixel intensity is calculated by summing the square of the horizontal pixel mean difference and the square of the vertical pixel mean difference, and then taking the square root of the sum. The calculated result is the pixel intensity.

[0040] The calculation method of the pixel direction is: the arc tangent angle of the ratio of the absolute value of the horizontal pixel mean change to the vertical pixel mean change is the pixel direction.

[0041] Step 6: After obtaining the maximum and minimum values ​​of the horizontal and vertical coordinates from the target contour pixel point set, correct the four sides of the second rectangular box, and finally obtain a rectangular box that fits the target contour, which is the new annotation box of the image after image enhancement, completing the re-annotation of the image after image data enhancement.

[0042] Example:

[0043] In this embodiment, a data enhancement image re-annotation method based on logistic regression provided by the present invention is adopted to realize the re-annotation of the data set after image enhancement, including the following steps:

[0044] S1. For the dataset that needs to perform image enhancement, read the target annotation information in the corresponding annotation file according to the dataset annotation format. For the target detection type, read the target category such as name under object, the annotation box information such as bndbox under object, and the image size such as size. The specific format is as follows:

[0045] <?xml version="1.0" encoding="utf-8"?>

[0046] <annotation>

[0047] <source>

[0048] optic rs image

[0049] <annotation>Lmars RSDS2016< / annotation>

[0050] <flickrid> 0< / flickrid>

[0051] <database>Lmars Detection Dataset of RS< / database>

[0052]

[0053] <object>

[0054] <bndbox>

[0055] <xmin> 937< / xmin>

[0056] <ymin> 85< / ymin>

[0057] <ymax> 160< / ymax>

[0058] <xmax> 1033< / xmax>

[0059] < / bndbox>

[0060] <difficult> 0< / difficult>

[0061] <pose>Left< / pose>

[0062] <name>aircraft< / name>

[0063] <truncated> 1< / truncated>

[0064] < / object>

[0065] <filename>aircraft_4.jpg< / filename>

[0066] <segmented> 0< / segmented>

[0067] <owner>

[0068] <name>Lmars, Wuhan University< / name>

[0069] <flickrid>I do not know< / flickrid>

[0070] < / owner>

[0071] <folder>RSDS2016< / folder>

[0072] <size>

[0073] <width> 1044< / width>

[0074] <depth> 3< / depth>

[0075] <height> 915< / height>

[0076] < / size>

[0077] < / annotation>

[0078] S2. Compare the image to be enhanced in the dataset with the target annotation box and the image size to find the appropriate sliding box size , because the target size is positively correlated with the feature data extracted by the sliding window. If the threshold T If the threshold is not less than 20, the target size is small and the feature data is small. A smaller sliding window is needed to extract more information and generate a 3*3 sliding window. T In the range of 5 to 20, the target size is normal, the feature data is normal, and a standard size sliding window is needed to extract information and generate a 6*6 sliding window; if the threshold T If it is not greater than 5, the target size is large and the feature data is large. A large-size sliding window can be used to extract the feature data, improve the calculation efficiency, and generate a 9*9 sliding window.

[0079] Select an appropriate sliding window, slide each pixel with a step size of 1 pixel, and intercept all pixels on the four sides. If the four sides exceed the image size, they are recorded as 0. Then calculate the pixel values ​​of the four boundaries of the top, bottom, left and right respectively. The mean and absolute value of , x is the number of pixels, as shown in the following formula:

[0080]

[0081] In the target annotation box area, the sliding box intercepts all pixels with a step size of 1 pixel and saves them into the pixel feature column, which is recorded as , by increasing the feature information in the region to improve the model training effect, which is used for subsequent pixel filling model training, is the color value of the center pixel. Taking a 6*6 sliding box as an example, the data is as follows:

[0082] <![CDATA[bottom_value1]]> <![CDATA[bottom_value16]]> <![CDATA[up_value11]]> <![CDATA[up_value16]]> <![CDATA[left_value11]]> <![CDATA[left_value16]]> <![CDATA[right_value11]]> <![CDATA[right_value16]]> pixel 44 … 125 37 … 26 201 … 54 188 … 77 50 … … … … … … … … … … … … 99 75 … 33 66 … 86 103 … 175 90 … 100 107

[0083] Due to the uneven distribution of data, the training convergence is poor. It is necessary to normalize all the data in the table except the last column to increase the effect of model training. Then all the data are input into the logistic regression model for training to generate a trained logistic regression model.

[0084] Calculate the square of the horizontal pixel mean difference and the vertical pixel mean difference, sum them up and then take the square root to get the pixel intensity , as shown below:

[0085]

[0086] Find the direction of the pixel The inverse tangent of the ratio of the horizontal rate of change to the vertical rate of change is as follows:

[0087]

[0088] After calculating the pixel intensity and pixel direction of all pixels in the sliding window, traverse all pixels and compare the pixel intensity of the 8 pixels around the pixel with the pixel intensity of the pixel. If the pixel intensity of more than 3 consecutive pixels is greater than the pixel intensity of the pixel, compare the direction of the pixel with these pixels. If the difference in the direction of the pixels is no more than 20°, the 3 consecutive pixels are considered to be the outline of the target.

[0089] Similarly, all pixels in the target annotation box are processed as above to obtain the target contour pixel set. The contour pixel set is compared horizontally from left to right to find the maximum and minimum values, which are , and then compare the contour pixel set vertically from top to bottom to find the maximum and minimum values, which are .

[0090] S3. Perform data enhancement on the image. Ordinary enhancement methods have no effect on the enhancement of labels. You only need to copy the content in the label file. However, when using enhancement methods that change the pixel position such as rotation and translation, copying the content in the label will cause incorrect annotation information of the enhanced image. The processing method of rotation enhancement is the most complicated. Taking rotation enhancement as an example, you first need to rotate the image, that is, rotate all the pixels in the image. The pixel position , rotation angle , the pixel position after rotation , calculated using the following formula:

[0091]

[0092]

[0093] After all the pixels are calculated according to the above formula, and The pixel point at the position is replaced by the rotated one and If there is a pixel missing, the sliding window is used to take the values ​​of all the pixels above, below, left and right around the pixel as the center, and the surrounding color values ​​are sorted using the above data processing method, and then input into the logistic regression model for inference. Fill the missing parts so that the filled pixel values ​​more realistically reflect the surrounding features. If there are out-of-bounds parts, such as negative numbers, larger than the width and height of the original image, and the position coordinates of the out-of-bounds pixels are negative numbers, that is, the rotated pixels are out of bounds at the left and upper boundaries of the enhanced image, then the negative coordinate values ​​are set to 0; if the out-of-bounds pixels are larger than the width and height of the original image, that is, the rotated pixels are out of bounds at the lower and right boundaries of the enhanced image, then the width and height of the original image are taken to complete the rotation of all pixels in the image.

[0094] S4. The four vertices of the target annotation information are also rotated, and the coordinates of the four points after rotation are obtained and recorded as , , , , then every two points and Subtract the absolute value to get the new point coordinates ; and perform out-of-bounds processing according to the above method, and then draw these four points to form a horizontal rectangular frame, which is the second annotation frame, and its border information is , , and then obtain the target contour in the annotation box, and use the maximum and minimum values ​​of the upper, lower, left and right contour pixel points of the target to correct the second annotation box. The specific correction method is to use the minimum value of the left contour and the upper contour Replace to the second callout box , the maximum value of the right contour and the lower contour Replace to the second callout box , so that the corrected annotation box fits the target contour, and the new annotation information is updated to the corresponding annotation file , , , In. That is the annotation information obtained by re-annotation.

[0095] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data augmentation image re-annotation method based on logistic regression, characterized in that: The specific steps include: Step 1: Determine the sliding window according to the ratio of the size of the annotation box of the image in the annotated data set to the size of the image, use the sliding window to slide and intercept the sub-image in the image, obtain the color value of the central pixel of the sub-image and the pixel points in each frame, and use the color value of the pixel points in each frame as the feature data and the color value of the central pixel point as the label to construct a training sample set; establish a first prediction model based on the logistic regression model, and use the training sample set to train the first prediction model; Step 2: obtain the original size, the first annotation frame and the first sliding window of the image to be processed, obtain the target image and the second annotation frame after image enhancement of the image to be processed, take the missing position in the target image as the pixel to be filled, take the pixel to be filled as the center pixel of the first sliding window to obtain the color value of the pixel in each frame, input the color value into the first prediction model to obtain the predicted color value, and use the predicted color value to fill the pixel to be filled; Step 3, repair the out-of-bounds pixels of the second annotation box, use the first sliding window to intercept a sub-image in the picture determined by the second annotation box, obtain the pixel intensity and pixel direction from the average color of the pixels in the sub-image border, determine the target contour pixels according to the pixel intensity and pixel direction, and use the maximum and minimum values ​​of the horizontal and vertical coordinates to correct the four sides of the second rectangular box to form a rectangular box as a new annotation box to complete the re-annotation.

2. The data-enhanced image re-annotation method according to claim 1, characterized in that: The method of using a sliding window to slide in the picture to capture a sub-image in step 1 is: sliding in the picture to capture a sub-image with a set step size.

3. The data-enhanced image re-annotation method according to claim 2, characterized in that: The set step size is 1 pixel.

4. The data-enhanced image re-annotation method according to claim 1, characterized in that: The training sample set in step 1 is normalized data.

5. The data-enhanced image re-annotation method according to claim 1, characterized in that: The expression of the sliding window is: in, T is the ratio of the annotation box size to the image size.

6. The data-enhanced image re-annotation method according to claim 1, characterized in that: The second annotation frame is calculated by obtaining the coordinates of the four vertices of the first annotation frame after image enhancement, subtracting the horizontal and vertical coordinates of the four vertices in pairs and taking the absolute values ​​to form four new vertex coordinates, and the horizontal rectangle determined by the four new vertex coordinates is the second annotation frame.

7. The data-enhanced image re-annotation method according to claim 1, characterized in that: The method of repairing the out-of-bounds pixels of the second annotation box in step 3 is: if there are out-of-bounds pixels, the vertex coordinates of the second annotation box are modified to zero when there are negative values, and modified to the maximum coordinate value when they are greater than the maximum coordinate value.

8. The data-enhanced image re-annotation method according to claim 7, characterized in that: The maximum coordinate value is determined by the original size.

9. The data-enhanced image re-annotation method according to claim 1, characterized in that: The method for obtaining the pixel intensity and pixel direction from the color average value of the pixels within the sub-image border in step 3 is: sum the square of the horizontal pixel mean difference and the square of the vertical pixel mean difference, and then take the square root of the sum, and the calculation result is the pixel intensity; the arc tangent angle of the ratio of the absolute value of the horizontal pixel mean change to the absolute value of the vertical pixel mean change is the pixel direction.

Images