Image dataset with segmentation annotation repetitive target deduplication method

Abstract

The application discloses a kind of image dataset repetitive target deduplication methods with segmentation annotation, when the picture of two two matching exists with the case of partial feature overlap, the annotation file of two pictures is loaded, and each feature point pair and its coordinate pair are obtained;The coordinate offset of x-axis and y-axis is calculated by cumulative mean method, and the coordinate conversion of the annotation file of matched picture is carried out;Intersection region is obtained, the coordinate and size of deduplication mask are generated, and the corresponding region of matched picture and its annotation file are zeroed deduplication processing using deduplication mask;It is suitable for dataset repetitive calibration individual deduplication processing, valuable information is retained to the greatest extent, and redundant information is avoided, and higher quality dataset can be generated, to provide higher quality data support for subsequent deep learning model training;Automatic deduplication can be formed, compared with artificial deduplication, with the advantages of avoiding missing, not repeating, with the remarkable beneficial effects of high efficiency, accurate operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image deduplication methods, in particular to a repeated target deduplication method for image data set with segmentation annotation. BACKGROUND

[0002] At present, with the vigorous development of deep learning image technology, in the model construction and training of deep learning, the primary task is how to efficiently and accurately obtain or construct image data set, but in the process of obtaining or constructing image data set, there will inevitably be information redundancy such as repeated collection of images (or pictures, the same below) and multiple appearances of the same target individual (or target object) in multiple pictures (or images, the same below), which will cause deviation in the model training process of deep learning. Therefore, how to efficiently and accurately deduplicate images and targets in image data set has important significance.

[0003] The existing image deduplication technology mainly uses the method of matching features to deduplicate repeated pictures or similar pictures. For example, in the existing patent literature "image deduplication method, device, electronic equipment and storage medium" (application number 202210394688.0), after obtaining image features, the matching of each image to be deduplicated is performed, and after grouping, the deduplication operation of each image to be deduplicated is performed according to each image grouping. For another example, in the existing patent literature "pipeline image deduplication method, computer device, product and storage medium" (application number 202210142276.8), two similar image frames of the pipeline are obtained by frame extraction, image features are obtained based on the FPN network, and the similarity of the two images is calculated to perform image deduplication operation.

[0004] However, the above two existing technologies only perform deduplication operation on the whole picture, and do not retain the non-overlapping information between the two pictures which still has value. If the target individual is in the discarded picture and the image background still has valuable information, then while deduplicating the whole picture, the valuable information in the data set is discarded. The main reason for this limitation or defect is that the above two existing technologies only consider deduplication at the image level and do not consider retaining and utilizing valuable information under the condition of annotated images.

[0005] Therefore, the existing technology still needs to be improved and developed. SUMMARY

[0006] To solve the above technical problems, the present application provides a repeated target deduplication method for image data set with segmentation annotation, which can efficiently and accurately deduplicate images and targets in image data set.

[0007] The technical scheme of the present application is as follows: a repeated target deduplication method for an image dataset with segmentation annotation, wherein

[0008] When the partially overlapped feature exists in the pairwise matched pictures i and j, the annotation files of the pictures i and j are loaded, and the feature point pairs and their coordinate pairs of the matching pair index are obtained n , wherein represents the horizontal coordinate of the matching point in the picture i, represents the vertical coordinate of the matching point in the picture i, represents the horizontal coordinate of the matching point in the picture j, represents the horizontal coordinate of the matching point in the picture j.

[0009] The coordinate offset of the x-axis and the y-axis is calculated by the cumulative mean method dx and dy : , , wherein N represents the number of matching pairs, n represents the matching pair index, represents the absolute value of the horizontal coordinates of the two matching points, represents the absolute value of the vertical coordinates of the two matching points, and the coordinate conversion of the annotation file of the picture i is performed.

[0010] The intersection region is obtained, and the coordinates and size (X, Y, W, H) of the deduplication mask are generated, wherein X represents the horizontal coordinate of the deduplication mask and takes the minimum value of all matching point horizontal coordinates , Y represents the vertical coordinate of the deduplication mask and takes the minimum value of all matching point vertical coordinates , W represents the width of the deduplication mask and takes the value dx , and H represents the height of the deduplication mask and takes the value dy .

[0011] The deduplication mask is used for zeroing deduplication processing in the corresponding region of the picture i and its annotation file.

[0012] The repeated target deduplication method for the image dataset with segmentation annotation, wherein: first, the image dataset is down-sampled and stored in a to-be-matched image folder; at the same time, the number M of images in the to-be-matched image folder is obtained, the minimum value T_min and the maximum value T_max of the matching degree threshold T are initialized, the initial values of the matched image index i and the matching image index j are both 1, and the initial value of the number D of deleted pictures is 0.

[0013] ​The image dataset with segmentation annotation repetitive target deduplication method, wherein: judging whether the matched image index i is equal to 1? Yes, then entering the end condition judgment step; otherwise, loading the picture j, matching the picture i with the picture j and obtaining the matching degree threshold T, entering the matching and screening process, after the matching and screening, returning to the to-be-matched image sorting process, letting j = j + 1, and entering the index matching judgment step.

[0014] The image dataset with segmentation annotation repetitive target deduplication method, wherein: judging whether the matched image index j is less than i? Yes, then loading the picture j, matching the picture i with the picture j and obtaining the matching degree threshold T, entering the matching and screening process, after the matching and screening, returning to the to-be-matched image sorting process, letting j = j + 1, and returning to the index matching judgment step; otherwise, entering the end condition judgment step.

[0015] The image dataset with segmentation annotation repetitive target deduplication method, wherein: judging whether the matched image index i is less than the image quantity M? Yes, then letting i = i + 1 and j = 1, loading the picture i, and entering the initialization judgment step; otherwise, the process ends.

[0016] The image dataset with segmentation annotation repetitive target deduplication method, wherein:

[0017] When the matching degree threshold T is less than or equal to the minimum value T_min, entering the matching image sorting process to continue matching, letting j = j + 1, and returning to the index matching judgment step;

[0018] When the matching degree threshold T is between T_min and T_max, entering the deduplication process to perform a partial deduplication operation;

[0019] When the matching degree threshold T is greater than or equal to the maximum value T_max, entering the deduplication process to delete the picture and the annotation file.

[0020] The image dataset with segmentation annotation repetitive target deduplication method, wherein: when the matching degree threshold T is greater than or equal to the maximum value T_max, deleting the whole picture i and the annotation file thereof, increasing the deleted picture quantity D by 1, returning to the to-be-matched image sorting process, letting j = j + 1, and returning to the index matching judgment step.

[0021] The repeated target deduplication method of the image dataset with segmentation annotation, wherein: when the corresponding area of the picture i and the annotation file thereof are subjected to the zero deduplication processing by using the deduplication mask, it is judged whether each feature point in the picture i falls within the area of the deduplication mask, if yes, the picture i feature point mask coordinates are obtained, the deduplication mask part is removed, the picture i deduplication mask part is zeroed, and the matching image sorting process is returned to continue the matching; otherwise, the picture i deduplication mask part is directly zeroed, and the matching image sorting process is returned to continue the matching, j is set to j+1, and the index matching judgment step is returned to.

[0022] The repeated target deduplication method of the image dataset with segmentation annotation, wherein: the segmentation annotation is semantic segmentation annotation, instance segmentation annotation or panorama segmentation annotation.

[0023] The repeated target deduplication method of the image dataset with segmentation annotation, wherein: the segmentation annotation information of each target contains the classification and the endpoint coordinate group of the contour line.

[0024] The repeated target deduplication method of the image dataset with segmentation annotation provided by the application is used for screening the target individual or target object repeatedly labeled by the image collection personnel, identifying the intersection area between the images in the dataset by the SIFT algorithm, generating the deduplication mask, and zeroing the overlapping part, deleting the individual labeling information and the like, thereby forming a specific deduplication method, which is suitable for the repeated individual deduplication processing of the dataset, and generating a higher-quality dataset, thereby providing higher-quality data support for the subsequent model training of the deep learning; after the minimum value T_min and the maximum value T_max of the threshold T and the size parameters of the image downsampling are determined in the early stage, the automatic deduplication can be formed, which has the advantages of avoiding missing and repetition compared with the artificial deduplication, and has the significant beneficial effects of high efficiency and accurate operation. BRIEF DESCRIPTION OF DRAWINGS

[0025] The drawings described herein are only for the purpose of explanation, and are not intended to limit the scope of the present application in any way; the shapes and scales of the components in the drawings are only illustrative, and are used to help understand the present application, and are not specific limitations on the shapes and scales of the components of the present application; those skilled in the art can select various possible shapes and scales to implement the present application according to the specific circumstances under the guidance of the present application.

[0026] Figure 1 is a flowchart of the repeated target deduplication method of the image dataset with segmentation annotation of the application;

[0027] Figure 2 is a segmentation annotation mask schematic diagram used by the repeated target deduplication method of the image dataset with segmentation annotation of the application;

[0028] Figure 3 is a schematic diagram of the de-duplication effect of the image dataset with segmentation annotation repeated target de-duplication method embodiment of the present application. DETAILED DESCRIPTION

[0029] The specific embodiments and examples of the present application will be described in detail below with reference to the accompanying drawings, and the specific embodiments described are only used to explain the present application and are not used to limit the specific embodiments of the present application.

[0030] As Figure 1 shown, Figure 1 is a flowchart of the image dataset with segmentation annotation repeated target de-duplication method embodiment of the present application, which runs from "start" to "end", as shown in the four dashed boxes in Figure 1 , in turn, the preprocessing flow 1, the to-be-matched image sorting flow 2, the matching and screening flow 3 and the de-duplication flow 4.

[0031] Specifically, the preprocessing flow 1 includes two steps of downsampling and obtaining the number of images and initializing parameters; downsampling refers to storing the image dataset in the to-be-matched image folder after downsampling operation, which can speed up the matching speed while retaining most of the features; obtaining the number of images and initializing parameters refers to obtaining (or reading) the number M of images (or pictures, the same below) in the to-be-matched image folder, initializing the minimum value T_min and the maximum value T_max of the matching degree threshold T (i.e. the matching value in Figure 1 ), initializing the matched image (or picture, the same below) index i, the matching image index j and the number D of deleted pictures, the initial values of i and j are both 1, and the initial value of D is 0, while the initial values of T_min and T_max can reflect the limits of the matching degree threshold T, which can be obtained by debugging to adapt to different types of image datasets.

[0032] The so-called downsampling is a term relative to upsampling, also known as downsampling, which generates a thumbnail corresponding to the original image, makes the image conform to the corresponding display area, reduces the dimension of the feature and retains the effective information, to a certain extent, can avoid overfitting and keep the rotation, translation and scaling invariable; its implementation is to change an s*s "window" located on the original image into a pixel, for example, if the original image is x*y, then the size of the original image after downsampling is (x / s)(y / s), where s is the greatest common divisor of x and y; in OpenCV, the function used for downsampling is pyrDown(), which completes the downsampling of the image by performing Gaussian filtering on the image and deleting all even rows and columns.

[0033] Specifically, the to-be-matched image sorting process 2 utilizes the comparison between the matched image index i and the matching image index j and the size of the image quantity M to perform pairwise matching, which can achieve the purpose of non-repetition and non-missing matching. Further, the to-be-matched image sorting process 2 mainly includes three judgment steps: an initialization judgment step 201, an index matching judgment step 202, and an end condition judgment step 203. After loading the picture (or image, the same below) i, the initialization judgment step 201 can be entered.

[0034] More specifically, the initialization judgment step 201 judges whether the matched image index i is equal to 1. If yes, it indicates that the matched image index i should not be equal to 1 at the same time, i.e., the picture 1 should not be matched with itself, and then the end condition judgment step 203 is entered. Otherwise, the picture j is loaded, the picture i is matched with the picture j to obtain the matching degree threshold T, the matching and screening process 3 is entered, and after matching and screening, it is returned to the to-be-matched image sorting process 2, i.e., j = j + 1, and the index matching judgment step 202 is entered.

[0035] More specifically, the index matching judgment step 202 judges whether the matching image index j is less than i. If yes, it indicates that the picture corresponding to the matching image index j which is less than the matched image index i has not been matched with the picture i, the picture j is loaded, the picture i is matched with the picture j to obtain the matching degree threshold T, the matching and screening process 3 is entered, and after matching and screening, it is returned to the to-be-matched image sorting process 2, i.e., j = j + 1, and the index matching judgment step 202 is entered. Otherwise, it indicates that the picture corresponding to the j which is less than i has been matched with the picture i, and the end condition judgment step 203 is entered.

[0036] More specifically, the end condition judgment step 203 judges whether the matched image index i is less than the image quantity M. If yes, it indicates that there is still a picture i to be matched in the to-be-matched image folder, i = i + 1, j = 1, the picture i is loaded, and the initialization judgment step 201 is entered. Otherwise, it indicates that i = M at this time, all pictures i have been loaded, and through the index matching judgment step 202, the pictures corresponding to the j which is less than i have also been matched with the picture i, so as to achieve non-repetition and non-missing. Then the process is ended.

[0037] Specifically, the matching and screening process 3 compares the matching degree threshold T with the set minimum value T_min and maximum value T_max, and according to different situations, the following different operations can be performed, which can maximize the retention of valuable information and avoid redundant information.

[0038] When the matching degree threshold T is less than or equal to the minimum value T_min, it indicates that there is no repeated part between the picture i and the picture j, and then the matching image sorting process 2 is entered to continue matching, i.e., j = j + 1, and the index matching judgment step 202 is entered.

[0039] When the matching degree threshold T is between T_min and T_max, it indicates that there is partial feature overlap between the picture i and the picture j, and then the partial deduplication operation in the deduplication process 4 is entered;

[0040] When the matching degree threshold T is greater than or equal to the maximum value T_max, it indicates that the picture i and the picture j are almost identical, and then the picture and label file deletion operation in the deduplication process 4 is entered.

[0041] Specifically, the deduplication process 4 includes two operations of whole picture deduplication and partial picture deduplication.

[0042] More specifically, the whole picture deduplication refers to that when the matching degree threshold T is greater than or equal to the maximum value T_max, it indicates that the picture i and the picture j are almost identical, and then the whole picture (or image) i and the label file thereof are deleted, the deleted picture number D value is increased by 1, i.e. D=D+1, and the matching image sorting process 2 is returned, i.e. j=j+1, and the index matching judgment step 202 is returned.

[0043] More specifically, the partial picture deduplication refers to that when the matching degree threshold T is between T_min and T_max, it indicates that there is partial feature overlap between the picture i and the picture j, the label files of the pictures i and j are loaded, and the feature point pairs and the coordinate pairs of the matching pair indexes are obtained n , , wherein, represents the horizontal coordinate of the matching point in the picture i, represents the vertical coordinate of the matching point in the picture i, represents the horizontal coordinate of the matching point in the picture j, represents the horizontal coordinate of the matching point in the picture j.

[0044] The coordinate offset dx and dy of the x-axis and the y-axis are calculated by the cumulative mean method, and the specific formula is: , , wherein N represents the number of matching pairs, n represents the matching pair index, represents the absolute value of the horizontal coordinates of the two matching points, represents the absolute value of the vertical coordinates of the two matching points, and the coordinate conversion is performed on the label file of the picture i.

[0045] The intersection region is obtained, and the coordinates and size (X, Y, W, H) of the deduplication mask are generated, wherein X represents the horizontal coordinate of the deduplication mask and takes the minimum value of all matching point horizontal coordinates , Y represents the vertical coordinate of the deduplication mask and takes the minimum value of all matching point vertical coordinates , W represents the width of the deduplication mask and takes the value dx, and H represents the height of the deduplication mask and takes the value dy.

[0046] The corresponding region of the picture i and its label file are subjected to zeroing and deduplication processing by using the deduplication mask, that is, whether each feature point (or target object, the same below) in the picture i falls within the region of the deduplication mask is judged, if yes, the picture i feature point mask coordinates are obtained, the deduplication mask part is removed, the picture i deduplication mask part is zeroed, and the matching image sorting process 2 is returned to continue matching, that is, j=j+1, and the index matching judgment step 202 is returned; otherwise, the picture i deduplication mask part is directly zeroed, and the matching image sorting process 2 is returned to continue matching; for example, the target coordinates in the image i are detected, and if they overlap with the deduplication mask, the corresponding target coordinates are subjected to deduplication processing, so that the label file is processed while the image is subjected to deduplication, re-labeling is avoided, and the deduplication workload is greatly reduced.

[0047] The image dataset with segmentation and label repetition target deduplication method of the application can be used for the image dataset with segmentation and label, and can be used for the case without rotation, without scaling and only with translation, for example, the image dataset collected under the same magnification of a microscope, can be matched two by two based on the SIFT (Scale-invariant feature transform, scale-invariant feature transform) algorithm, and the matching value is obtained, and when the matching value is within the range, the matched picture and the label file are subjected to the processing of removing the intersection of the two pictures.

[0048] The SIFT algorithm is a computer vision algorithm for detecting and describing local features in images, which searches for extreme points in the spatial scale and extracts the position, scale and rotation invariant; and can be divided into the following four steps:

[0049] Scale space extreme detection: search for image positions in all scales. A Gaussian differential function is used to identify potential interest points that are invariant to scale and rotation;

[0050] Key point positioning: in each candidate position, a fine fitting model is used to determine the position and scale. The selection of key points is based on their stability. Low-contrast candidate points and edge candidate points are removed in the key point positioning step;

[0051] Direction determination: based on the gradient direction of the local image, one or more directions are assigned to each key point position. All subsequent operations on the image data are transformed relative to the direction, scale and position of the key points, thereby providing invariance to these transformations;

[0052] Key point description: the gradients of the local image in the neighborhood around each key point are measured at the selected scale. These gradients are transformed into a representation that allows comparison of large local shape deformations and illumination changes.

[0053] The segmentation annotation in the image dataset with segmentation annotation repeated target deduplication method of the application can be semantic segmentation annotation, instance segmentation annotation or panorama segmentation annotation; and the segmentation annotation information of each target can contain classification and endpoint coordinate group of contour line, for example, the coordinate annotation of a certain target is ["star", {(x1, y1), (x2, y2)…(xi, yi)}].

[0054] As shown in Figure 2 , Figure 2 is a segmentation annotation mask diagram used in the embodiment of the image dataset with segmentation annotation repeated target deduplication method of the application; (a) is a schematic diagram of an original image, in which two hexagons have overlapping parts; (b) is a schematic diagram of semantic segmentation annotation, in the segmentation annotation file thereof, the annotation information of two targets is contained, which are hexagon and heart shape respectively; (c) is a schematic diagram of instance segmentation annotation, in the segmentation annotation file thereof, the annotation information of four targets is contained, which are the first hexagon, the second hexagon, the first heart shape and the second heart shape; (d) is a schematic diagram of panorama segmentation annotation, in the segmentation annotation file thereof, in addition to the annotation information of four targets as in the instance segmentation annotation, the annotation information of background is also contained.

[0055] As shown in Figure 3 , Figure 3 is a schematic diagram of deduplication effect of the embodiment of the image dataset with segmentation annotation repeated target deduplication method of the application, before the deduplication operation, there is a repeated part between image (or picture, the same below) i and image j, and the target individual part in image i exists in the overlapping area; and after the deduplication operation, there is no repeated part between image i and image j, and the information of the non-repeated target individual part is still retained in image i.

[0056] It should be noted that the image dataset with segmentation annotation repeated target deduplication method of the application does not realize simple functions with complex steps, nor does it combine or stack conventional or simple features, so it meets the common sense of technical improvement and has practicality.

[0057] The contents not described in detail in the specification all belong to the prior art known to those skilled in the art, for example, up-sampling, down-sampling, SIFT algorithm, etc.

[0058] It should be understood that the above only describes the preferred embodiments of the application and is not intended to limit the technical solutions of the application. Those skilled in the art can add, replace, transform or improve the above description according to the spirit and principles of the application, and all the technical solutions after the addition, replacement, transformation or improvement should belong to the protection scope of the appended claims of the application.

Claims

1. A method for repeating target deduplication of image dataset with segmentation annotation, characterized in that: first, down-sampling operation is performed on the image dataset and stored in a to-be-matched image folder; at the same time, the number M of images in the to-be-matched image folder is obtained, the minimum value T_min and the maximum value T_max of the matching threshold T are initialized, the initial values of the matched image index i and the matching image index j are both 1, and the initial value of the number D of deleted images is 0; it is judged whether the matched image index i is equal to 1; if yes, the ending condition judgment step is entered; otherwise, the image j is loaded, the image i is matched with the image j to obtain the matching threshold T, the matching and screening process is entered, after the matching and screening, the to-be-matched image sorting process is returned, j is set to j+1, and the index matching judgment step is entered; when the matching threshold T is less than or equal to the minimum value T_min, the matching image sorting process is continued, j is set to j+1, and the index matching judgment step is returned; when the matching threshold T is between T_min and T_max, the partial deduplication operation is performed in the deduplication process; when the matching threshold T is greater than or equal to the maximum value T_max, the image and the annotation file are deleted in the deduplication process; dx dy dx dy the deduplication mask is used to perform zero deduplication processing on the corresponding area of the image i and the annotation file thereof; it is judged whether the matching image index j is less than i; if yes, the image j is loaded, matched with the image i to obtain the matching threshold T, the matching and screening process is entered, after the matching and screening, the to-be-matched image sorting process is returned, j is set to j+1, and the index matching judgment step is returned; otherwise, the ending condition judgment step is entered; it is judged whether the matched image index i is less than the number M of images; if yes, i is set to i+1, j is set to 1, the image i is loaded, and the initialization judgment step is entered; otherwise, the process is ended; when the matching threshold T is greater than or equal to the maximum value T_max, the whole image i and the annotation file thereof are deleted, the number D of deleted images is increased by 1, the to-be-matched image sorting process is returned, j is set to j+1, and the index matching judgment step is returned; when the deduplication mask is used to perform zero deduplication processing on the corresponding area of the image i and the annotation file thereof, it is judged whether each feature point in the image i falls within the area of the deduplication mask; if yes, the feature point mask coordinates of the image i are obtained, the deduplication mask part is removed, the deduplication mask part of the image i is set to zero, and the matching image sorting process is returned to continue matching; otherwise, the deduplication mask part of the image i is directly set to zero, the matching image sorting process is returned to continue matching, j is set to j+1, and the index matching judgment step is returned; the segmentation annotation is semantic segmentation annotation, instance segmentation annotation or panoramic segmentation annotation; the segmentation annotation information of each target contains the classification and the endpoint coordinate group of the contour line. ​ ​ ​ ​ ​ ​ When there is partial feature overlap between the two matched pictures i and j, load the annotation files of pictures i and j, obtain n a feature point pair and its coordinate pair of the matching pair index , wherein, represents the horizontal coordinate of the matching point in picture i, represents the vertical coordinate of the matching point in picture i, represents the horizontal coordinate of the matching point in picture j, represents the horizontal coordinate of the matching point in picture j; The coordinate offset of the x-axis and the y-axis is calculated by the cumulative mean method ​ and ​ : , wherein, N represents the number of matching pairs, n represents the matching pair index, represents the absolute value of the horizontal coordinates of two matching points, represents the absolute value of the vertical coordinates of two matching points, and the coordinate conversion is performed on the annotation file of the picture i; An intersection region is acquired, and a coordinate and size (X, Y, W, H) of the deduplication mask are generated, where X represents a horizontal coordinate of the deduplication mask and takes a minimum value of horizontal coordinates of all the matching points , Y represents a vertical coordinate of the deduplication mask and takes a minimum value of vertical coordinates of all the matching points , W represents a width of the deduplication mask and takes a value ​ , and H represents a height of the deduplication mask and takes a value ​ . ​ 2. The image dataset with segmentation annotation repetitive target deduplication method of claim 1, wherein: ​ 3. The image dataset with segmentation annotation repetitive target deduplication method of claim 2, wherein: ​ 4. The image dataset with segmentation annotation repetitive target deduplication method of claim 1, wherein: ​ 5. The image dataset with segmentation annotation repetitive target deduplication method of claim 4, wherein: ​ ​ 6. The image dataset with segmentation annotation repetitive target deduplication method of claim 1, wherein: ​ 7. The image dataset with segmentation annotation repetitive target deduplication method of claim 1, wherein: ​

Images