A target object automatic labeling method and system

Abstract

The application discloses a kind of target object automatic labeling method and system, including steps: S1, obtain the background color image containing only background board;S2, obtain the color image and depth image containing target object;S3, the depth image containing target object is pre-labeled to obtain pre-labeled target image;S4, based on the background color image and the color image containing target object, pixel-level labeling is carried out to the pre-labeled target image to obtain pixel-level labeled target image.The application realizes the automatic labeling of target object by combining RGB color image and depth image, can obtain pixel-level labeling without manual labeling, improves the efficiency of labeling, reduces labeling cost.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and system for automatic annotation of target objects. Background Technology

[0002] In the process of developing deep learning models, sufficient datasets and accurate data annotations are crucial to the final model performance. Both deep learning-based product detection and segmentation require a large amount of annotation.

[0003] Taking a product detection model as an example, product detection annotation involves drawing rectangles around each relevant product and providing information about the product's corresponding category. Depending on the requirements, users can choose to use axis-aligned rectangles or free rectangles as bounding boxes for annotation data. Through annotation, imported images can be easily assigned to the corresponding product label classes. Product detection model projects involve a vast amount of data. Faced with a massive dataset of products, developers can only manually annotate each image, a time-consuming process. Furthermore, it requires training for annotators, and manual operation is prone to errors, leading to incorrect annotations. Therefore, the overall operational cost is very high, requiring developers to spend a significant amount of time on data annotation.

[0004] Currently, all existing pixel-level image annotation methods are done manually, starting from scratch to annotate the original image. This completely manual annotation method is extremely inefficient, costly, and requires a lot of manpower and financial resources.

[0005] Another non-manual annotation method is through synthetic data, which is generated by computer. It is a promising alternative to manual annotation, but synthetic data differs from real data, making it difficult to achieve the desired effect and output images with realistic results.

[0006] The above background information is provided only to aid in understanding the inventive concept and technical solution of this invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above information was disclosed on the filing date of this patent application, the above background information should not be used to evaluate the novelty and inventiveness of this application. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for automatic annotation of target objects, so as to solve at least one of the problems in the background art described above.

[0008] To achieve the above objectives, the technical solution of this invention is implemented as follows:

[0009] An automatic object annotation method includes the following steps:

[0010] S1. Obtain a background color image containing only the background panel;

[0011] S2. Obtain a color image and a depth image containing the target object;

[0012] S3. Pre-annotate the depth image containing the target object to obtain a pre-annotated target image;

[0013] S4. Based on the background color image and the color image containing the target object, perform pixel-level annotation on the pre-annotated target image to obtain a pixel-level annotated target image.

[0014] In some embodiments, step S2 includes:

[0015] The target object is placed between the background panel and the RGBD camera; wherein the distance D1 between the RGBD camera and the target object is greater than the distance D0 between the target object and the background panel;

[0016] The RGBD camera acquires color and depth images containing the target object.

[0017] In some embodiments, step S3 includes:

[0018] Iterate through each pixel of the depth image containing the target and read its pixel value;

[0019] Determine whether the pixel value is within the range D0 to D1. If so, mark the corresponding pixel as the target pixel and then obtain the pre-labeled target image.

[0020] In some embodiments, step S4 includes:

[0021] The pre-annotated target image is subjected to erosion processing to obtain the inner edge of the target object;

[0022] The inner edge of the target object is expanded to obtain the outer edge of the target object, wherein the region between the inner edge and the outer edge is the edge region;

[0023] Traverse each pixel p in the edge region, and the pixels p1 and p2 in the background color image and the color image containing the target object respectively corresponding to the pixel p, to obtain the effective edge region;

[0024] The pixels contained in the inner edge and / or the pixels contained in the effective edge region are defined as the target object pixels.

[0025] In some embodiments, a bracket is provided between the background panel and the RGBD camera for placing the target object; the background panel is a solid color background panel.

[0026] In some embodiments, in step S4, a threshold is set. If the pixel values ​​of pixels p1 and p2, which are respectively located at the positions corresponding to pixel p in the background color image and the color image containing the target object, are both greater than the threshold, and the pixel value of pixel p is greater than 0, then pixel p is determined to be a target object pixel, and the area formed by the target object pixels is the effective edge area.

[0027] In some embodiments, the steps further include:

[0028] S5. Replace the background board or change the ambient lighting to record multiple background color images so that the same set of annotations corresponds to different images.

[0029] Another technical solution of this invention is:

[0030] An automatic target object annotation system includes: a background panel, an RGBD camera, and a support disposed between the background panel and the RGBD camera; wherein the support is used to place the target object; the RGBD camera is used to record a background color image containing only the background panel and a color image and a depth image of the target object, and transmits the recorded image to an external processor, which processes the target object automatic annotation method according to any one of claims 1-7.

[0031] In some embodiments, when the target object is placed on the bracket, the distance D1 between the RGBD camera and the target object is greater than the distance D0 between the target object and the background plate.

[0032] In some embodiments, the background panel is a solid color background panel.

[0033] The beneficial effects of the technical solution of this invention are:

[0034] Compared with existing technologies, this invention achieves automatic annotation of target objects by combining RGB color images and depth images, obtaining pixel-level annotations without manual annotation, thus improving annotation efficiency and reducing annotation costs. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1This is a flowchart illustrating an automatic target object annotation method according to an embodiment of the present invention;

[0037] Figure 2 This is a schematic diagram of the process of processing pre-labeling according to an automatic target object labeling method of the present invention;

[0038] Figure 3 This is a schematic diagram of an automatic product labeling system according to another embodiment of the present invention. Detailed Implementation

[0039] To make the technical problems, technical solutions, and beneficial effects of the embodiments of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0040] It should be noted that when a component is referred to as "fixed to" or "set on" another component, it can be directly on or indirectly on that other component. When a component is referred to as "connected to" another component, it can be directly connected to or indirectly connected to that other component. Furthermore, a connection can be for both fixing and circuit connection purposes.

[0041] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention.

[0042] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0043] Reference Figure 1 As shown, Figure 1 The diagram shown is a flowchart of an automatic target object annotation method according to an embodiment of the present invention. The method includes the following steps:

[0044] S1. Obtain a background color image containing only the background panel;

[0045] A background color image was captured using a fixed RGBD camera in a scene without any foreground objects. For ease of description, this background color image will be denoted as ImgB. For details, refer to... Figure 2 As shown, in some embodiments, a background panel 20 is set up, and an RGBD camera 21, which is horizontally fixed, is used to capture the background panel 20, obtaining a background color image ImgB containing only the background panel. The background panel 20 is a solid-color background panel with a large color difference from the target object, such as cardboard. In some embodiments, a support 22 is provided between the background panel 20 and the RGBD camera 21 for placing the target object 23. When capturing the background color image ImgB, it is necessary to ensure that there are no foreground objects in the scene.

[0046] Reference Figure 2 As shown, when the target object 23 is placed on the support 22, the distance D1 between the RGBD camera 21 and the target object 23 is greater than the distance D0 between the target object 23 and the background plate 20; wherein, the distance D0 between the target object 23 and the background plate 20 is greater than 100mm. In one embodiment, the distance D0 between the target object 23 and the background plate 20 is 500mm, and the distance D1 between the RGBD camera 21 and the target object 23 is 1000mm.

[0047] S2. Obtain a color image and a depth image containing the target object 23;

[0048] The target object 23 is placed between the background plate 20 and the RGBD camera 21. A color image and a depth image containing the target object 23 are acquired using the RGBD camera 21. The distance D1 between the RGBD camera 21 and the target object 23 is greater than the distance D0 between the target object 23 and the background plate 20. Specifically, the target object 23 is placed on a support 22, and under uniform lighting conditions, a fixed RGBD camera 21 is used to capture a color image and a depth image of the target object within the scene. During shooting, it is necessary to avoid the RGBD camera 21 capturing the support 22. The color image acquired in step S2 is denoted as ImgC.

[0049] S3. Based on the depth image containing the target object obtained in step S2, perform pre-annotation to obtain a pre-annotated target image;

[0050] Traverse each pixel of the depth image containing the target obtained in step S2 and read its pixel value. Determine whether the pixel value is within the range D0 to D1. If so, label the corresponding pixel as the target pixel and obtain the pre-labeled target image. Denote the pre-labeled target image as R1.

[0051] Specifically, in this embodiment of the invention, each pixel of the depth image containing the target is traversed by array traversal, pointer traversal, or iterator traversal.

[0052] During target object annotation, the rich colors of the target object can lead to situations where some areas of the target object are similar in color to the background. Directly using RGB color thresholds for annotation can easily result in missed detections. Furthermore, RGBD cameras typically exhibit salt-and-pepper noise, making false detections more likely when using RGB color thresholds directly. This invention uses a target object depth map for pre-annotation, thus avoiding missed and false detections.

[0053] S4. Perform pixel-level annotation on the pre-annotated target image based on the background color image and the color image containing the target object to obtain the pixel-level annotated target image.

[0054] Generally, RGBD cameras, such as structured light and TOF cameras, have the defect of inaccurate image edges, so it is necessary to optimize the edges of the pre-labeled target image R1.

[0055] In this embodiment of the invention, step S4 specifically includes the following steps:

[0056] S40. Perform erosion processing on the pre-annotated target image R1 to obtain the inner edge of the target object;

[0057] Specifically, the pre-annotated target image R1 is subjected to erosion processing, and n rounds of erosion processing are performed to obtain the inner edge of the target object, denoted as R2; in some embodiments, the erosion processing of the pre-annotated target image R1 includes:

[0058] A 3×3 structure matrix is ​​selected, with all elements being 1. Using a step size of 1, each pixel in the pre-labeled target image R1 is scanned. A logical AND operation is performed between the structure matrix and the pre-labeled target image R1. If both the structure matrix and the pre-labeled target image R1 have a value of 1, then the pixel at that point on the inner edge R2 of the target object is output as 1; otherwise, the pixel on the inner edge R2 of the target object is output as 0. This erosion process is repeated n times to eliminate the boundary points of the pre-labeled target image R1, causing the boundary to shrink inwards. This reduces the size of the pre-labeled target image R1 by n rounds, yielding the inner edge R2 of the target object. It should be noted that the structure matrix can be selected in different sizes according to actual needs, such as 3×3, 5×5, 7×7, etc. Generally, different sizes of the structure matrix will result in different extracted image edge features. Larger structure matrices can filter out large noise points but will miss many detailed features of the image; conversely, smaller structure matrices can retain detailed features of the image but have poor suppression effect on large noise. In this embodiment of the invention, a 3×3 structure matrix is ​​selected, but this should not be taken as evidence that the invention is limited to this structure matrix.

[0059] In one embodiment, the erosion process is performed using the erode function, i.e.:

[0060] dst=cv2.erode(src,kernel,iterations)

[0061] Where src represents the pre-labeled target image, kernel represents the convolution kernel (structure matrix), and iterations represents the number of iterations. The number of iterations refers to the number of erosion cycles performed, which is done multiple times as needed, i.e., multiple rounds of erosion.

[0062] S41. The inner edge R2 of the target object obtained in step S40 is expanded to obtain the outer edge of the target object, wherein the area between the inner edge and the outer edge is the edge area.

[0063] The inner edge R2 of the target object is expanded. After 2n rounds of expansion, the outer edge of the target object is obtained, denoted as R3. The difference between the inner edge R2 and the outer edge R3 is calculated as R3-R2, thus obtaining the edge region.

[0064] Specifically, the expansion process for the inner edge R2 of the target object includes:

[0065] Based on the inner edge R2 of the target object, corresponding to the erosion process, a 3×3 structure matrix is ​​selected, where all elements in the matrix are 1. A logical AND operation is performed between this structure matrix and the inner edge R2 of the target object. If the values ​​of the structure matrix and the inner edge R2 of the target object are both 0, then the pixel of that point on the outer edge R3 of the target object is output as 0. Otherwise, the pixel of the outer edge R3 of the target object is output as 1. This dilation process is repeated for 2n rounds, merging all background points that come into contact with the inner edge R2 of the target object into the inner edge R2 of the target object, causing the boundary of the inner edge R2 of the target object to expand outward to fill the holes in the inner edge R2 of the target object, thus obtaining the outer edge R3 of the target object.

[0066] In one embodiment, the dilate function is used for dilation, i.e.:

[0067] dst=cv2.dilate(src,kernel,iterations)

[0068] Where src represents the inner edge R2 of the target object, kernel represents the convolution kernel (structure matrix), and iterations represents the number of iterations of dilation. Multiple iterations are performed as needed, i.e., multiple rounds of dilation are performed.

[0069] S42. Traverse each pixel p in the edge region, as well as pixels p1 and p2 in the background color image and the color image containing the target object, respectively, at the positions corresponding to pixel p, to obtain the effective edge region.

[0070] Specifically, a threshold is set, and each pixel p in the edge region, as well as pixels p1 and p2 in the background color image and the color image containing the target object, are traversed. If the pixel values ​​of p1 and p2 are both greater than the threshold, and the pixel value of p is greater than 0, then the pixel p is determined to be a target object pixel, and the area formed by the target object pixels is a valid edge region.

[0071] S43. Pixels contained in the inner edge and / or pixels contained in the effective edge region are defined as the target object pixels.

[0072] In one embodiment, the following steps are also included:

[0073] S5. Record more data based on the same set of annotations; specifically, this includes:

[0074] S50: Change the background panel and record a color background image; or...

[0075] S51. Change the ambient lighting and record a background color image, such as illuminating the product with lights of different intensities from different angles.

[0076] By creating realistic data augmentations for the target object and background (such as diverse backgrounds, realistic shadow effects, and diverse lighting conditions) while keeping the annotations unchanged, the same set of annotations can correspond to different images, thus enriching the dataset.

[0077] In one embodiment, the following steps are also included:

[0078] S6. Change the target object and repeat steps S1-S4 to obtain a dataset with a large amount of data.

[0079] The automatic object annotation method of this invention can obtain pixel-level annotations without manual annotation, and can match a large number of backgrounds and lighting scenes for the same object to record more data and enrich the dataset.

[0080] Reference Figure 2As shown, an automatic target object annotation system 200, as an embodiment of the present invention, includes a background plate 20, an RGBD camera 21, and a support 22 disposed between the background plate 20 and the RGBD camera 21. The support 22 is used to place a target object 23. When the target object 23 is placed on the support 22, the distance D1 between the RGBD camera 21 and the target object 23 is greater than the distance D0 between the target object 23 and the background plate 20. The RGBD camera 21 is used to record a color image of the background plate 20 and a color image and a depth image of the target object 23, and transmits the recorded images to an external processor (not shown), which processes the images according to the automatic target object annotation method described in any of the foregoing embodiments.

[0081] In some embodiments, the RGBD camera 21 is horizontally fixed, and the support 22 is positioned outside the field of view of the RGBD camera 21, thereby ensuring that the support cannot be captured when the RGBD camera is working. The RGBD camera 21 can be a structured light camera, a ToF camera, or other depth camera.

[0082] In some embodiments, a light source 24 is also included for illuminating a target object 23 placed on a support 22; wherein the illumination angle and light intensity of the light source are set to be adjustable.

[0083] In some instances, the RGBD camera 21 connects to an external processor via a wired connection such as USB or HDMI. It is understood that in some embodiments, the RGBD camera 21 may also connect to an external processor via a wireless connection.

[0084] The automatic product labeling system of this invention specifically implements the target object automatic labeling method described in the foregoing embodiments. For a detailed description, please refer to the target object automatic labeling method, which will not be repeated here.

[0085] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the target object automatic annotation method described in any of the foregoing embodiments.

[0086] The storage medium can be implemented by any type of volatile or non-volatile storage device, or a combination thereof. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM). The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0087] It is understood that the above description is a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be considered that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered to fall within the protection scope of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., means that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention.

[0088] In this specification, the illustrative expressions of the terms used do not necessarily refer to the same embodiments or examples. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of different embodiments or examples, without contradiction. Although embodiments of the invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications may be made herein without departing from the scope defined by the appended claims.

[0089] Furthermore, the scope of this invention is not intended to be limited to the specific embodiments of the processes, machines, manufactures, compositions of matter, means, methods, and steps described in the specification. Those skilled in the art will readily understand that existing or later-developed disclosures, processes, machines, manufactures, compositions of matter, means, methods, or steps that perform substantially the same function as the corresponding embodiments described herein or obtain substantially the same results as the embodiments described herein can be utilized. Therefore, the appended claims are intended to include such processes, machines, manufactures, compositions of matter, means, methods, or steps within their scope.

Claims

1. A method for automatically labeling target objects, characterized in that, Including the following steps: S1. Obtain a background color image containing only the background panel; S2. Acquire a color image and a depth image containing the target object; place the target object between the background plate and the RGBD camera; wherein the distance D1 between the RGBD camera and the target object is greater than the distance D0 between the target object and the background plate; acquire a color image and a depth image containing the target object through the RGBD camera; S3. Pre-annotate the depth image containing the target object to obtain a pre-annotated target image; S4. Based on the background color image and the color image containing the target object, perform pixel-level annotation on the pre-annotated target image to obtain the target object pixels and obtain a pixel-level annotated target image; The pre-annotated target image is subjected to erosion processing to obtain the inner edge of the target object; The inner edge of the target object is expanded to obtain the outer edge of the target object, wherein the region between the inner edge and the outer edge is the edge region; Traversing each pixel p in the edge region, and pixels p1 and p2 in the background color image and the color image containing the target object respectively corresponding to pixel p, to obtain an effective edge region; including: setting a threshold, if the pixel values ​​of pixels p1 and p2 in the background color image and the color image containing the target object respectively corresponding to pixel p are both greater than the threshold, and the pixel value of pixel p is greater than 0, then pixel p is determined to be a target object pixel, and the region formed by the target object pixel is the effective edge region; The pixels contained in the inner edge and / or the pixels contained in the effective edge region are defined as the target object pixels.

2. The automatic target object annotation method as described in claim 1, characterized in that, Step S3 includes: Iterate through each pixel of the depth image containing the target and read its pixel value; Determine whether the pixel value is within the range D0~D1. If so, mark the corresponding pixel as the target pixel and then obtain the pre-labeled target image.

3. The automatic target object annotation method as described in claim 1, characterized in that, A bracket is provided between the background panel and the RGBD camera for placing the target object; the background panel is a solid color background panel.

4. The automatic target object annotation method as described in claim 1, characterized in that, It also includes the following steps: S5. Replace the background board or change the ambient lighting to record multiple background color images so that the same set of annotations corresponds to different images.

5. An automatic target object labeling system, characterized in that, include: A background panel, an RGBD camera, and a support disposed between the background panel and the RGBD camera; wherein the support is used to place the target object; the RGBD camera is used to record a background color image containing only the background color image of the background panel and a color image and a depth image of the target object, and transmits the recorded image to an external processor, which processes the target object automatic annotation method according to any one of claims 1-4.

6. The automatic target object annotation system as described in claim 5, characterized in that, When the target object is placed on the bracket, the distance D1 between the RGBD camera and the target object is greater than the distance D0 between the target object and the background plate.

7. The automatic target object annotation system as described in claim 5, characterized in that, The background panel is a solid color background panel.

Images