Image data set enhancement method for processing occlusion relation in three-dimensional space

Abstract

The invention discloses an image data set enhancement method for processing an occlusion relationship in a three-dimensional space, and the method comprises the steps: generating a three-dimensional space of a scene expressed by an image, and extracting objects which can be placed at different positions in the image; object objects extracted from the image are divided into two types: object objects capable of being suspended and object objects needing to be supported and placed; respectively finding out the placing areas of the object which can be suspended and the object which needs to be supported and placed in the three-dimensional space corresponding to the image; objects capable of being suspended and objects needing to be supported and placed are copied and placed respectively, and various possible placement combinations are carried out; and generating a corresponding new image for each placement combination. The synthetic image obtained by the method is more real and reasonable, the newly generated data set is added into the training set of the deep learning method, the ability of the deep learning model to process three-dimensional space information can be powerfully improved, and related tasks such as image segmentation and target detection can be better executed.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning technology for image processing, and specifically relates to processing occlusion relationships in three-dimensional space to generate more reasonable and realistic new images and better enhance image data sets. Background Art

[0002] Existing deep learning methods for image processing rely on high-quality data sets for training. Improving the rationality and diversity of data in the data set is an important means to improve deep learning methods. For image data set enhancement methods, existing technical methods are divided into the following three categories: performing overall operations on the image itself to generate new images, such as horizontally flipping the image, enhancing the color of image pixels, etc.; dividing the image into many blocks, and then performing various combinations of image blocks to generate new images; extracting objects in the image, and then copying / pasting the objects to different positions in the image to generate new images.

[0003] The above three methods all generate new images by operating in the two-dimensional image space. It is difficult to effectively process the reasonable occlusion relationship between each object in the image corresponding to their three-dimensional space, and the richness gain of the occlusion relationship in the image data set is not good. Even if some enhancement methods use templates to occlude part of the content in the image to increase the image data of the occlusion of the object in the image, they cannot increase the occlusion between each object in the image. Although the image enhancement method that extracts the object in the image and copies / pastes the object can increase the image data of the occlusion relationship between each object in the image, the copied / pasted object cannot be reasonably enlarged / reduced corresponding to its three-dimensional space, and it is difficult to reflect the occlusion relationship between the real object in the three-dimensional space. Therefore, the existing image data set enhancement method is insufficient in expressing the occlusion relationship between objects for the deep learning method for processing three-dimensional tasks, which is not conducive to the deep learning method for processing three-dimensional tasks, such as semantic segmentation and target detection of images collected from three-dimensional space. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes an image data set enhancement method for processing occlusion relationships in three-dimensional space, which places multiple positions of objects in the image in three-dimensional space to generate new images, so that the objects can make effective occlusion relationship judgments based on depth information in three-dimensional space, and project and generate two-dimensional images, so that the objects have appropriate size expressions in the generated two-dimensional images and the occlusion relationships between them are reasonable. Therefore, the present invention can increase the diversity of occlusion relationships between objects in the image data set and the increased occlusion situations are reasonable, which will effectively improve the ability of deep learning methods to handle three-dimensional tasks.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0006] A method for enhancing an image data set for processing occlusion relationships in a three-dimensional space, the method comprising the following steps:

[0007] Step 1: Generate a three-dimensional space of the scene expressed by the image, extract objects that can be placed at different positions in the image and generate a three-dimensional expression of them that is independent of the coordinate system;

[0008] Step 2: objects extracted from the image that can be placed at different positions are divided into two categories: suspended objects and objects that need to be supported. The suspended objects refer to objects that can be placed freely in three-dimensional space; the objects that need to be supported refer to objects that need to be attached to other objects for placement.

[0009] Step 3, respectively finding the areas where the suspended object and the object to be supported can be placed in the three-dimensional space corresponding to the image;

[0010] Step 4: Copy and place the suspended objects and the objects that need to be supported to their respective placement areas in the three-dimensional space, and perform various possible placement combinations;

[0011] Step 5: For each placement combination of the suspended objects and the objects to be supported in the three-dimensional space, a corresponding new image is generated.

[0012] Furthermore, step 1 includes: performing depth estimation on each pixel of the image, and obtaining a three-dimensional expression of the object object that is independent of the coordinate system based on the depth difference between each pixel of the object object and its anchor point, which is called a depth difference template.

[0013] Furthermore, step 2 includes: detecting edge pixels of the object, calculating the depth difference between each edge pixel and its adjacent pixels that are not the object, and if the depth difference of all edge pixels of the object exceeds a threshold, it indicates that the object is not in contact with the surrounding environment and is marked as an object that can be suspended; otherwise, it is marked as an object that needs support and placement.

[0014] Furthermore, the step 3 comprises:

[0015] For an object that needs support, find the support of the object. The area where the support is found in the image is the area where the object can be placed.

[0016] For the suspended object, according to the three-dimensional space of the scene expressed by the image, the empty area of ​​the scene is found as the area where the suspended object can be placed;

[0017] The area where the objects to be supported can be placed is divided into two-dimensional grids, and each two-dimensional grid has the same size; the area where the objects that can be suspended can be placed is divided into three-dimensional grids, and each three-dimensional grid has the same size.

[0018] Furthermore, step 4 includes: for each suspended object and object that needs to be supported, randomly select a position in each grid cell in the relevant two-dimensional grid and three-dimensional grid in its respective placeable area to place the object, or do not place it.

[0019] Further, the step 5 comprises:

[0020] According to the new position of the object in the three-dimensional scene, the size of the object in the new image is calculated in combination with the inverse relationship between the depth of the object and its size in the image;

[0021] Assigning a depth value of the position of the anchor point of the object, and obtaining the depth value of each pixel in the new image corresponding to the new position of the object based on the depth difference template of the object;

[0022] The occlusion relationship between objects is determined based on depth comparison, and based on projection calculation, image synthesis is performed on the unoccluded parts of each object to generate a new image.

[0023] The beneficial effects of the present invention are:

[0024] The present invention can generate new images with reasonable occlusion relationships and increase the diversity of occlusion relationships in image data sets. Applying such enhanced image data sets to deep learning methods that need to process three-dimensional spatial information will improve their processing capabilities. For example, in image segmentation tasks, deep learning models trained with data obtained according to the present invention can effectively segment objects that are incomplete or even lack key information due to occlusion; in target detection tasks, the minimum bounding box of occluded objects can be more effectively detected; in human key point detection tasks, the coordinates of occluded human key points can be more effectively obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram of an image data set enhancement method for processing occlusion relationships in three-dimensional space according to the present invention;

[0026] Figure 2 Schematic diagram of three-dimensional expression of objects that can be placed in different positions; (a) is the input image, (b) is the extracted image object, and (c) is the three-dimensional expression of the extracted image object;

[0027] Figure 3Schematic diagram of mesh division of the area where objects can be placed in the image, (a) is a schematic diagram of mesh division of objects that need to be supported, and (b) is a schematic diagram of mesh division of objects that can be suspended;

[0028] Figure 4 Schematic diagram of a new image with a reasonable occlusion relationship generated based on the method of the present invention, (a) is an input image, (b) is an example of output, and (c) is another example;

[0029] Figure 5 Schematic diagrams comparing the occlusion relationship in new images generated by the present invention and the prior art method; (a) is the original image, (b) is the new image generated by the Cutout method, (c) is the new image generated by the LocalAug method, (d) is the new image generated by the ObjectAug method, (e) is the new image generated by the YOCO method, and (f) is the new image generated by the present invention.

[0030] Figure 6 The figure is a schematic diagram comparing the effects of performing image segmentation tasks based on the present invention and the prior art method. DETAILED DESCRIPTION

[0031] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0032] like Figure 1 FIG. 1 is a flow chart of an image data set enhancement method for processing occlusion relationships in three-dimensional space according to the present invention, which specifically includes the following steps:

[0033] Step 1: Generate a three-dimensional space of the scene expressed by the image using an existing method;

[0034] Step 2, extracting objects that can be placed at different positions in the image and generating a three-dimensional expression thereof that is independent of the coordinate system; dividing the objects that can be placed at different positions extracted from the image into two categories: objects that can be suspended in the air and objects that need to be supported for placement, wherein the objects that can be suspended in the air refer to objects that can be freely placed in three-dimensional space, such as balloons; and the objects that need to be supported for placement refer to objects that need to be attached to other content for placement, such as cars that must be placed on supports such as the ground;

[0035] Step 3: For the suspended objects and the objects that need to be supported, find their respective placement areas in the three-dimensional space corresponding to the image;

[0036] Step 4: Copy and place the suspended objects and the objects that need to be supported in the three-dimensional space respectively in their placeable position areas, and perform various possible combination processing to increase the occlusion between the objects;

[0037] Step 5: For various combinations of suspended objects and supported objects placed in the three-dimensional space, corresponding new images are generated, and relevant object annotations are output.

[0038] Wherein, in the step 1, an existing method is used to estimate the depth of each pixel of the image, thereby obtaining an approximate expression of the three-dimensional scene reflected by the image, wherein the existing method may be the ZoeDepth method ("Depthanything: Unleashing the power of large-scale unlabeled data", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, Yang L, Kang B, Huang Z et al., 2024), which estimates the depth value of each pixel in the image through a deep neural network, that is, the distance between the spatial position represented by the pixel and the camera source. Objects that can be placed at different positions in the image are extracted, and the incompletely expressed objects are completed so that these objects have a reasonable expression when the new image is generated. For each object that can be placed at a different position, a three-dimensional expression is performed, and the three-dimensional expression is independent of the specific coordinate system, such as Figure 2 As shown, for an input image (such as Figure 2 (a)), extract one of the objects that can be placed in different positions (such as Figure 2 As shown in (b), each pixel of the object is assigned its depth value in the scene. In the extracted object, find an anchor point (usually the bottom point, such as point p), then, for each pixel in the object, calculate the depth difference between it and the anchor point and record it, thereby obtaining a depth difference template expressing the object (such as Figure 2 (c) shows a 3D representation of an object that is independent of the coordinate system. Since the depth difference between pixels inside an object does not change due to a change in position, when the object is moved, the depth of all pixels in the object at the new position in 3D space can be calculated based on the depth difference template and the new anchor point depth.

[0039] In step 2, objects that can be placed at different positions are divided into two categories: 1) objects that can be suspended in the air and 2) objects that need to be supported for placement. The present invention proposes a simple method based on depth comparison to distinguish between objects that can be suspended in the air and objects that need to be supported for placement, which is briefly described as follows:

[0040] Detect the pixels at the edge of the object and calculate the depth difference between these pixels and their respective adjacent non-object pixels. If the depth difference of all edge pixels of the object is large, such as exceeding a threshold, it indicates that the object is not in contact with its surroundings and is marked as a suspended object. On the contrary, it indicates that the object is in contact with its surroundings and is marked as an object that needs support. In this way, a suspended object placed on a support will be detected as an object that needs support, such as a balloon placed on the ground. However, this is still effective for adding occlusion relationships between objects in the newly generated image, except that the added occlusion will be reduced.

[0041] In step 3, for the objects that need to be supported and the objects that can be suspended, their respective placement areas are detected and processed as follows:

[0042] For an object that needs to be supported, find the support (such as the road surface that supports the car) and find the location area of ​​the support in the image. The location area of ​​the support is the location area where the object that needs to be supported can be placed;

[0043] For suspended objects, the empty areas of the scene can be known based on the three-dimensional scene expression of the image content. Then, these empty areas are the areas where suspended objects can be placed.

[0044] Generally speaking, any combination of objects that need to be supported and objects that can be suspended in their respective placement areas can generate a new image with a reasonable presentation of occlusion relationships. However, if the locations where these objects are placed are too dense, it will consume a lot of computing resources, but the gain in the diversity of occlusion relationships is not much. Therefore, the present invention divides the areas where they can be placed (that is, the location areas of the supports and the open areas) into grids. Then, in each grid unit, one or several locations are randomly selected to place the object, which can better ensure the increase in the diversity of occlusion relationships in the data set without consuming too many computing resources. Figure 3 As shown in (a), for the street where pedestrians are located, the street area is divided into some 2D grid cells, such as Figure 3 As shown in (b), for a basketball, the open area where it is located is divided into some 3D grid units.

[0045] In step 4, the objects that need to be supported and the objects that can be suspended are placed in their respective placement areas to form a new scene expression. Obviously, placing each object in more positions to generate new images can better enhance the diversity of the data set, which is beneficial to the ability of deep learning methods. However, the computing resources required to generate a large number of new images are very large. In order to balance the use of computing resources and the gain of image data diversity, a related processing of the present invention is: for a portable object, after finding the area where it can be placed, the area is evenly gridded, and one or several positions are randomly selected in each grid unit to place the object, or not. The size of the grid unit is based on the bounding box of the object.

[0046] In step 5, for any combination of placements, a new image is generated, and a new image is obtained, in which the expression of the occlusion relationship between objects is reasonable. The relevant processing is briefly described as follows:

[0047] Based on the relationship that the depth of an object is inversely proportional to its size in the image: h1 d1=h2 d2, here, h1, d1 represent the height and depth of an object at a certain position in the image, while h2, d2 represent the height and depth of the object at another position in the image. Therefore, when an object is placed at a new position in the three-dimensional scene corresponding to the new image, its size presented in the new image can be calculated. Then, according to the depth of its new position in the three-dimensional scene, a corresponding depth value can be assigned to its anchor point. Based on the depth value and the depth difference template of the object, the depth value of each pixel in the new image corresponding to the new position of the object can be obtained.

[0048] Based on the size of an object in a new image at a new position and the depth value of each pixel, the occlusion relationship between objects can be determined based on depth comparison. Then, based on projection calculation, for the unoccluded parts of each object, an image is generated based on image synthesis methods such as the matting method. The matting method refers to mixing the foreground image and the background image together according to a certain ratio of transparency weights to make the synthesized image transition natural and realistic. Figure 4 As shown, (a) is the input image, and (b) and (c) are two new images with reasonable occlusion relationships generated based on the present invention.

[0049] like Figure 5As shown in the figure, the method of the present invention is compared with the prior art method, where (a) is used as the input image, (b), (c), (d), (e), and (f) represent the new image generation results of Cutout, LocalAug, ObjectAug, YOCO, and the present invention, respectively. It can be seen that the present invention can generate new images with reasonable occlusion relationships, while the new images generated by the existing image enhancement methods will produce unreasonable occlusion relationships or improper expression of objects.

[0050] For two commonly used image datasets Cityscapes and PASCAL VOC 2012, the present invention and some existing image data enhancement methods are used to perform data enhancement processing, and then the obtained results are measured by two indicators: diversity and authenticity. The diversity indicator is Dist cos and Dist euc , the larger the value, the better the diversity; the smaller the value of the authenticity indicator FID, the better the authenticity. The relevant statistical data are shown in Table 1. From these data, it can be seen that the present invention is comparable to the current method in terms of diversity enhancement, but is much better than the existing image enhancement method in terms of authenticity. This shows that although the existing image dataset enhancement method can increase the diversity of occlusion relationships, many of the added situations are unreasonable, which will hinder the deep learning method from processing tasks related to three-dimensional space, such as image segmentation.

[0051] Table 1

[0052]

[0053] The data enhanced by the present invention and the existing image dataset enhancement method improve the ability of the image semantic segmentation method, as shown in Tables 2 and Figure 6 These results are obtained by testing on the Cityscapes dataset, and the image semantic segmentation methods tested are MobileNetV3, SegFormer, PointRend, DDRNet, and deeplabv3+. From the statistical data, it can be seen that the improvement effect of the present invention on the image segmentation method is significantly better than the existing image data enhancement method. The datasets obtained by the existing image data enhancement methods sometimes reduce the ability of the image segmentation method. For example, the enhanced data of LocalAug and YOCO reduce the ability of DDRNet. Figure 6The segmentation result of an image is shown as an example, indicating that using the enhanced data of the present invention can obtain better segmentation results. (The improvement range △=ab, a represents the mIoU value of the segmentation effect of a segmentation method using a certain enhanced data, b represents the mIoU value of the segmentation effect of the segmentation method without using enhanced data (baseline), and the average △ represents the average improvement mIoU value of a certain enhanced method over these segmentation methods).

[0054] Table 2

[0055]

[0056] The specific embodiments described above have described the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for enhancing an image dataset for processing occlusion relationships in three-dimensional space, characterized in that: The method comprises the following steps: Step 1: Generate a three-dimensional space of the scene expressed by the image, extract objects that can be placed at different positions in the image, and generate a three-dimensional expression of the object that is independent of the coordinate system; Step 2: objects extracted from the image that can be placed at different positions are divided into two categories: suspended objects and objects that need to be supported. The suspended objects refer to objects that can be placed freely in three-dimensional space; the objects that need to be supported refer to objects that need to be attached to other objects for placement. Step 3, respectively finding the areas where the suspended object and the object to be supported can be placed in the three-dimensional space corresponding to the image; Step 4: Copy and place the suspended objects and the objects that need to be supported to the areas where they can be placed in the three-dimensional space, and perform various possible placement combinations; Step 5: For each placement combination of the suspended objects and the supported objects placed in the three-dimensional space, a corresponding new image and related object annotation information are generated.

2. The method for enhancing an image data set for processing occlusion relationships in three-dimensional space according to claim 1, characterized in that: The step 1 includes: performing depth estimation on each pixel of the image, and obtaining a three-dimensional expression of the object object that is independent of the coordinate system based on the depth difference between each pixel of the object object and its anchor point, which is called a depth difference template.

3. The method for enhancing an image data set for processing occlusion relationships in three-dimensional space according to claim 1, characterized in that: The step 2 includes: detecting edge pixels of the object, calculating the depth difference between each edge pixel and its adjacent pixels that are not the object, and if the depth difference of all edge pixels of the object exceeds a threshold, it indicates that the object is not in contact with the surrounding environment and is marked as an object that can be suspended; otherwise, it is marked as an object that needs support and placement.

4. The method for enhancing an image data set for processing occlusion relationships in three-dimensional space according to claim 1, characterized in that: The step 3 comprises: For an object that needs support, find the support of the object. The area where the support is found in the image is the area where the object can be placed. For the suspended object, according to the three-dimensional space of the scene expressed by the image, the empty area of ​​the scene is found as the area where the suspended object can be placed; The area where the objects to be supported can be placed is divided into two-dimensional grids, and each two-dimensional grid has the same size; the area where the objects that can be suspended can be placed is divided into three-dimensional grids, and each three-dimensional grid has the same size.

5. The method for enhancing an image data set for processing occlusion relationships in three-dimensional space according to claim 1, characterized in that: The step 4 comprises: for each suspended object and object to be supported, randomly selecting a position in each grid cell in the relevant two-dimensional grid and three-dimensional grid in the respective placeable area to place the object, or not placing the object.

6. The method for enhancing an image data set for processing occlusion relationships in three-dimensional space according to claim 1, characterized in that: The step 5 comprises: According to the new position of the object in the three-dimensional scene, the size of the object in the new image is calculated in combination with the inverse relationship between the depth of the object and its size in the image; Assigning a depth value of the position of the anchor point of the object, and obtaining the depth value of each pixel in the new image corresponding to the new position of the object based on the depth difference template of the object; The occlusion relationship between objects is determined based on depth comparison, and based on projection calculation, image synthesis is performed on the unoccluded parts of each object to generate a new image.

Images