3D image entity segmentation methods, apparatus, equipment, storage media, and vehicles

Abstract

This disclosure provides a method, apparatus, device, storage medium, and vehicle for three-dimensional image entity segmentation, relating to the fields of artificial intelligence and autonomous driving technology, particularly to artificial intelligence technologies such as image processing, deep learning, mapping, and navigation. The specific implementation scheme is as follows: acquiring a two-dimensional image; segmenting multiple target objects in the two-dimensional image into mutually independent entity objects; mapping each entity object from the two-dimensional image to a three-dimensional image to obtain point cloud data for each entity object; and determining the three-dimensional geometric entity of each entity object based on the point cloud data. Using the method provided in this disclosure transforms the segmentation of entity objects in a three-dimensional image into first segmenting the entity objects in a two-dimensional image and then mapping them to three-dimensional space, thereby achieving the segmentation of three-dimensional entities, reducing the difficulty of entity segmentation, and ensuring the accuracy of segmentation.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and autonomous driving technology, and in particular to artificial intelligence technologies such as image processing, deep learning, mapping, and navigation. Background Technology

[0002] Two-dimensional images, lacking depth information, cannot meet application requirements, thus three-dimensional images have become increasingly sought after.

[0003] When image content is complex, it's necessary to segment out certain targets to meet the needs of downstream operations. For example, in the surveillance field, this involves segmenting out individuals or people. Furthermore, with the development of intelligent transportation and autonomous driving, map data has gradually expanded from traditional two-dimensional road networks to current high-precision three-dimensional maps. These high-precision three-dimensional maps contain not only road information but also various other types of information. For instance, signs, as crucial infrastructure for indicating road traffic, can also be used in autonomous vehicles.

[0004] However, how to accurately segment 3D targets in 3D images generated using point clouds is a problem of concern in the industry. Summary of the Invention

[0005] This disclosure provides a method, apparatus, device, storage medium, and vehicle for three-dimensional image entity segmentation.

[0006] According to one aspect of this disclosure, a three-dimensional image entity segmentation method is provided, comprising:

[0007] Acquire a two-dimensional image;

[0008] Segmenting multiple target objects in a 2D image into independent entity objects;

[0009] Each entity object is mapped from a two-dimensional image to a three-dimensional image to obtain the point cloud data of each entity object;

[0010] Based on the point cloud data of each entity object, the three-dimensional geometric entity of each entity object is determined.

[0011] According to another aspect of this disclosure, a three-dimensional image entity segmentation apparatus is provided, comprising:

[0012] The acquisition module is used to acquire two-dimensional images;

[0013] The segmentation module is used to segment multiple target objects in a two-dimensional image into mutually independent entity objects;

[0014] The mapping module is used to map each entity object from a two-dimensional image to a three-dimensional image to obtain the point cloud data of each entity object.

[0015] The entity determination module is used to determine the three-dimensional geometric entity of each entity based on the point cloud data of each entity object.

[0016] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0017] At least one processor; and

[0018] The memory is communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods of any embodiment of the present disclosure.

[0020] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a method according to any embodiment of this disclosure.

[0021] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a method according to any embodiment of this disclosure.

[0022] According to another aspect disclosed herein, a vehicle is provided that carries the electronic equipment described above.

[0023] Using the method provided in this disclosure, the segmentation of entity objects in a 3D image is transformed into segmenting the entity objects in a 2D image and then mapping them to a 3D space, thereby achieving the segmentation of 3D entities, reducing the difficulty of entity segmentation, and ensuring the accuracy of segmentation.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0025] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0026] Figure 1 This is a flowchart illustrating a three-dimensional image entity segmentation method according to an embodiment of the present disclosure;

[0027] Figure 2(a) is a schematic flowchart of two-dimensional image segmentation according to an embodiment of the present disclosure;

[0028] Figure 2(b) is a scene diagram of two-dimensional image segmentation according to an embodiment of the present disclosure;

[0029] Figure 3 This is a scene diagram of determining the segmentation plane provided in one embodiment of the basic disclosure;

[0030] Figure 4(a) is a scene diagram of two-dimensional image target detection of traffic signs provided in an embodiment of the basic disclosure;

[0031] Figure 4(b) is a scene diagram of two-dimensional image semantic segmentation of a traffic sign provided in a basic disclosed embodiment;

[0032] Figure 4(c) is a scene diagram of two-dimensional image instance segmentation of a traffic sign provided in a basic disclosed embodiment.

[0033] Figure 5(a) is a depth map of a two-dimensional image provided according to an embodiment of the present disclosure;

[0034] Figure 5(b) is a point cloud diagram of a traffic sign provided according to an embodiment of the present disclosure;

[0035] Figure 6(a) is a schematic diagram of a three-dimensional geometric model provided according to an embodiment of the present disclosure;

[0036] Figure 6(b) is a schematic diagram of a fine-tuned geometric model provided according to an embodiment of the present disclosure;

[0037] Figure 7 This is a schematic diagram of a three-dimensional image entity segmentation apparatus according to an embodiment of the present disclosure;

[0038] Figure 8 This is a block diagram of an electronic device used to implement the three-dimensional image entity segmentation method of the embodiments of this disclosure. Detailed Implementation

[0039] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0040] With the development of related technologies, 3D imaging technology has been applied in many scenarios. Compared with 2D images, 3D images are more intuitive and contain much information that cannot be shown in 2D images. For example, by constructing a 3D model of a physical object, one can gain a comprehensive understanding of some of the object's characteristics.

[0041] Because 3D images contain more information and their production process is more complex, they present many challenging scenarios. For example, a 3D image may contain multiple objects that are close together or even overlap, which can easily lead to errors when segmenting the 3D objects.

[0042] In view of this, in order to accurately segment three-dimensional objects, embodiments of this disclosure provide a three-dimensional image entity segmentation method. This method first performs segmentation of the target object on a two-dimensional image, separating different target objects, and then maps this segmentation onto a three-dimensional image to achieve segmentation of the target object on the three-dimensional image. Therefore, the problem of segmenting the target object is transformed from three-dimensional space to two-dimensional space, and the target objects accurately segmented in two-dimensional space can then be accurately and quickly segmented into three-dimensional target objects.

[0043] like Figure 1 The diagram shown is a flowchart of the method, including:

[0044] S101. Obtain a two-dimensional image.

[0045] S102. Divide multiple target objects in a two-dimensional image into independent entity objects.

[0046] S103. Map each entity object from a two-dimensional image to a three-dimensional image to obtain the point cloud data of each entity object.

[0047] Since there is a mapping relationship between two-dimensional and three-dimensional images, pixels in the two-dimensional image can be mapped onto the three-dimensional image. In this embodiment, the two-dimensional and three-dimensional images have at least partially overlapping imaging areas. Therefore, entity segmentation of target objects within the common area can be achieved.

[0048] S104. Based on the point cloud data of each entity object, determine the three-dimensional geometric entity of each entity object.

[0049] The segmentation of entity objects in a 3D image is transformed into a 2D space that accurately segments different target objects. Then, the entity objects accurately segmented in the 2D image are mapped back into the 3D space, resulting in independent entity objects within the 3D space. Segmenting independent entity objects in 2D space not only ensures the accuracy of the segmentation results but also reduces the difficulty of segmenting entity objects in 3D space. Therefore, this disclosure provides a general, accurate, and easy-to-implement method for 3D entity object segmentation.

[0050] For the entity segmentation of the two-dimensional image in step S102, the segmentation of the two-dimensional image can be completed based on the flowchart shown in Figure 2(a), including the following:

[0051] S201. Perform semantic segmentation on the two-dimensional image to obtain a connected region containing at least one target object.

[0052] As shown in Figure 2(b), the image contains two people, Person 1 and Person 2, and also includes a sun. Person 1 and Person 2 partially overlap. After semantic segmentation of Figure 2(b), two connected regions can be obtained: region A containing Person 1 and Person 2, and region B containing the sun. Semantic segmentation of a two-dimensional image can be specifically implemented as follows:

[0053] The semantic segmentation model is used to classify each pixel of the two-dimensional image. For example, the pixels of the person in Figure 2(b) are classified as category a, and the pixels of the sun are classified as category b.

[0054] Pixels of the same category are combined to form a connected region. For example, in Figure 2(b), pixels of category a are combined to form region A, and pixels of category b are combined to form region B.

[0055] In order to ensure that the connected regions can accurately represent the location of the target objects, in this embodiment of the disclosure, after obtaining the connected regions of each target object, the connected regions of each target object can be subjected to erosion or expansion processing.

[0056] S202. Perform instance segmentation on the image content within connected regions of a two-dimensional image to obtain mutually independent entity objects.

[0057] First, semantic segmentation is used to obtain connected regions containing at least one target object, filtering out most of the image content that is irrelevant to the target object. Then, instance segmentation is used within the connected regions to separate the entity objects in the two-dimensional image, which can improve the accuracy of entity object segmentation in two-dimensional images.

[0058] Taking Figure 2(b) as an example, after obtaining regions A and B, assuming the target object is a person, instance segmentation of region A in the image can yield entity object 1 and entity object 2 corresponding to person 1 and person 2, respectively. Entity object 1 and entity object 2 have the same category. Instance segmentation of image content within a connected region can be specifically implemented as follows:

[0059] Obtain all pixels within the connected region; then, input these pixels into the instance segmentation model to obtain mutually independent entity objects.

[0060] Regarding the mapping of the two-dimensional image to the three-dimensional image in step S103, the mapping method can be the same for different acquisition methods of the two-dimensional and three-dimensional images. In one possible implementation, two-dimensional and three-dimensional images of the same imaging area can be acquired simultaneously. For example, a set of acquisition equipment can be equipped with a camera for acquiring two-dimensional images and a laser device for acquiring three-dimensional images. In this case, two-dimensional images and point cloud data can be obtained directly, and the mapping relationship between the two-dimensional and three-dimensional images can be obtained through post-processing. In this embodiment of the disclosure, this is also referred to as a depth map.

[0061] In another possible implementation, when acquiring a two-dimensional image, a depth map corresponding to the two-dimensional image can also be obtained. For example, a two-dimensional image and its corresponding depth map can be acquired based on a depth camera (RGB-D) or a stereo camera, and the depth map can be used as a mapping relationship between the two-dimensional image and the three-dimensional image.

[0062] A depth map can be understood as a sparse representation of a 3D image in 3D space. Therefore, each point in the depth map can also find its corresponding point cloud data in the point cloud of the 3D image.

[0063] Based on the depth map, mapping each entity object from a 2D image to a 3D image can be implemented as follows:

[0064] Step A1: Obtain the pixel coordinates of each entity object in the two-dimensional image.

[0065] Step A2: Extract the 3D points of each entity object from the depth map based on the pixel coordinates of each entity object.

[0066] Step A3: Extract the point cloud data of each entity object from the 3D image based on the 3D points of each entity object.

[0067] After segmenting solid objects on a two-dimensional image, depth maps are used to transform these objects into three-dimensional space, resulting in relatively independent solid objects within the three-dimensional image. Since depth maps accurately represent the mapping relationship between two-dimensional and three-dimensional images, the accuracy of segmenting three-dimensional geometric entities from three-dimensional space can be improved.

[0068] In some embodiments, after obtaining the 3D points of the target object in the depth map, a kdtree (multidimensional tree) can be used to obtain the complete point cloud data of each entity object. Specifically, this can be implemented as follows:

[0069] Step B1: In the three dimensions of the point cloud data, determine the dimension with the largest variance, and then select the median value on that dimension to divide the data set into two subsets.

[0070] For example, if the point cloud data has the largest variance on the x-axis, and the x-coordinates are x1, x2, x3, x4, and x4 in ascending order, then the point cloud data can be divided into two subsets from the plane x = x3.

[0071] Step B2: Repeat step B1 for each subset until all subsets can no longer be divided, thus obtaining the kdtree.

[0072] Step B3: Use the kdtree search function to search for the point cloud data of the entity object based on the 3D points of the entity object in the depth map.

[0073] The kdtree search function finds points that are close to 3D points in the depth map to construct the point cloud data of the entity object. For example, for a 3D point A to be retrieved, the kdtree search function finds point cloud data that are within a distance threshold range from point A, which are then used as the point cloud data of the entity object.

[0074] In other embodiments, an octree can also be used to construct the search path, specifically as follows:

[0075] Step C1: Divide the three-dimensional space into 8 equal cubes;

[0076] Step C2: Repeat step C1 for each cube until all cubes contain only one three-dimensional point.

[0077] KD-trees offer high search efficiency for small datasets, while OC-trees have a simpler algorithm implementation. Combining KD-trees and OC-trees—first using OC-trees to partition the larger space, then using KD-trees for further subdivision—can improve efficiency to some extent.

[0078] The method used to obtain the 3D point cloud data of each entity object can be determined according to the specific circumstances. This disclosure does not limit the method used to obtain the 3D point cloud data of each entity object.

[0079] After obtaining the three-dimensional geometric entities of each object, overlapping areas may occur between the obtained three-dimensional geometric entities because each target object is processed independently. In some embodiments, fine-tuning can be performed to ensure that each target object can be sufficiently separated in three-dimensional space. This can be implemented as follows: when two adjacent three-dimensional geometric entities have overlapping areas, a dividing surface for the overlapping area is determined; the boundary of the overlapping area in the two adjacent three-dimensional geometric entities is shrunk by a specified distance along the direction perpendicular to the dividing surface to obtain two independent adjacent geometric entities.

[0080] Among them, two adjacent three-dimensional geometric entities have two adjacent faces (i.e., the boundary in the overlapping area). Multiple center points between the two adjacent faces can be determined. Data fitting is performed using these multiple center points to obtain the segmented face.

[0081] For example Figure 3 As shown, two overlapping cubes have overlapping boundaries 1 and 2, where each boundary is essentially a face. Therefore, sampling can be performed on either boundary 1 or boundary 2 to obtain multiple sampling points. Figure 3 For example, sampling is performed on boundary 1 to obtain multiple sampling points. Then, the distance from each sampling point to boundary 2 is calculated, and the center point at that distance is taken (e.g., ...). Figure 3 (Triangle points in the diagram). Then, using the three-dimensional coordinates of multiple center points in three-dimensional space, a plane is fitted to obtain a segmentation surface.

[0082] Of course, if the boundaries of the two 3D geometric entities within the overlapping region are curved surfaces, then the fitted segmentation surface can also be curved. To improve efficiency, in this embodiment, even if the boundaries are curved surfaces, a plane can be fitted using multiple center points to obtain a segmentation plane. The segmentation plane is used to divide the points within the overlapping region, that is, to assign the points within the overlapping region to the corresponding 3D geometric entities. This redefines the boundaries of the 3D geometric entities within the original overlapping region. To further separate the two 3D geometric entities, the redefined boundary can be shrunk.

[0083] In this embodiment, three-dimensional geometric entities with partially overlapping regions can be automatically identified, and then segmented further using segmentation surfaces. This enables further fine-tuning of the three-dimensional geometric entities in three-dimensional space, thereby improving the accuracy of three-dimensional image entity segmentation and making the final three-dimensional entity objects completely independent.

[0084] In some embodiments, in addition to determining the three-dimensional geometric entities of each entity object in a three-dimensional image, the attributes of each entity object can also be determined for application requirements, so that the segmented three-dimensional geometric entities have more attribute information to facilitate the use of the target object.

[0085] When the attribute information includes the category of the target object, the embodiments of this disclosure may include target detection of the two-dimensional image to obtain the category of each target object. Taking Figure 2(b) as an example, target detection is performed on the two-dimensional image to obtain two categories: people and the sun, and rectangular boxes corresponding to the two people and a rectangular box corresponding to the sun are obtained respectively.

[0086] Based on the preceding explanation, semantic segmentation can be performed on the bounding box of the target object to determine the category of each pixel within the bounding box. The reason for performing object detection before semantic segmentation is that object detection can only determine the bounding box of the target object in a two-dimensional image, but it cannot accurately identify whether each pixel within the bounding box belongs to the target object. Therefore, object detection can filter out most of the content irrelevant to the target object, and then semantic segmentation can be performed on the bounding box of the target object, reducing the amount of data used for semantic segmentation and improving its efficiency.

[0087] Furthermore, to overcome the problem that in some cases the bounding box obtained by object detection may not completely contain the target object, the bounding box obtained by object detection can be enlarged, for example, by 1.2 times or 1.5 times, with the specific enlargement factor determined according to actual needs. Then, semantic segmentation is performed on the pixels within the enlarged bounding box.

[0088] Based on the preceding explanation, after semantic segmentation, instance segmentation is required. After obtaining the instance segmentation results, the bounding boxes are replaced with the instance segmentation results to obtain the accurate location of each target object, i.e., the independent target objects.

[0089] It should be noted that the target detection model used in the embodiments of this disclosure is not specifically limited. For example, the Faster-RCNN (Faster Region-Convolutional-Neural-Networks) model can be used, which uses a region proposal network for binary classification and then uses a head of interest to achieve target detection. Another example is YOLO (YouOnly Look Once), which uses multiple convolutional layers and fully connected layers to achieve target detection.

[0090] Furthermore, in other embodiments, instance segmentation models can be used to obtain the category information of the target object. For example, by adding a branch that outputs a binary mask to the Faster-RCNN model, a mask-RCNN (mask-Region-Convolutional-Neural-Networks) model is obtained, which is used to identify the accurate location and category of the target object.

[0091] Detecting objects in a 2D image to obtain the category of the entity object allows the resulting 3D image to contain the category information of the entity object, thus meeting the needs of more application scenarios.

[0092] When the attribute information includes the shape information of the target object, after the target object is segmented into instances, a shape recognition module can be added to classify the outline of each entity object and obtain the shape of each entity object.

[0093] Taking Figure 2(b) as an example, after obtaining target object 1 and target object 2 through instance segmentation, the outline shape of each entity object is identified to obtain the posture shape of the two people, such as whether they are sideways or facing forward and backward.

[0094] When segmenting traffic signs in a 3D high-precision map, since traffic signs have different shapes, shape recognition can be used to obtain the shape information of each segmented traffic sign, which is then used for later rendering, content analysis of traffic signs, autonomous driving navigation, and other purposes.

[0095] It should be noted that the shape classification of the two-dimensional image in this embodiment is for the purpose of obtaining shape recognition results. The shape recognition model used in this embodiment is not specifically limited.

[0096] In summary, shape recognition of entities in a 2D image to obtain the shape of the target object allows the resulting 3D image to contain the shape information of the target object, thus meeting the needs of more application scenarios.

[0097] In another implementation, the category of the target object can be determined through object detection, and the accurate location and shape of the target object can be determined based on an instance segmentation model. In this case, in addition to segmenting independent entity objects as described above, the instance segmentation model can also identify the shape category of the entity objects. When instance segmentation is used for shape recognition, the image content within a connected region of a two-dimensional image can be segmented to obtain the shape of each entity object. Specifically, the location of the target object is obtained using the instance segmentation model, and classification processing is performed to obtain the shape category of the target object.

[0098] Adding shape recognition to the instance segmentation model can not only obtain the accurate shape of the target object, but also improve the utilization efficiency of the instance segmentation model and save resources.

[0099] To facilitate understanding of the solutions provided in the embodiments of this disclosure, the following will take traffic signs as an example to illustrate how the embodiments of this disclosure cut out individual traffic signs from a three-dimensional image.

[0100] Creating 3D maps during map production offers significant advantages over traditional 2D road network maps. 3D maps contain more information, such as road gradients and traffic signs. In 3D maps, traffic signs possess rich semantic information, playing a crucial role in guiding traffic and enabling autonomous driving. However, traffic signs in 3D maps are prone to overlapping, necessitating 3D segmentation.

[0101] Equipped with both a camera and a LiDAR, the vehicle can simultaneously acquire two-dimensional images and their corresponding laser point clouds. Through subsequent data processing, a mapping relationship between the two-dimensional image space and the three-dimensional laser point cloud, i.e., a depth map, can be obtained. The acquired two-dimensional image is shown in Figure 4(a), which contains four traffic signs and one billboard. Object detection is first performed on the two-dimensional image shown in Figure 4(a) to obtain the category of each object and its location represented by a bounding box. A total of four bounding boxes are obtained, indicating that these four bounding boxes may contain traffic signs.

[0102] Next, semantic segmentation is performed on the 2D image to classify the pixels. As shown in Figure 4(b), the pixels in the sign are classified as category c, and the pixels in the billboard are classified as category d. Since the goal is to segment traffic signs, pixels outside category c are filtered out, that is, the pixels contained in the billboard in the 2D image are filtered out. After filtering, the points of category c are analyzed to obtain the connected regions of the traffic signs. As shown in Figure 4(b), after semantic segmentation of the 2D image, two connected regions can be obtained, namely the connected region that combines three traffic signs in Figure 4(b), and the connected region of the camera monitoring sign in the lower left corner of Figure 4(b).

[0103] After obtaining connected regions through semantic segmentation, instance segmentation is performed on the signs within the connected regions in the 2D image. The result of instance segmentation is shown in Figure 4(c), which includes four independent traffic signs.

[0104] The results of instance segmentation are then used to overlay the bounding boxes obtained from object detection, resulting in four traffic signs with accurate borders and type and shape information. These four traffic signs are independent of each other. Thus, traffic sign segmentation on a 2D image is achieved, and the category of each sign ("traffic sign") and its shape are obtained.

[0105] Next, as shown in Figure 5(a), each traffic sign, after being split in the two-dimensional image, is mapped to three-dimensional space using a depth map to obtain the point cloud data of each traffic sign. The specific operation method can be found in steps A1-A3, and will not be elaborated here. The obtained point cloud data is shown in Figure 5(b), which shows the point cloud data for each of the four traffic signs.

[0106] The point cloud data of the traffic signs is subjected to RANSAC (Random Sample Consensus) fitting and denoising to obtain a planar point cloud of the traffic signs. Specifically, the central plane of the traffic sign is first fitted, and points in 3D space perpendicular to the 2D image that are more than a preset distance from the fitted plane are filtered out. The preset distance can be determined according to the type of traffic sign. After fitting and denoising, the planar geometry of the traffic sign in the point cloud is determined. Then, its 3D geometric model is determined based on the height range of its 3D coordinates. That is, based on the planar range of the traffic sign in the 2D image and the points determined in the 3D space perpendicular to the 2D plane, the 3D geometric model of the traffic sign is obtained.

[0107] Figure 6(a) shows a schematic diagram of the generated 3D geometric model. Since each traffic sign is processed individually, there may be overlapping areas when mapped into 3D space. As shown in Figure 6(a), overlapping traffic signs appear. Therefore, the center plane of the overlapping area is determined as the dividing plane. The upper boundary of the sign above is contracted upwards by a specified distance along the direction perpendicular to the dividing plane, and the upper boundary of the sign below is contracted downwards by a specified distance along the direction perpendicular to the dividing plane. This specified distance is, for example, n pixels, where n is a positive integer. Thus, the traffic signs are segmented in the 3D image, and the final 3D geometric model of the sign is shown in Figure 6(b), completely separating the two adjacent traffic signs.

[0108] In summary, traffic sign entities can be segmented on 3D images. The generated individual traffic signs can provide traffic guidance on 3D maps, and as 3D image entities, they can also be used to control autonomous vehicles.

[0109] Based on the same technical concept, this disclosure also provides a three-dimensional image entity segmentation apparatus, such as... Figure 7 As shown, the device includes:

[0110] Acquisition module 701 is used to acquire two-dimensional images;

[0111] The segmentation module 702 is used to segment multiple target objects in a two-dimensional image into mutually independent entity objects;

[0112] The mapping module 703 is used to map each entity object from a two-dimensional image to a three-dimensional image to obtain the point cloud data of each entity object.

[0113] The entity determination module 704 is used to determine the three-dimensional geometric entity of each entity based on the point cloud data of each entity object.

[0114] In some embodiments, the segmentation module 702 is configured to:

[0115] Semantic segmentation of a two-dimensional image yields a connected region containing at least one target object;

[0116] Instance segmentation is performed on the image content within connected regions of a two-dimensional image to obtain mutually independent entity objects.

[0117] In some embodiments, the device further includes:

[0118] The object detection module is used to detect objects in two-dimensional images and obtain the category of each entity object.

[0119] In some embodiments, the device further includes:

[0120] The shape recognition module is used to classify the outlines of each entity object to obtain the shape of each entity object.

[0121] In some embodiments, where instance segmentation is also used for shape recognition, the segmentation module is further used to perform instance segmentation on the image content within a connected region in a two-dimensional image, and to obtain the shape of each entity object.

[0122] In some embodiments, the mapping module 703 is used for:

[0123] Obtain the pixel coordinates of each entity object in the 2D image;

[0124] Based on the pixel coordinates of each entity object, extract the 3D points of each entity object from the depth map;

[0125] Based on the 3D points of each entity object, the point cloud data of each entity object is extracted from the 3D image.

[0126] In some embodiments, the device further includes:

[0127] The overlap determination module is used to determine the dividing surface of the overlapping region when two adjacent three-dimensional geometric entities have overlapping regions;

[0128] The fine-tuning module is used to shrink the boundary of the overlapping area of ​​two adjacent 3D geometric entities by a specified distance along the direction perpendicular to the dividing surface, so as to obtain two independent adjacent geometric entities.

[0129] In some embodiments, the target object includes a traffic sign.

[0130] The specific functions and examples of each module and submodule of the apparatus in this disclosure can be found in the relevant descriptions of the corresponding steps in the above method embodiments, and will not be repeated here.

[0131] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0132] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0133] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0134] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0135] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the three-dimensional image entity segmentation method. For example, in some embodiments, the three-dimensional image entity segmentation method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the three-dimensional image entity segmentation method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the three-dimensional image entity segmentation method by any other suitable means (e.g., by means of firmware).

[0136] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0137] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0138] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0139] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0140] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0141] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0142] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0143] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for three-dimensional image entity segmentation, comprising: obtaining a two-dimensional image; segmenting a plurality of target objects in the two-dimensional image into independent entity objects; mapping each entity object from the two-dimensional image to a three-dimensional image to obtain point cloud data of each entity object; determining a three-dimensional geometric entity of each entity object based on the point cloud data of each entity object; in a case where two adjacent three-dimensional geometric entities have an overlapping region, determining a segmentation surface of the overlapping region, comprising: determining two surfaces corresponding to the boundaries of the two adjacent three-dimensional geometric entities in the overlapping region; determining a plurality of center points between the two surfaces; and performing data fitting on the obtained plurality of center points to obtain the segmentation surface; shrinking the boundaries of the overlapping region in the two adjacent three-dimensional geometric entities by a specified distance along a direction perpendicular to the segmentation surface to obtain the two adjacent three-dimensional geometric entities which are independent of each other.

2. The method of claim 1, wherein, segmenting a plurality of target objects in the two-dimensional image into independent entity objects, comprising: performing semantic segmentation on the two-dimensional image to obtain a connected region containing at least one target object; performing instance segmentation on image content in the connected region in the two-dimensional image to obtain independent entity objects. 3.The method of claim 1 or 2, further comprising: performing target detection on the two-dimensional image to obtain the category of each entity object. 4.The method of claim 1 or 2, further comprising: performing classification processing on the contour of each entity object to obtain the shape of each entity object. 5.The method of claim 2, further comprising: in a case where the instance segmentation is also used for shape recognition, performing instance segmentation on image content in the connected region in the two-dimensional image to obtain the shape of each entity object.

6. The method of claim 1, wherein, mapping each entity object from the two-dimensional image to a three-dimensional image, comprising: obtaining pixel coordinates of each entity object in the two-dimensional image; extracting three-dimensional points of each entity object from a depth map according to the pixel coordinates of each entity object; extracting point cloud data of each entity object from the three-dimensional image based on the three-dimensional points of each entity object.

7. The method of claim 1, wherein, The target objects include traffic signs. 8.A three-dimensional image entity segmentation apparatus, comprising: an obtaining module configured to obtain a two-dimensional image; a segmentation module configured to segment a plurality of target objects in the two-dimensional image into independent entity objects; a mapping module configured to map each entity object from the two-dimensional image to a three-dimensional image to obtain point cloud data of each entity object; an entity determination module configured to determine a three-dimensional geometric entity of each entity object based on the point cloud data of each entity object; further comprising: a determination overlap module configured to, in a case where two adjacent three-dimensional geometric entities have an overlapping region, determine a segmentation surface of the overlapping region, comprising: determining two surfaces corresponding to the boundaries of the two adjacent three-dimensional geometric entities in the overlapping region; determining a plurality of center points between the two surfaces; and performing data fitting on the obtained plurality of center points to obtain the segmentation surface; a fine-tuning module, configured to contract a boundary in the overlapping region in the two adjacent three-dimensional geometric entities by a specified distance along a direction perpendicular to the partition surface, to obtain the two adjacent three-dimensional geometric entities independent of each other.

9. The apparatus of claim 8, wherein, The segmentation module is configured to: perform semantic segmentation on the two-dimensional image to obtain a connected region containing at least one target object; perform instance segmentation on image content in the connected region in the two-dimensional image to obtain entity objects independent of each other.

10. The apparatus of claim 8 or 9, further comprising: a target detection module configured to perform target detection on the two-dimensional image to obtain a category of each entity object.

11. The apparatus of claim 8 or 9, further comprising: a shape recognition module configured to perform classification processing on a contour of each entity object to obtain a shape of each entity object.

12. The apparatus of claim 9, in the case where the instance segmentation is also used for shape recognition, the segmentation module is further configured to perform instance segmentation on image content in the connected region in the two-dimensional image to also obtain a shape of each entity object.

13. The apparatus of claim 8, wherein, The mapping module is configured to: obtain pixel coordinates of each entity object in the two-dimensional image; extract three-dimensional points of each entity object from the depth map according to the pixel coordinates of each entity object; extract point cloud data of each entity object from the three-dimensional image based on the three-dimensional points of each entity object.

14. The apparatus of claim 8, wherein, The target object includes a traffic sign.

15. An electronic device, comprising: at least one processor; and a memory connected in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to enable a computer to perform the method of any one of claims 1-7.

17. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-7.

18. A vehicle carrying the electronic device of claim 15.

Images