Three-dimensional mesh-based target point cloud marking method, device, equipment and medium

Abstract

Embodiments of the present disclosure provide a three-dimensional grid-based target point cloud marking method, device, equipment and medium. The method comprises: constructing a three-dimensional grid containing depth information; converting a depth map contained in the three-dimensional grid into a point cloud; traversing the point cloud in the three-dimensional grid, and marking the target point cloud that should be marked after light projection in the three-dimensional grid according to a pre-generated relationship mapping matrix and an observed depth map shot by a depth camera; wherein the relationship mapping matrix is used to represent the corresponding relationship between the position of each point in the first two coordinate dimensions of the three-dimensional grid and the position of each point in the observed depth map. The technical solution of the embodiments of the present disclosure realizes light projection on the point cloud based on the three-dimensional grid, reduces the algorithm operation amount and complexity by pre-generating the relationship mapping matrix, and improves the efficiency of marking the point cloud.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the technical field of computer, and particularly, to a target point cloud marking method and device based on three-dimensional grid, equipment and medium. BACKGROUND

[0002] Ray casting is the most widely used volume rendering method at present. In the related art, the ray casting algorithm based on search needs to convert the point cloud to voxels for ray casting, and the process of searching the to-be-marked point cloud in the direction of the ray from the optical center to the point cloud involves a large number of operations when traversing each point cloud, and the algorithm complexity is high. SUMMARY

[0003] The present disclosure provides a target point cloud marking method and device based on three-dimensional grid, equipment and medium, to realize ray casting of point cloud based on three-dimensional grid, reduce the algorithm operation amount and complexity, and improve the efficiency of marking point cloud.

[0004] In a first aspect, the embodiments of the present disclosure provide a target point cloud marking method based on three-dimensional grid, comprising:

[0005] constructing a three-dimensional grid containing depth information;

[0006] converting the depth map contained in the three-dimensional grid into a point cloud;

[0007] traversing the point cloud in the three-dimensional grid, and marking the target point cloud to be marked after ray casting in the three-dimensional grid according to a pre-generated relationship mapping matrix and an observed depth map shot by a depth camera;

[0008] The relationship mapping matrix is used to represent the corresponding relationship between the positions of points in the first two coordinate dimensions in the three-dimensional grid and the positions of points in the observed depth map.

[0009] In a second aspect, the embodiments of the present disclosure further provide a target point cloud marking device based on three-dimensional grid, comprising:

[0010] a grid construction module configured to construct a three-dimensional grid containing depth information;

[0011] a conversion module configured to convert the depth map contained in the three-dimensional grid into a point cloud;

[0012] a point cloud marking module configured to traverse the point cloud in the three-dimensional grid, and mark the target point cloud to be marked after ray casting in the three-dimensional grid according to a pre-generated relationship mapping matrix and an observed depth map shot by a depth camera;

[0013] The relationship mapping matrix is used to represent the corresponding relationship between the positions of points in the first two coordinate dimensions in the three-dimensional grid and the positions of points in the observed depth map.

[0014] In a third aspect, the embodiments of the present disclosure further provide an electronic device, comprising:

[0015] one or more processors;

[0016] a storage device configured to store one or more programs,

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for marking a target point cloud based on a three-dimensional grid according to the embodiments of the present disclosure.

[0018] In a fourth aspect, the embodiments of the present disclosure further provide a storage medium containing computer executable instructions for executing the method for marking a target point cloud based on a three-dimensional grid according to the embodiments of the present disclosure when executed by a computer processor.

[0019] According to the embodiments of the present disclosure, a three-dimensional grid containing depth information is constructed, a depth map contained in the three-dimensional grid is converted into a point cloud, the point cloud in the three-dimensional grid is traversed, and a target point cloud to be marked after light projection is marked in the three-dimensional grid according to a pre-generated relationship mapping matrix and an observed depth map shot by a depth camera, wherein the relationship mapping matrix is used to represent the corresponding relationship between the position of each point in the first two coordinate dimensions of the three-dimensional grid and the position of each point in the observed depth map, thereby solving the problem of high algorithm complexity and large amount of calculation of the light projection algorithm in the related art, realizing light projection on the point cloud based on the three-dimensional grid, reducing the amount of algorithm calculation and complexity by pre-generating the relationship mapping matrix, and improving the efficiency of marking the point cloud. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the original and elements are not necessarily drawn according to the scale.

[0021] Figure 1 is a flowchart of a method for marking a target point cloud based on a three-dimensional grid according to an embodiment of the present disclosure;

[0022] Figure 2 is a schematic diagram of light projection on a point cloud based on a three-dimensional grid according to an embodiment of the present disclosure;

[0023] Figure 3 is a flowchart of another method for marking a target point cloud based on a three-dimensional grid according to an embodiment of the present disclosure;

[0024] Figure 4is a structure schematic diagram of a target point cloud marking device based on a three-dimensional grid provided by an embodiment of the present disclosure.

[0025] Figure 5 is a structure schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0026] Embodiments of the present disclosure will be described in more detail with reference to the drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be interpreted as being limited to the embodiments set forth herein, but rather the embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only and are not intended to limit the scope of protection of the present disclosure.

[0027] It should be understood that each step recited in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0028] The term “comprising” and variations thereof as used herein are open-ended, that is, “including but not limited to”. The term “based on” is “based, at least in part, on”. The term “one embodiment” means “at least one embodiment”; the term “another embodiment” means “at least one additional embodiment”; the term “some embodiments” means “at least some embodiments”. Related definitions of other terms will be given in the description below.

[0029] It should be noted that the concepts of “first”, “second”, etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0030] It should be noted that the modification of “one”, “multiple” mentioned in the present disclosure is illustrative and not limiting, and those skilled in the art should understand that unless otherwise explicitly indicated in the context, it should be understood as “one or more”.

[0031] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not used to limit the scope of the messages or information.

[0032] Figure 1is a three-dimensional grid-based target point cloud marking method flowchart provided by an embodiment of the disclosure. The embodiment of the disclosure is applicable to a situation of finding a point cloud to be marked after light projection in a three-dimensional grid based on a depth map observed by a depth camera. The method can be executed by a three-dimensional grid-based target point cloud marking device. The device can be implemented in the form of software and / or hardware. Optionally, the device is implemented by an electronic device, which can be a mobile terminal, a PC terminal, a server, or the like. As shown in Figure 1 , the method comprises the following steps.

[0033] S110, constructing a three-dimensional grid containing depth information.

[0034] The depth information can be the distance of an object from a camera sensor. The three-dimensional grid can be constructed by superimposing multiple depth maps with different depth values in a specified coordinate system, such as the three-dimensional grid in Figure 2 . The specified coordinate system can be a virtual coordinate system whose coordinate axis direction is different from that of the real-world coordinate system. For example, in the xyz coordinate system corresponding to the real world, the x coordinate axis is forward, the y coordinate axis is left, and the z coordinate axis is right. In the uvd coordinate system shown in Figure 2 , the u coordinate axis can be right, the v coordinate axis can be down, and the d coordinate axis can be forward.

[0035] Optionally, constructing the three-dimensional grid containing the depth information comprises: constructing the three-dimensional grid in the specified coordinate system based on the width and height of the depth map and a custom depth value set according to the real distance between adjacent two points; wherein the width of the depth map, the height of the depth map, and the custom depth value sequentially correspond to three dimensions of the specified coordinate system.

[0036] In this embodiment, the width W and the height H of the depth map are obtained, and then in the uvd coordinate system, a set of points with coordinate values of u i = 1, 2, 3, …, W on the u coordinate axis, v i = 1, 2, 3, …, H on the v coordinate axis, and d i = i * S voxel , i = 1, 2, 3, …, D on the d coordinate axis are generated, so as to form a three-dimensional grid V with a size of [W, H, D, 3]. The last dimension in [W, H, D, 3] represents the coordinates of each point cloud in the 3D space. D * S voxel is a custom depth value, D is a modifiable constant, and S voxel is the distance between adjacent two points in the real world in the three-dimensional grid V, i.e., the resolution of the three-dimensional grid V.

[0037] Optionally, after the three-dimensional grid containing depth information is constructed, the method further comprises: converting the depth map contained in the three-dimensional grid into a point cloud according to the camera intrinsic parameter corresponding to the three-dimensional grid; converting the point cloud to the same camera extrinsic parameter as the depth camera through the camera extrinsic parameter corresponding to the three-dimensional grid; converting the point cloud into a target depth map in the same coordinate system as the observation depth map shot by the depth camera through the same camera intrinsic parameter as the depth camera; determining the corresponding relationship between the positions of each point cloud in the three-dimensional grid in the first two dimensions and the positions of each point in the observation depth map according to the target depth map, and generating a relationship mapping matrix.

[0038] In the embodiment, after the three-dimensional grid is constructed, the depth map in the three-dimensional grid can be mapped to the observation plane of the depth camera by performing intrinsic and extrinsic parameter transformation on the three-dimensional grid, the corresponding relationship between the positions of each point in the u and v coordinate dimensions in the three-dimensional grid and the positions of each point in the observation depth map is determined, and a corresponding relationship mapping matrix is generated, so that the search process in the ray direction is optimized to a fixed memory space access assignment process, that is, the runtime operation is optimized to a single data reading, and the calculation amount at runtime is reduced. The ray casting is implemented through fewer loop times, and the algorithm complexity is reduced.

[0039] In the embodiment, the camera intrinsic parameter refers to a parameter related to the characteristics of the camera itself, such as the focal length and pixel size of the camera. The camera extrinsic parameter refers to a parameter in the world coordinate system, such as the position and rotation direction of the camera. The camera intrinsic parameter corresponding to the three-dimensional grid can be a self-defined setting or the intrinsic parameter of a specified depth camera. The depth map contained in the three-dimensional grid is multiplied by the inverse of the intrinsic parameter matrix corresponding to the three-dimensional grid to obtain a point cloud in the uvd coordinate system. Then, based on the difference between the camera extrinsic parameter corresponding to the three-dimensional grid and the extrinsic parameter of the depth camera, the point cloud is subjected to corresponding position translation and direction rotation, and the point cloud is converted to the same camera extrinsic parameter as the depth camera. The point cloud is multiplied by the intrinsic parameter matrix of the depth camera to convert the point cloud into a target depth map in the same coordinate system as the observation depth map shot by the depth camera. By comparing the target depth map and the observation depth map, the corresponding relationship between the positions of each point cloud in the three-dimensional grid in the first two dimensions and the positions of each point in the observation depth map is obtained, and a relationship mapping matrix is generated.

[0040] S120, converting the depth map contained in the three-dimensional grid into a point cloud.

[0041] In the embodiment, each depth map in the three-dimensional grid is converted into a point cloud in the uvd coordinate system through the camera intrinsic parameter corresponding to the three-dimensional grid, and the point cloud satisfies the viewing cone with the camera as the optical center, for example, the viewing cone in the formula. Figure 2

[0042] ​In an optional embodiment, converting the depth maps contained in the three-dimensional grid into point clouds comprises: regarding the points in the three-dimensional grid with the same depth value as a depth map; and determining the point cloud coordinates corresponding to the points in the depth map by multiplying the coordinates of the points in the depth map in the first two dimensions by the inverse of the intrinsic matrix corresponding to the three-dimensional grid.

[0043] For example, for each depth value d i , all the points in the third dimension of the three-dimensional grid V with the same coordinate value d i are regarded as a depth map with the depth d i , so that the three-dimensional grid can be regarded as i depth maps with the depth d i . For each depth map in the three-dimensional grid, the coordinates (u i , v i ) of the points in the depth map in the first two dimensions are multiplied by the inverse of the intrinsic matrix corresponding to the three-dimensional grid, so that the shape of the three-dimensional grid is still [W, H, d i , 3], but the coordinate values of the points in the three-dimensional grid are converted from the "depth map" to the dense "point cloud", which satisfies the viewing frustum with the camera as the optical center. After the depth maps contained in the three-dimensional grid are converted into point clouds, all the point clouds in the three-dimensional grid with the same coordinates in the first two dimensions are located on one of the rays emitted with the camera corresponding to the three-dimensional grid as the optical center.

[0044] S130, traversing the point clouds in the three-dimensional grid, and marking the target point clouds in the three-dimensional grid after the light projection according to the pre-generated relationship mapping matrix and the observation depth map shot by the depth camera.

[0045] The relationship mapping matrix is used to represent the corresponding relationship between the positions of the points in the first two coordinate dimensions in the three-dimensional grid and the positions of the points in the observation depth map. The target point cloud can include the first point on the surface of the object when a ray emitted with the camera as the optical center passes through the object, and other sampling points along the direction of the ray after the first point.

[0046] Optionally, traversing the point clouds in the three-dimensional grid, and marking the target point clouds in the three-dimensional grid after the light projection according to the pre-generated relationship mapping matrix and the observation depth map shot by the depth camera comprises: traversing the point clouds in the three-dimensional grid, and indexing the corresponding depth map position in the relationship mapping matrix according to the coordinates of the point clouds in the first two dimensions in the three-dimensional grid; obtaining the depth value corresponding to the depth map position in the observation depth map, and determining the target depth value corresponding to the point cloud based on the depth value; and marking the point cloud if the coordinate value of the point cloud in the third dimension in the three-dimensional grid is greater than or equal to the target depth value.

[0047] In the process of marking the point cloud in the three-dimensional grid V in this embodiment, all point clouds in the three-dimensional grid need to be traversed, and the coordinates (u i , v i ) of the traversed point cloud in the first two dimensions of the three-dimensional grid are indexed in the relationship mapping matrix to find at least one depth map position corresponding thereto. When the size of the three-dimensional grid in the first two dimensions is smaller than the size of the observed depth map actually observed by the depth camera, one point cloud in the three-dimensional grid corresponds to multiple points in the observed depth map. In an optional implementation, if the indexing obtains multiple depth map positions corresponding to the point cloud, the depth values corresponding to each depth map position in the observed depth map are obtained, the maximum depth value is selected as the target depth value corresponding to the point cloud, or the mean value of the depth values is calculated as the target depth value corresponding to the point cloud. Of course, other ways can also be used to determine the target depth value corresponding to the point cloud. Then, the coordinate value of the traversed point cloud in the third dimension of the three-dimensional grid is compared with the target depth value, and if it is greater than or equal to the target depth value, the point cloud is marked, otherwise the point cloud is not marked.

[0048] In an optional implementation, since the operation performed each time is the same in the process of traversing the point cloud in the three-dimensional grid, that is, twice memory access and sequential comparison operation, the point cloud in the three-dimensional grid can be traversed in parallel using a graphics processing unit (GPU) to further speed up the finding of the target point cloud in the grid and improve the marking efficiency of the target point cloud in the grid.

[0049] It needs to be further explained that the embodiment realizes direct ray casting on the point cloud through the three-dimensional grid containing depth information, and the value of D can be adjusted according to actual needs in the depth direction of the grid to adjust the density of the depth value. In the case of ray casting on the point cloud in voxels, the size of the three-dimensional grid V can also be appropriately adjusted to reduce the complexity.

[0050] The technical scheme of the embodiment of the present disclosure constructs a three-dimensional grid containing depth information, converts the depth map contained in the three-dimensional grid into a point cloud, traverses the point cloud in the three-dimensional grid, and marks the target point cloud that should be marked after ray casting in the three-dimensional grid according to the pre-generated relationship mapping matrix and the observed depth map shot by the depth camera. The relationship mapping matrix is used to represent the corresponding relationship between the position of each point in the first two coordinate dimensions of the three-dimensional grid and the position of each point in the observed depth map, solving the problem of high algorithm complexity and large amount of calculation of the ray casting algorithm in the related art, realizing direct ray casting on the point cloud based on the three-dimensional grid, optimizing the runtime calculation in the related art into the access assignment process of the fixed memory space, reducing the algorithm calculation amount and complexity, and improving the efficiency of marking the point cloud.

[0051] Figure 3 is another three-dimensional grid-based target point cloud marking method flowchart provided by the embodiments of the present disclosure. The embodiments provide another implementation manner of ray casting based on a three-dimensional grid for a point cloud. As shown in Figure 3 , the method comprises:

[0052] S310, constructing a three-dimensional grid containing depth information based on the width, height and custom depth value of the depth map.

[0053] In the embodiments, a three-dimensional grid with a size of [W, H, D, 3] can be constructed in the uvd coordinate system as shown in Figure 2 . Specifically, the coordinate values of each point in the u coordinate axis in the three-dimensional grid can be determined according to the width W of the depth map, i.e., u i = 1, 2, 3, …, W. The coordinate values of each point in the v coordinate axis in the three-dimensional grid can be determined according to the height H of the depth map, i.e., v i = 1, 2, 3, …, H. The real distance S voxel between adjacent points is defined as D*S voxel , where D is a custom depth value, and the coordinate values of each point in the d coordinate axis in the three-dimensional grid are determined as d i = i*S voxel , i = 1, 2, 3, …, D, where D is a modifiable constant.

[0054] S320, converting the depth map contained in the three-dimensional grid into a point cloud according to the camera intrinsic parameter corresponding to the three-dimensional grid.

[0055] In the embodiments, the points with the same depth value in the three-dimensional grid can be regarded as a depth map. For each depth map in the three-dimensional grid, the coordinates of the point cloud corresponding to each point in the depth map can be determined by multiplying the coordinates (u i , v i ) of each point in the depth map by the inverse of the intrinsic matrix corresponding to the three-dimensional grid. At this time, all point clouds with the same coordinates in the first two dimensions in the three-dimensional grid are located on one of the rays emitted from the camera corresponding to the three-dimensional grid. The camera intrinsic parameter corresponding to the three-dimensional grid can be a custom setting or the intrinsic parameter of a specified depth camera.

[0056] S330, obtaining an observation depth map captured by a depth camera, and converting the observation depth map into a projection depth map in the same coordinate system as the three-dimensional grid based on camera intrinsic and extrinsic parameter conversion.

[0057] In this embodiment, in order to determine the correspondence between the positions of each point in the first two coordinate dimensions of the 3D mesh and the positions of each point in the observation depth map, and to facilitate the location and marking of the target point cloud in the 3D mesh, after acquiring the observation depth map captured by the depth camera, the observation depth map needs to be converted into a projected depth map with the same coordinate system as the 3D mesh through camera intrinsic and extrinsic parameter conversion, such as... Figure 2 As shown, the observed depth map can be transformed into a 3D point cloud by multiplying it by the inverse of the depth camera's intrinsic parameter matrix. Then, by utilizing the differences between the depth camera's extrinsic parameters and the camera's extrinsic parameters corresponding to the 3D mesh, the point cloud is translated and rotated accordingly, transforming it to the same extrinsic parameters as the 3D mesh. Finally, based on the camera's intrinsic parameters corresponding to the 3D mesh, the point cloud is converted into a projected depth map in the same coordinate system as the 3D mesh.

[0058] S340. Traverse the point cloud in the 3D mesh and mark the target point cloud that should be marked after the light is projected in the 3D mesh based on the projection depth map.

[0059] In this embodiment, after converting the observed depth map into a projected depth map with the same coordinate system as the 3D mesh, the (u) in the projected depth map... i v i Position and coordinates in the 3D mesh [u i v i The point clouds corresponding to [D, 3] are corresponding. For example, Figure 2 The position (1, 2) in the projection depth map corresponds to the point cloud coordinates (1, 2, 1) and (1, 2, 2) in the 3D mesh. Based on this, a GPU can be used to perform parallel traversal of the point cloud in the 3D mesh. For each traversed point cloud, if the depth value of the point cloud coordinates is greater than or equal to (u) in the projection depth map... i v i If the point cloud is located at a certain depth value, it is marked in the 3D mesh; otherwise, it is not marked. The marked point cloud is the point obtained after the light is projected. For example, suppose the coordinates of the traversed point cloud are (1, 2, 2). The depth value of this point cloud in the projection depth map is 2. Since the depth value of the point cloud coordinates is also 2, the two are equal, so the point cloud can be marked in the 3D mesh.

[0060] The technical scheme of the embodiment of the present disclosure constructs a three-dimensional grid containing depth information based on the width, height and self-defined depth value of the depth map; converts the depth map contained in the three-dimensional grid into a point cloud according to the camera internal parameter corresponding to the three-dimensional grid; obtains an observation depth map photographed by the depth camera, and converts the observation depth map into a projection depth map in the same coordinate system as the three-dimensional grid based on the camera internal and external parameters; and marks the target point cloud that should be marked after light projection in the three-dimensional grid based on the projection depth map. The light projection on the point cloud based on the three-dimensional grid is realized, the point cloud is marked by converting the observation depth map into the same coordinate system as the three-dimensional grid, the algorithm operation amount and complexity are reduced, and the efficiency of marking the point cloud is improved.

[0061] Figure 4 is a three-dimensional grid-based target point cloud marking device structure schematic diagram provided by the embodiment of the present disclosure, as Figure 4 shown, the device comprises a grid construction module 410, a conversion module 420 and a point cloud marking module 430.

[0062] The grid construction module 410 is configured to construct a three-dimensional grid containing depth information.

[0063] The conversion module 420 is configured to convert the depth map contained in the three-dimensional grid into a point cloud.

[0064] The point cloud marking module 430 is configured to traverse the point cloud in the three-dimensional grid, and mark the target point cloud that should be marked after light projection in the three-dimensional grid according to the pre-generated relationship mapping matrix and the observation depth map photographed by the depth camera.

[0065] The relationship mapping matrix is used to represent the corresponding relationship between the position of each point in the first two coordinate dimensions of the three-dimensional grid and the position of each point in the observation depth map.

[0066] The technical scheme provided by the embodiment of the present disclosure constructs a three-dimensional grid containing depth information; converts the depth map contained in the three-dimensional grid into a point cloud; traverses the point cloud in the three-dimensional grid, and marks the target point cloud that should be marked after light projection in the three-dimensional grid according to the pre-generated relationship mapping matrix and the observation depth map photographed by the depth camera; and the relationship mapping matrix is used to represent the corresponding relationship between the position of each point in the first two coordinate dimensions of the three-dimensional grid and the position of each point in the observation depth map. The problem of high algorithm complexity and large operation amount of the light projection algorithm in the related art is solved, the light projection on the point cloud based on the three-dimensional grid is realized, the runtime operation in the related art is optimized to the access assignment process in the fixed memory space by pre-generating the relationship mapping matrix, the algorithm operation amount and complexity are reduced, and the efficiency of marking the point cloud is improved.

[0067] Optionally, the grid construction module 410 is configured to:

[0068] constructing a three-dimensional grid under a specified coordinate system based on the width and height of the depth map and the custom depth value set according to the real distance between two adjacent points;

[0069] wherein the width of the depth map, the height of the depth map and the custom depth value correspond to three dimensions of the specified coordinate system in sequence.

[0070] Optionally, the conversion module 420 is configured to:

[0071] regard points with the same depth value in the three-dimensional grid as a depth map;

[0072] for each depth map in the three-dimensional grid, determine the point cloud coordinates corresponding to each point in the depth map by multiplying the coordinates of each point in the depth map in the first two dimensions by the inverse of the intrinsic matrix corresponding to the three-dimensional grid.

[0073] Optionally, the method further comprises: after constructing the three-dimensional grid containing depth information,

[0074] converting the depth map contained in the three-dimensional grid into a point cloud according to the camera intrinsic parameter corresponding to the three-dimensional grid;

[0075] converting the point cloud to the same camera extrinsic parameter as the depth camera through the camera extrinsic parameter corresponding to the three-dimensional grid;

[0076] converting the point cloud into a target depth map in the same coordinate system as the observed depth map captured by the depth camera through the same camera intrinsic parameter as the depth camera;

[0077] determining the correspondence between the positions of each point cloud in the three-dimensional grid in the first two dimensions and the positions of each point in the observed depth map according to the target depth map, and generating a relationship mapping matrix.

[0078] Optionally, the point cloud marking module 430 is configured to:

[0079] traversing the point cloud in the three-dimensional grid, and indexing the corresponding depth map position in the relationship mapping matrix according to the coordinates of the point cloud in the first two dimensions in the three-dimensional grid;

[0080] obtaining the depth value corresponding to the depth map position in the observed depth map, and determining the target depth value corresponding to the point cloud based on the depth value;

[0081] if the coordinate value of the point cloud in the third dimension in the three-dimensional grid is greater than or equal to the target depth value, marking the point cloud.

[0082] Optionally, the point cloud marking module 430 is configured to:

[0083] If the index corresponds to multiple depth map positions of the point cloud, a depth value corresponding to each depth map position in the observation depth map is obtained;

[0084] The maximum depth value is selected as the target depth value corresponding to the point cloud; or

[0085] The average of the depth values is calculated as the target depth value corresponding to the point cloud.

[0086] Optionally, the point cloud marking module 430 is configured to perform parallel traversal on the point cloud in the three-dimensional grid using a graphics processing unit (GPU).

[0087] The target point cloud marking apparatus based on a three-dimensional grid provided in the embodiments of the present disclosure can perform the target point cloud marking method based on a three-dimensional grid provided in any of the embodiments of the present disclosure, and has the corresponding function modules and advantages of performing the method.

[0088] It should be noted that each unit and module included in the apparatus is only divided according to the function logic, but is not limited to the above division, as long as the corresponding function can be implemented; in addition, the specific name of each functional unit is only for convenient mutual distinction, and does not serve to limit the protection scope of the embodiments of the present disclosure.

[0089] Figure 5 A structural schematic diagram of an electronic device provided in the embodiments of the present disclosure is shown in FIG. 5. Hereinafter, the structure of the electronic device (for example, a terminal device or a server) 500 suitable for implementing the embodiments of the present disclosure will be described with reference to FIG. 5. Figure 5 The terminal device in the embodiments of the present disclosure can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet Personal Computer), a PMP (Portable Multimedia Player), a vehicle terminal (for example, a car navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 5 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure. Figure 5 As shown in FIG. 5, the electronic device 500 can include a processing device (for example, a central processing unit, a graphics processing unit, and the like) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0090] Figure 5 As shown in FIG. 5, the electronic device 500 can include a processing device (for example, a central processing unit, a graphics processing unit, and the like) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.​

[0091] In general, the following devices can be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 509. The communication devices 509 can allow the electronic device 500 to communicate wirelessly or wired with other devices to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it is understood that all of the shown devices need not be implemented or present. More or fewer devices can alternatively be implemented or present.

[0092] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 509, or installed from the storage devices 508, or installed from the ROM 502. When the computer program is executed by the processing devices 501, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are performed.

[0093] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0094] The electronic device provided by the embodiments of the present disclosure and the target point cloud labeling method based on a three-dimensional grid provided by the above-mentioned embodiments belong to the same inventive concept, and the technical details not described in detail in the present embodiment can be referred to the above-mentioned embodiments, and the present embodiment has the same beneficial effects as the above-mentioned embodiments.

[0095] The embodiments of the present disclosure provide a computer storage medium, which stores a computer program, and the program is executed by a processor to implement the target point cloud labeling method based on a three-dimensional grid provided by the above-mentioned embodiments.

[0096] It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the disclosure, the computer-readable signal medium can include a data signal propagated in baseband or as a carrier wave in a propagated data signal, in which the computer-readable program code is contained. Such a propagated data signal can take many forms, including but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium that can send, propagate or transfer a program for use by or in connection with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to, wire, cable, RF (radio frequency), etc., or any suitable combination of the above.

[0097] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0098] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device, and is not assembled into the electronic device.

[0099] The computer readable medium carries one or more programs when the one or more programs are executed by the electronic device, so that the electronic device: constructs a three-dimensional grid containing depth information; converts the depth map contained in the three-dimensional grid into a point cloud; traverses the point cloud in the three-dimensional grid, and according to a pre-generated relationship mapping matrix and an observed depth map shot by the depth camera, marks the target point cloud that should be marked after light projection in the three-dimensional grid; and the relationship mapping matrix is used to represent the corresponding relationship between the position of each point in the first two coordinate dimensions in the three-dimensional grid and the position of each point in the observed depth map.

[0100] Computer program code for carrying out operations of the present disclosure can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0101] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, can be implemented by special purpose hardware-based systems that perform the specified functions or operations, or combinations of special purpose hardware and computer instructions.

[0102] The units described in the embodiments of the present disclosure can be implemented by software or by hardware. In some cases, the names of the units do not constitute a limitation on the units themselves, for example, the first obtaining unit can also be described as an "obtaining at least two internet protocol addresses unit".

[0103] The functionality described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Application-specific Integrated Circuits (ASICs), Application-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0104] In the context of this disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0105] According to one or more embodiments of the present disclosure, example 1 provides a three-dimensional grid-based target point cloud labeling method, comprising:

[0106] constructing a three-dimensional grid containing depth information;

[0107] converting the depth map contained in the three-dimensional grid into a point cloud;

[0108] traversing the point cloud in the three-dimensional grid, and according to the pre-generated relationship mapping matrix and the observed depth map taken by the depth camera, marking the target point cloud to be marked after light projection in the three-dimensional grid;

[0109] wherein the relationship mapping matrix is used to represent the corresponding relationship between the position of each point in the first two coordinate dimensions of the three-dimensional grid and the position of each point in the observed depth map.

[0110] According to one or more embodiments of the present disclosure, example 2, according to the method of example 1, constructing a three-dimensional grid containing depth information, comprises:

[0111] constructing a three-dimensional grid in a specified coordinate system based on the width and height of the depth map and the custom depth value set according to the real distance between adjacent two points;

[0112] wherein the width of the depth map, the height of the depth map, and the custom depth value correspond to the three dimensions of the specified coordinate system in order.

[0113] According to one or more embodiments of this disclosure, Example 3 converts a depth map contained in a three-dimensional mesh into a point cloud according to the method of Example 1, including:

[0114] Points with the same depth value in a 3D mesh are considered as a depth map;

[0115] For each depth map in the 3D mesh, the point cloud coordinates corresponding to each point in the depth map are determined by multiplying the coordinates of each point in the depth map in the first two dimensions by the inverse of the intrinsic parameter matrix corresponding to the 3D mesh.

[0116] According to one or more embodiments of this disclosure, Example 4, based on the method of Example 1, further includes, after constructing a 3D mesh containing depth information:

[0117] Based on the camera intrinsic parameters corresponding to the 3D mesh, the depth map contained in the 3D mesh is converted into a point cloud;

[0118] By using the camera extrinsics corresponding to the 3D mesh, the point cloud is transformed to the same camera extrinsics as the depth camera;

[0119] Using the same camera intrinsics as the depth camera, the point cloud is converted into a target depth map in the same coordinate system as the observation depth map captured by the depth camera;

[0120] Based on the target depth map, determine the correspondence between the position of each point cloud in the three-dimensional grid in the first two dimensions and the position of each point in the observed depth map, and generate a relationship mapping matrix.

[0121] According to one or more embodiments of this disclosure, Example 5, following the method of Example 4, traverses the point cloud in a three-dimensional mesh and, based on a pre-generated relational mapping matrix and an observation depth map captured by a depth camera, marks the target point cloud that should be marked after ray projection in the three-dimensional mesh, including:

[0122] Traverse the point cloud in the 3D mesh, and obtain the corresponding depth map position by indexing the coordinates of the point cloud in the first two dimensions of the 3D mesh in the relation mapping matrix.

[0123] Obtain the depth value corresponding to the location in the depth map in the observed depth map, and determine the target depth value corresponding to the point cloud based on the depth value;

[0124] If the coordinate value of the point cloud in the third dimension of the 3D mesh is greater than or equal to the target depth value, then the point cloud is marked.

[0125] According to one or more embodiments of this disclosure, Example 6, based on the method of Example 5, obtains the depth value corresponding to the location in the observed depth map, and determines the target depth value corresponding to the point cloud based on the depth value, including:

[0126] If the index corresponds to multiple depth map positions corresponding to the point cloud, obtaining a depth value corresponding to each depth map position in the observation depth map;

[0127] selecting a maximum depth value as the target depth value corresponding to the point cloud; or

[0128] calculating a mean value of the depth values as the target depth value corresponding to the point cloud.

[0129] According to one or more embodiments of the present disclosure, example 7 provides a method according to example 1, wherein traversing the point cloud in the three-dimensional grid comprises: using a graphics processing unit (GPU) to traverse the point cloud in the three-dimensional grid in parallel.

[0130] According to one or more embodiments of the present disclosure, example 8 provides a three-dimensional grid-based target point cloud marking device, comprising:

[0131] a grid construction module configured to construct a three-dimensional grid containing depth information;

[0132] a conversion module configured to convert a depth map contained in the three-dimensional grid into a point cloud;

[0133] a point cloud marking module configured to traverse the point cloud in the three-dimensional grid, and mark a target point cloud to be marked after light projection in the three-dimensional grid according to a pre-generated relationship mapping matrix and an observation depth map captured by a depth camera.

[0134] The relationship mapping matrix is used to represent a correspondence between a position of each point in the first two coordinate dimensions of the three-dimensional grid and a position of each point in the observation depth map.

[0135] According to one or more embodiments of the present disclosure, example 9 provides an electronic device, comprising:

[0136] one or more processors;

[0137] a storage device configured to store one or more programs,

[0138] when the one or more programs are executed by the one or more processors, the one or more processors implement a three-dimensional grid-based target point cloud marking method as in any one of examples 1-7.

[0139] According to one or more embodiments of the present disclosure, example 10 provides a storage medium containing computer executable instructions for performing a three-dimensional grid-based target point cloud marking method as in any one of examples 1-7 when executed by a computer processor.

[0140] The above description merely illustrates the preferred embodiments of the disclosure and a principle for applying the technologies. It is understood by those skilled in the art that the disclosed scope of the disclosure is not limited to the technical solutions formed by the specific combinations of the technical features described above, and should also cover other technical solutions formed by the combinations of the technical features described above or their equivalent features without departing from the disclosed concept. For example, the technical solutions formed by the mutual replacement of the above-described features and the technical features with similar functions disclosed in the disclosure (but not limited to) can be formed.

[0141] Further, although operations are depicted in a particular, sequential order, this should not be understood as requiring or implying that the operations are performed in the order illustrated or sequentially. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, although specific implementation details are included for the purpose of providing a thorough disclosure, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0142] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A target point cloud marking method based on a three-dimensional mesh, characterized in that, include: Construct a 3D mesh that includes depth information; Convert the depth map contained in the 3D mesh into a point cloud; Traverse the point cloud in the three-dimensional grid, and mark the target point cloud that should be marked after the light is projected in the three-dimensional grid according to the pre-generated relation mapping matrix and the observation depth map captured by the depth camera; The relational mapping matrix is ​​used to represent the correspondence between the positions of each point in the first two coordinate dimensions of the three-dimensional grid and the positions of each point in the observation depth map; After constructing the 3D network containing depth information, the process also includes: Based on the camera intrinsic parameters corresponding to the 3D mesh, the depth map contained in the 3D mesh is converted into a point cloud; By using the camera extrinsic parameters corresponding to the 3D mesh, the point cloud is transformed to the same camera extrinsic parameters as the depth camera; Using the same camera intrinsic parameters as the depth camera, the point cloud is converted into a target depth map in the same coordinate system as the observation depth map captured by the depth camera; Based on the target depth map, determine the correspondence between the position of each point cloud in the three-dimensional grid in the first two dimensions and the position of each point in the observation depth map, and generate a relationship mapping matrix.

2. The method according to claim 1, characterized in that, The construction of the 3D mesh containing depth information includes: A 3D mesh is constructed in a specified coordinate system based on the width and height of the depth map and a custom depth value set according to the actual distance between two adjacent points. The width, height, and custom depth values ​​of the depth map correspond to the three dimensions of the specified coordinate system.

3. The method according to claim 1, characterized in that, The step of converting the depth map contained in the 3D mesh into a point cloud includes: Points with the same depth value in the three-dimensional mesh are considered as a depth map; For each depth map in the three-dimensional mesh, the point cloud coordinates corresponding to each point in the depth map are determined by multiplying the coordinates of each point in the depth map in the first two dimensions by the inverse of the intrinsic parameter matrix corresponding to the three-dimensional mesh.

4. The method according to claim 1, characterized in that, The process of traversing the point cloud in the 3D mesh, based on a pre-generated relational mapping matrix and the observation depth map captured by the depth camera, marks the target point cloud that should be marked after ray projection in the 3D mesh, including: Traverse the point cloud in the three-dimensional mesh, and obtain the corresponding depth map position by indexing the coordinates of the point cloud in the first two dimensions of the three-dimensional mesh in the relational mapping matrix. Obtain the depth value corresponding to the location in the depth map in the observation depth map, and determine the target depth value corresponding to the point cloud based on the depth value; If the coordinate value of the point cloud in the third dimension of the three-dimensional mesh is greater than or equal to the target depth value, then the point cloud is marked.

5. The method according to claim 4, characterized in that, The step of obtaining the depth value corresponding to the location in the depth map in the observation depth map, and determining the target depth value corresponding to the point cloud based on the depth value, includes: If the index yields multiple depth map locations corresponding to the point cloud, then the depth value corresponding to each depth map location in the observed depth map is obtained; Select the maximum depth value as the target depth value corresponding to the point cloud; or The mean of each depth value is calculated and used as the target depth value corresponding to the point cloud.

6. The method according to claim 1, characterized in that, The traversal of the point cloud in the 3D mesh includes: using a graphics processing unit (GPU) to perform parallel traversal of the point cloud in the 3D mesh.

7. A target point cloud marking device based on a three-dimensional mesh, characterized in that, include: The mesh building module is used to build 3D meshes that include depth information; A conversion module is used to convert the depth map contained in the 3D mesh into a point cloud; The point cloud marking module is used to traverse the point cloud in the three-dimensional grid and mark the target point cloud that should be marked after the light is projected in the three-dimensional grid according to the pre-generated relation mapping matrix and the observation depth map captured by the depth camera. The relational mapping matrix is ​​used to represent the correspondence between the positions of each point in the first two coordinate dimensions of the three-dimensional grid and the positions of each point in the observation depth map; The network construction module, after constructing a 3D network containing depth information, is further used for: Based on the camera intrinsic parameters corresponding to the 3D mesh, the depth map contained in the 3D mesh is converted into a point cloud; By using the camera extrinsic parameters corresponding to the 3D mesh, the point cloud is transformed to the same camera extrinsic parameters as the depth camera; Using the same camera intrinsic parameters as the depth camera, the point cloud is converted into a target depth map in the same coordinate system as the observation depth map captured by the depth camera; Based on the target depth map, determine the correspondence between the position of each point cloud in the three-dimensional grid in the first two dimensions and the position of each point in the observation depth map, and generate a relationship mapping matrix.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the target point cloud marking method based on a three-dimensional mesh as described in any one of claims 1-6.

9. A storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform the target point cloud marking method based on a three-dimensional mesh as described in any one of claims 1-6.

Images