Point cloud normal vector estimation method, device, computer equipment and storage medium

Abstract

The present application is applicable to the field of computer technology, and provides a method, device, computer equipment and storage medium for estimating a point cloud normal vector. The method for estimating a point cloud normal vector includes: obtaining point cloud data and a color image corresponding to the point cloud data; associating each coordinate point in the target point cloud area in the point cloud data with the color image, and determining the image feature information corresponding to each coordinate point in the target point cloud area; fusing the coordinate information of each coordinate point in the target point cloud area with the corresponding image feature information, and determining the normal vector of the target point cloud area according to the fusion result. This scheme obtains the image feature information corresponding to each coordinate point in a point cloud area, fuses the coordinate information of each coordinate point with the corresponding image feature, and then determines the normal vector of the point cloud area according to the fusion result. Compared with fitting the point cloud area into a plane to use the normal vector of the fitted plane as the normal vector of the point cloud area, the estimation accuracy of the normal vector is improved.

Description

Technical Field

[0001] The present application belongs to the technical field of computer science, and in particular, relates to a method, device, computer equipment and storage medium for estimating a point cloud normal vector. Background Art

[0002] With the development of science and technology, artificial intelligence has gradually penetrated into our lives. For example, driverless cars and indoor robots have the ability to sense the environment and human visual functions through computer vision technology. The processing of three-dimensional point cloud data is one of the important research directions of computer vision technology. Point cloud normal vector estimation is a basic task of three-dimensional point cloud processing. Accurate normal vector estimation can be directly used for tasks such as three-dimensional reconstruction, point cloud registration and scene rendering. A point cloud is a collection of three-dimensional coordinate points. Each point in the point cloud can be regarded as a coordinate point in space. We generally obtain it directly from space detection sensors such as laser radar and binocular cameras. The normal vector of a point cloud refers to the vector in the vertical direction corresponding to each three-dimensional coordinate point in the point cloud on the surface of a real object.

[0003] In the existing point cloud data processing process, the estimation of the point cloud normal vector is generally through a regression model based on principal component analysis, which directly fits the point cloud of a local area of ​​a preset size into a plane, and then calculates the normal vector of the plane as the normal vector of the point cloud in the local area.

[0004] It can be seen that the accuracy of the estimated point cloud normal vector is low using the existing point cloud normal vector estimation method. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a point cloud normal vector estimation method, apparatus, computer equipment and storage medium, which can solve the technical problem of low accuracy of the point cloud normal vector estimated using the existing point cloud normal vector estimation method.

[0006] A first aspect of an embodiment of the present application provides a method for estimating a point cloud normal vector, comprising:

[0007] Acquire point cloud data collected by the first device and a color image corresponding to the point cloud data collected by the second device;

[0008] Associating each coordinate point in a target point cloud region in the point cloud data with the color image to determine image feature information corresponding to each coordinate point in the target point cloud region; the target point cloud region is any point cloud region in the point cloud data;

[0009] The coordinate information of each coordinate point in the target point cloud area is fused with its corresponding image feature information, and the normal vector of the target point cloud area is determined based on the fusion result. In a possible implementation of the first aspect, the target point cloud area can be obtained by first acquiring point cloud data, and then generating a number of point cloud areas from the point cloud data. Exemplarily, each coordinate in the point cloud data can be used as a center point to search for coordinate points within a specified range around the center point. Each center point and the points within a specified range around it constitute a point cloud area. It should be understood that the above is only one implementation method for obtaining a point cloud area, and the method for obtaining a point cloud area is not limited to this.

[0010] A second aspect of an embodiment of the present application provides a point cloud normal vector estimation device, the point cloud normal vector estimation device comprising:

[0011] An acquisition module, used to acquire point cloud data collected by the first device and a color image corresponding to the point cloud data collected by the second device;

[0012] an association module, used to associate each coordinate point in a target point cloud region in the point cloud data with the color image, and determine image feature information corresponding to each coordinate point in the target point cloud region; the target point cloud region is any point cloud region in the point cloud data;

[0013] The normal vector determination module is used to fuse the coordinate information of each coordinate point in the target point cloud area with its corresponding image feature information, and determine the normal vector of the target point cloud area according to the fusion result.

[0014] A third aspect of an embodiment of the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the point cloud normal vector estimation method described in the first aspect above are implemented.

[0015] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, comprising: a computer program stored therein, which, when executed by a processor, implements the steps of the method for estimating a point cloud normal vector as described in the first aspect above.

[0016] The beneficial effects of the embodiments of the present application compared to the prior art are as follows: the present application solution can obtain the image feature information of each coordinate point in the point cloud area by associating each point in a point cloud area in the point cloud data with the image corresponding to the point cloud data, and then fuse the coordinate information of each coordinate point in the point cloud area with its corresponding image feature, and further determine the normal vector of the point cloud area according to the fusion result. When the normal vectors of all point cloud areas in the point cloud data are determined, the estimated result of the normal vector of the entire point cloud data is obtained. Compared with the prior art for estimating the normal vector of the point cloud, this solution can effectively improve the accuracy of the estimation of the normal vector of the point cloud data by fitting the local area in the point cloud data into a plane and then using the normal vector of the fitted plane as the normal vector of the local area of ​​the point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Figure 1 is an application environment diagram of the point cloud normal vector estimation method provided in an embodiment of the present application;

[0019] Figure 2 is a flow chart of a method for estimating a point cloud normal vector provided in an embodiment of the present application;

[0020] Figure 3 Schematic diagram of a point cloud collection of a scene collected by a point cloud collection device

[0021] Figure 4 is a flow chart of determining image feature information corresponding to each point in a target point cloud area provided by an embodiment of the present application;

[0022] Figure 5 It is a flowchart of fusing the coordinate information of each point in the target point cloud area with the corresponding image feature information provided by the embodiment of the present application, and determining the normal vector of the target point cloud area according to the fusion result;

[0023] Figure 6 A schematic diagram of a schematic diagram of a process for encoding coordinate information of a point in a point cloud region into high-dimensional feature information provided by an embodiment of the present application;

[0024] Figure 7 A schematic diagram of determining the normal vector of a target point cloud area according to the spliced ​​fusion vector provided in an embodiment of the present application;

[0025] Figure 8is a flowchart of obtaining training data for a normal vector estimation model provided in an embodiment of the present application;

[0026] Fig. 9 A projection diagram of sample point cloud data on a camera plane provided in an embodiment of the present application;

[0027] Fig.10 A virtual point cloud projection diagram obtained in an embodiment of the present application;

[0028] Fig.11 is a structural block diagram of a point cloud data processing device provided in an embodiment of the present application;

[0029] Fig.12 This is a block diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0031] In order to illustrate the technical solution described in this application, a specific embodiment is provided below for illustration.

[0032] Figure 1 An application environment diagram of a point cloud normal vector estimation method provided in an embodiment of the present application, such as Figure 1 As shown, in the application environment, a computer device 110 , a first device 120 and a second device 130 are included.

[0033] The computer device 110 may be an independent physical server or terminal, or a server cluster composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud servers, cloud databases, cloud storage, and CDN.

[0034] The first device 120 is a point cloud acquisition device, which may be a radar device, but is not limited thereto. The first device 120 may be connected to the computer device 110 for communication.

[0035] The second device 130 may be an image acquisition device, and the image acquisition device may be a camera, but is not limited thereto. The second device 130 may be communicatively connected to a computer device.

[0036] like Figure 2As shown, in one embodiment, a method for estimating a point cloud normal vector is proposed. This embodiment mainly applies this method to the above Figure 1 A method for estimating a point cloud normal vector may specifically include the following steps:

[0037] Step S202: Acquire the point cloud data collected by the first device and the color image corresponding to the point cloud data collected by the second device.

[0038] In the embodiment of the present application, point cloud data refers to point cloud data collected by a point cloud collection device, which is a collection of multiple three-dimensional coordinate points. Figure 3 As shown, it is a point cloud image of a scene collected by the point cloud acquisition device. The elliptical area in the upper right corner of the figure is a local enlarged image of the corresponding position in the figure. All the coordinate points collected in the scene constitute the point cloud data of the scene. The color image corresponding to the point cloud data is the color image of the scene corresponding to the point cloud data collected by the image acquisition device. Each coordinate point in the point cloud data collected by the point cloud acquisition device can correspond to the spatial coordinate information of each object in the color image at the coordinate point.

[0039] Step S204, associating each coordinate point in the target point cloud area in the point cloud data with the color image to determine the image feature information corresponding to each coordinate point in the target point cloud area; the target point cloud area is any point cloud area in the point cloud data.

[0040] In the embodiment of the present application, the target point cloud region can be any point cloud region in the point cloud data, and the point cloud region is a local region including a plurality of coordinate points in the coordinate point set of the point cloud data. By dividing the point cloud data into a plurality of point cloud regions, the normal vector of each coordinate point in the point cloud data can be calculated, and then the normal vector of each point cloud region can be estimated. In the point cloud normal vector estimation method, the point cloud region whose normal vector is calculated is the target point cloud region.

[0041] In the embodiment of the present application, before associating each coordinate point in the target point cloud area in the point cloud data with the color image corresponding to the point cloud data, the point cloud data is first generated into a plurality of point cloud areas. The point cloud data is generated into a plurality of point cloud areas, including: taking each coordinate point in the point cloud data as a center point, searching for coordinate points within a spherical range according to a specified radius, and each of the center points and the coordinate points within the spherical range constitute a point cloud area.

[0042] In an embodiment of the present application, when a point in the coordinate point set of the point cloud data is taken as the center point, a spherical search is performed around the center point with a specified radius, and the coordinate points within the spherical range constitute a point cloud area. When each coordinate point in the point cloud data is taken as the center point, the point cloud data can be used to generate several point cloud areas. There is no restriction on the specific value of the specified radius r. For example, in KITTI data, we can select the size of the horizontal span multiplied by 0.01 as the specified radius. For example, when we process point cloud observations frame by frame, for the current frame (a point cloud), assume that the farthest point on the left side of the radar is A, and the farthest point on the right side is B. Then the horizontal span refers to the lateral distance between A and B, that is, the depths of A and B for the left and right sides are added together. Specifically, this span may be 30, so the specified radius is 30*0.01=0.3. The number of coordinate points contained in several point cloud areas generated according to the point cloud data may all be specified values. This embodiment does not limit the specific numerical value of the specified value. For example, the specified value may be 20. In order to make the number of points in each point cloud area generated by the point cloud data all have specified values, after performing a spherical search, the number of coordinate points obtained by searching within the spherical search range may be compared with the specified value. When the number of points obtained by searching within the spherical range is less than the specified value, points consistent with the coordinates of the center point may be added to the point cloud area so that the number of points contained in the point cloud area is the specified value. The points that are added to the point cloud area and consistent with the coordinates of the center point refer to the points that are added to the point cloud area and have the coordinates of the center point assigned to them. Taking the specified value of 20 as an example, since the point cloud data has numerical information of three spatial dimensions, each target point cloud area can be described by a 20×3 matrix P, and then the geometric center and radius r of the target point cloud area are normalized to the entire local patch so that the values ​​are stably distributed around 0. After normalization, the values ​​of the 20×3 matrix storing the original coordinates can be converted into another 20×3 matrix whose element values ​​are stably distributed around 0. The matrix formula of the target point cloud area can be specifically expressed as: Among them, P is the matrix expression of the target point cloud area, row is the row index of the matrix, norm refers to the normalization of the data, r is the specified radius, and 20 is the specified value of the number of point clouds in the target point cloud area.

[0043] In an embodiment of the present application, the feature information of an image may also be referred to as the semantic information of the image. Both the image feature information and the image semantic information are information that can be used to represent features such as color features, texture features, or shape features of the image. For example, both the image feature information and the image semantic information may include image size information, pixel information, etc., but are not limited to this. The extraction of image feature information is the extraction of image semantic information. This embodiment does not limit the method for extracting the semantic information of the image. For example, it can be obtained through network models such as U-Net (U Neural Network) or Res-Net (Residual Neural Network) or VGG (Visual Geometry Group).

[0044] In the embodiment of the present application, there is no limitation on the specific method of associating each coordinate point in the target point cloud area with the color image corresponding to the point cloud data to determine the image feature information corresponding to each coordinate point in the target point cloud area. For example, Figure 4 As shown, step S204 may specifically include the following steps:

[0045] Step S402, projecting each coordinate point in the target cloud area into the camera imaging plane where the color image is located, to obtain a projection point corresponding to each coordinate point in the target point cloud area.

[0046] In the embodiment of the present application, projecting each point in the target point cloud area to the camera imaging plane where the image is located can be achieved by coordinate transformation. For example, the transformation matrix T from the radar coordinate system to the camera coordinate system and the camera's internal parameter matrix can be used to project the spatial coordinates of 20 points in each target point cloud area to the camera imaging plane, thereby realizing the association of each coordinate point in the target point cloud area with the color image corresponding to the point cloud data. The calculation formula of the obtained projection point is as follows: Where P 1 is the coordinate of the projection point, P 2 is the coordinate of the point corresponding to the projection point in the target point cloud area, K is the intrinsic parameter matrix of the camera, and T is the transformation matrix from the radar coordinate system to the camera coordinate system.

[0047] Step S404: extracting image feature information corresponding to the projection point position in the color image to obtain image feature information of each coordinate point in the target point cloud area.

[0048] In an embodiment of the present application, before associating each coordinate point in the target point cloud area with the color image corresponding to the point cloud set, the image feature information of the color image corresponding to the point cloud data can be obtained first. For example, the semantic information of the color image corresponding to the point cloud data can be obtained through the U-Net network model, thereby obtaining the image feature map of the color image corresponding to the point cloud data, and then the image feature information of each point in the image can be obtained from the image feature map. The U-Net network model includes an encoder-decoder architecture and a residual connection operation, and the length and width of its output features are consistent with the input, which can ensure that the obtained image feature information is subsequently associated with the coordinate information using a fully convolutional neural network. The U-Net network model can also receive inputs of any size, which is convenient for generalization to other scenarios in the future. In addition, each position on the image feature map output by the U-Net network model is associated with the pixel area at the same position in the color image corresponding to the point cloud data. For example, the image feature information value at the coordinate position (10,20) on the image feature map must be calculated from the area near the coordinate position (10,20) in the image corresponding to the point cloud set. This method of obtaining image feature information of the image corresponding to the point cloud set is consistent with the idea of ​​processing point cloud data through the target point cloud area, and both use a unified local idea.

[0049] In an embodiment of the present application, when the image feature information of the color image corresponding to the known point cloud data is obtained, after the projection point corresponding to each coordinate point in the target point cloud area is obtained, the image feature information corresponding to the projection point can be obtained, that is, the image feature information of each coordinate point in the target point cloud area is obtained.

[0050] Step S206 , fusing the coordinate information of each point in the target point cloud region with its corresponding image feature information, and determining the normal vector of the target point cloud region according to the fusion result.

[0051] In the embodiment of the present application, the coordinate information of each coordinate point in the target point cloud region is the three-dimensional coordinate information of each point in the spatial coordinate system, and the three-dimensional coordinate information can be directly obtained from the point cloud data collected by the point cloud data collection device 120. This embodiment does not limit the specific fusion method for fusing the coordinate information of each point in the target point cloud region with its corresponding image feature information, and the method for determining the normal vector of the target point cloud region based on the fusion result. For example, Figure 5 As shown, step S206 may specifically include the following steps:

[0052] S502 : splicing the image feature information of each coordinate point in the target point cloud region with the coordinate information of the coordinate point to obtain a first splicing vector corresponding to each coordinate point.

[0053] In the embodiment of the present application, the image feature information is taken as a vector composed of length dimension value, width dimension value and color value R (red) value, G (green) value and B (blue) value as an example, and the coordinate information is a vector composed of the coordinate values ​​of the point in three directions in space, and the first splicing vector corresponding to each point is obtained as a vector composed of length dimension value, width dimension value, R value, G value, B value and coordinate values ​​in three directions in space. Of course, before splicing the image feature information of each point in the target point cloud area and the coordinate information of the coordinate point, the coordinate information of each coordinate point in the target point cloud area can be encoded as a high-dimensional feature, so that the coordinate information encoded as a high-dimensional feature can be spliced ​​and fused with the image feature information. The present embodiment does not limit the specific method of encoding the coordinate information as a high-dimensional feature. For example, Figure 6 As shown in the figure, it is a schematic diagram of the process of encoding the coordinate information of the coordinate points in the point cloud area into high-dimensional feature information. The three-dimensional coordinate information of the point cloud can be passed through three one-dimensional convolutional networks Conv1d in sequence, which perform the same multi-layer perceptron operation on the input 20 three-dimensional vectors respectively, and finally map them to a 256-dimensional space, thereby realizing the encoding of the 3D coordinate information into high-dimensional feature information.

[0054] S504 , performing a mean pooling operation on the first splicing vectors corresponding to all coordinate points in the target point cloud region to obtain a second splicing vector corresponding to the target point cloud region.

[0055] In an embodiment of the present application, a point cloud area includes 20 coordinate points, and each coordinate point can correspond to a corresponding first splicing vector. This embodiment does not limit the method of calculating the mean of the first splicing vectors corresponding to all coordinate points in the target point cloud area. For example, the 20 first splicing vectors corresponding to the 20 coordinate points in the target point cloud area can be processed by one-dimensional convolution to transform the length of each first splicing vector, and then the 20 first splicing vectors are compressed into a single vector through a mean pooling operation to obtain a second splicing vector.

[0056] S506, splicing the first splicing vector of each coordinate point in the target point cloud area with the second splicing vector, or splicing the image feature information of each coordinate point in the target point cloud area with the second splicing vector, or splicing the coordinate information of each coordinate point in the target point cloud area with the second splicing vector to determine a splicing fusion vector.

[0057] In an embodiment of the present application, the coordinate information of each coordinate point in the target point cloud region is fused with its corresponding image feature information, and the obtained fusion result is a spliced ​​fusion vector, wherein the spliced ​​fusion vector can be a vector obtained by splicing the first splicing vector and the second splicing vector of each coordinate point in the target point cloud region, or a vector obtained by splicing the image feature information and the second splicing vector of each coordinate point in the target point cloud region, or a vector obtained by splicing the coordinate information and the second splicing vector of each coordinate point in the target point cloud region. Preferably, the vector obtained by splicing the first splicing vector and the second splicing vector can be used as the spliced ​​fusion vector. Since the first splicing vector contains the image feature information and coordinate information corresponding to each coordinate point in the target point cloud region, the vector obtained by splicing the first splicing vector and the second splicing vector is used as the spliced ​​fusion vector, which is equivalent to the obtained spliced ​​fusion vector obtained by the image feature information, coordinate information and the second splicing vector of a coordinate point, so that the normal vector of the target point cloud region is further calculated according to the spliced ​​fusion vector obtained by splicing the first splicing vector and the second splicing vector, thereby improving the calculation accuracy of the normal vector.

[0058] S508: Determine a normal vector of the target point cloud area according to the spliced ​​fusion vector.

[0059] In an embodiment of the present application, there is no limitation on the specific method of determining the normal vector of the target point cloud area according to the splicing and fusion vector. For example, the normal vector of the target point cloud area can be determined according to the splicing and fusion vector through a normal vector estimation network model.

[0060] This embodiment does not limit the specific structure of the normal vector estimation network model. For example, the normal vector estimation network model may include two one-dimensional convolutional networks, a maximum pooling function, and three fully connected neural network layers, such as Figure 7 As shown in FIG. 1 , it is a schematic diagram of the process of determining the normal vector of the target point cloud area according to the spliced ​​fusion vector. For example, the spliced ​​fusion vector is first processed by two one-dimensional convolutional networks Conv1d in sequence, and then a maximum pooling function Max Pool is used to convert the 20 vectors into a single vector. Finally, the regression process is completed through three fully connected neural network layers Linear to obtain a three-dimensional vector, which is the normal vector of the target point cloud area. Figure 7As shown, this embodiment takes n as 20 as an example, and the obtained spliced ​​fusion feature vector is [20, 1024+64+256], that is, 20 vectors with a length of 1344. Through a convolution kernel and a one-dimensional convolution network with a moving step of 1, the feature of [20, 1344] is transformed to [20, 512] size. Further, through another convolution kernel and a one-dimensional convolution network with a moving step of 1, the feature of [20, 512] is transformed to [20, 1024]. Then, the feature vector after maximum pooling is transformed into 512 dimensions through a fully connected neural network, and then further transformed into 256 dimensions and 3 dimensions through two fully connected neural networks, and finally the normal vector of the target point cloud area is obtained.

[0061] In an embodiment of the present application, in the process of estimating the point cloud normal vector, after each one-dimensional convolutional network and fully connected neural network layer (except the last fully connected neural network layer) of the data is processed, a batch normalization (Batch Normalization) and a linear rectification function (ReLU) operation will be performed. By using batch normalization, the training can be stabilized and the speed can be improved. Applying the linear rectification function after batch normalization can enhance the nonlinear ability of the model.

[0062] The embodiment of the present application provides a method for estimating the normal vector of a point cloud. By associating each point in a point cloud area in the point cloud data with the image corresponding to the point cloud data, the image feature information of each coordinate point in the point cloud area can be obtained, and then the coordinate information of each coordinate point in the point cloud area is fused with its corresponding image features, and the normal vector of the point cloud area can be further determined based on the fusion result. When the normal vectors of all point cloud areas in the point cloud data are determined, the estimated result of the normal vector of the entire point cloud data is obtained. Compared with the prior art for estimating the normal vector of a point cloud, this solution can effectively improve the accuracy of the normal vector estimation of the point cloud data by fitting a local area in the point cloud data into a plane and then using the normal vector of the fitted plane as the normal vector of the local area of ​​the point cloud.

[0063] like Figure 8 As shown, in another embodiment of the present application, the normal vector estimation network model is trained using training data, and the training data is obtained based on the following method:

[0064] Step S602, obtaining the number and distribution area of ​​sample projection points of each coordinate point in the sample point cloud data projected on the camera imaging plane.

[0065] In the embodiment of the present application, the sample point cloud data is the point cloud data used to prepare the training sample data. The specific method for obtaining the number and distribution area of ​​the sample projection points of each coordinate point in the sample point cloud data projected on the camera imaging plane in the present embodiment is not limited. For example, each coordinate point in the sample point cloud data can be projected on the camera imaging plane to obtain a projection map of the point cloud, such as Fig. 9 As shown, the number of sample projection points in the projection image and the distribution area of ​​the sample projection points in the projection image are then obtained, wherein the distribution area of ​​the sample projection points can be determined by the coordinates of the leftmost projection point, the coordinates of the rightmost projection point, the coordinates of the topmost projection point, and the coordinates of the bottommost projection point in the projection image.

[0066] Step S604: determining a virtual point cloud projection map according to the number and distribution area of ​​the sample projection points, wherein the virtual point cloud projection map is a two-dimensional map in which the sample projection points are evenly distributed.

[0067] In the embodiment of the present application, the virtual point cloud projection diagram is as follows: Fig.10 As shown, the distribution area of ​​the projection points in the virtual point cloud projection map can be determined according to the distribution area of ​​the sample projection points in the camera imaging plane. For example, the vertical coordinate of the uppermost projection point in the sample projection point distribution area can be taken as the upper boundary, so that it and the left and right boundaries and the lower boundary of the virtual point cloud projection map together constitute a regular distribution area of ​​the projection points in the virtual point cloud projection map. Then, according to the number of sample projection points, the number of projection points is evenly distributed in the regular distribution area of ​​the projection points in the virtual point cloud projection map, thereby obtaining the virtual point cloud projection map.

[0068] Step S606, determining the depth value of each sample projection point in the virtual point cloud projection image.

[0069] In the embodiment of the present application, the depth value refers to the distance value from the image acquisition device to each point in the actual scene. This embodiment does not limit the specific method for determining the depth value of each point in the virtual point cloud projection image. For example, the depth map of the image corresponding to the sample point cloud data can be obtained, and then the depth value corresponding to each projection point in the virtual point cloud projection image can be taken out from the depth map, wherein the depth map of the image corresponding to the sample point cloud data can be directly acquired by the image acquisition device.

[0070] Step S608, projecting each of the sample projection points in the virtual point cloud projection map to a spatial coordinate system according to the depth value, determining the three-dimensional coordinates of each of the sample projection points in the virtual point cloud projection map, and using the obtained three-dimensional coordinates corresponding to each of the sample projection points as the training data.

[0071] In the embodiment of the present application, determining the depth value of each point in the virtual point cloud projection map is equivalent to determining the coordinate value of each point in one direction in the spatial coordinate system. Then, by projecting each point in the virtual point cloud projection map to the spatial coordinate system, it is equivalent to determining the coordinate value of each point in the other two directions in the spatial coordinate system. Thus, the three-dimensional coordinates of each point in the virtual point cloud projection map can be determined, that is, the coordinates of the points corresponding to each projection point in the virtual point cloud projection map in the spatial coordinate system are obtained. These three-dimensional coordinates of the points are used for training the point cloud normal vector estimation model. The specific calculation formula for projecting each point in the virtual point cloud projection map to the spatial coordinate system is: 21 (T T K T P 11 ) T , where K is the intrinsic parameter matrix of the camera, T is the transformation matrix from the radar coordinate system to the camera coordinate system, and P 21 is the coordinate of the projection point corresponding to the spatial coordinate system, P 11 is the coordinate of the projection point in the camera coordinate system.

[0072] An embodiment of the present application provides a method for estimating a point cloud normal vector, wherein training data for a normal vector estimation model for determining a first point cloud normal vector is generated through simulation, thereby making use of a large amount of existing simulation three-dimensional modeling in the unmanned driving and robotics community, from which high-quality training data can be obtained without manual labeling, thereby effectively improving the training efficiency of the normal vector estimation model, and thereby improving the processing efficiency of point cloud data.

[0073] Corresponding to the method for estimating the point cloud normal vector in the above embodiment, Fig.11 A structural block diagram of a point cloud normal vector estimation device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown. Fig.11 The example point cloud normal vector estimation device may be an execution subject of the point cloud normal vector estimation method provided in the above embodiment. The point cloud normal vector estimation device may be integrated into the above-mentioned computer device 110.

[0074] Reference Fig.11 The point cloud normal vector estimation device includes: an acquisition module 710, an association module 720 and a normal vector determination module 730.

[0075] The acquisition module 710 is used to acquire the point cloud data collected by the first device and the color image corresponding to the point cloud data collected by the second device.

[0076] In the embodiment of the present application, point cloud data refers to point cloud data collected by a point cloud collection device, which is a collection of multiple three-dimensional coordinate points. Figure 3As shown, it is a point cloud image of a scene collected by the point cloud acquisition device. The elliptical area in the upper right corner of the figure is a local enlarged image of the corresponding position in the figure. All the coordinate points collected in the scene constitute the point cloud data of the scene. The color image corresponding to the point cloud data is the color image of the scene corresponding to the point cloud data collected by the image acquisition device. Each coordinate point in the point cloud data collected by the point cloud acquisition device can correspond to the spatial coordinate information of each object in the color image at the coordinate point.

[0077] The association module 720 is used to associate each coordinate point in the target point cloud area in the point cloud data with the color image to determine the image feature information corresponding to each coordinate point in the target point cloud area; the target point cloud area is any point cloud area in the point cloud data.

[0078] In the embodiment of the present application, the target point cloud region can be any point cloud region in the point cloud data, and the point cloud region is a local region including a plurality of coordinate points in the coordinate point set of the point cloud data. By dividing the point cloud data into a plurality of point cloud regions, the normal vector of each coordinate point in the point cloud data can be calculated, and then the normal vector of each point cloud region can be estimated. In the point cloud normal vector estimation method, the point cloud region whose normal vector is calculated is the target point cloud region.

[0079] In the embodiment of the present application, the feature information of the image can also be referred to as the semantic information of the image. Both the image feature information and the image semantic information are information that can be used to represent the color features, texture features, or shape features of the image. For example, both the image feature information and the image semantic information can include the size information, pixel information, etc. of the image, but are not limited thereto. The extraction of image feature information is the extraction of image semantic information. The present embodiment does not limit the method for extracting the semantic information of the image. For example, it can be obtained through network models such as U-Net (U Neural Network) or Res-Net (Residual Neural Network) or VGG (Visual Geometry Group). The present embodiment does not limit the specific method for associating each coordinate point in the target point cloud area with the color image corresponding to the point cloud data to determine the image feature information corresponding to each coordinate point in the target point cloud area.

[0080] The normal vector determination module 730 is used to fuse the coordinate information of each coordinate point in the target point cloud area with its corresponding image feature information, and determine the normal vector of the target point cloud area according to the fusion result.

[0081] In the embodiment of the present application, the coordinate information of each coordinate point in the target point cloud region is the three-dimensional coordinate information of each coordinate point in the spatial coordinate system, and the three-dimensional coordinate information can be directly obtained from the point cloud data collected by the point cloud data collection device. This embodiment does not limit the specific fusion method for fusing the coordinate information of each coordinate point in the target point cloud region with its corresponding image feature information, and the method for determining the normal vector of the target point cloud region based on the fusion result.

[0082] The embodiment of the present application provides a device for estimating the normal vector of a point cloud. By setting an acquisition module 710, an association module 720 and a normal vector determination module 730, the image feature information of each coordinate point in the point cloud area can be obtained by associating each point in a point cloud area in the point cloud data with the image corresponding to the point cloud data, and then the coordinate information of each coordinate point in the point cloud area is fused with the corresponding image features, and the normal vector of the point cloud area can be further determined according to the fusion result. When the normal vectors of all point cloud areas in the point cloud data are determined, the estimation result of the normal vector of the entire point cloud data is obtained. Compared with the prior art for estimating the normal vector of a point cloud, this scheme can effectively improve the accuracy of estimating the normal vector of the point cloud data by fitting a local area in the point cloud data into a plane and then using the normal vector of the fitting plane as the normal vector of the local area of ​​the point cloud. It should be understood that the size of the sequence number of each step in the above embodiment does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0083] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0084] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0085] It should also be understood that the drawings in this application specification Figure 9-10 The point cloud reference schematic diagram and the projection diagram of the point cloud in the camera plane coordinate system are only reference schematic diagrams to help understand the present application solution.

[0086] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0087] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish descriptions, and cannot be understood as indicating or suggesting relative importance. It should also be understood that although the terms "first", "second", etc. are used to describe various elements in some embodiments of the present application in the text, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, the first table can be named as the second table, and similarly, the second table can be named as the first table without departing from the scope of the various described embodiments. The first table and the second table are both tables, but they are not the same table.

[0088] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0089] Fig.12 Schematic diagram of the structure of a computer device provided by an embodiment of the present application. Fig.12 As shown, the computer device 7 of this embodiment includes: at least one processor 70 ( Fig.12 Only one is shown in the figure), a memory 71, wherein the memory 71 stores a computer program 72 that can be run on the processor 70. When the processor 70 executes the computer program 72, the steps in the above-mentioned method for estimating the normal vector of the point cloud are implemented, for example Figure 2 Alternatively, when the processor 70 executes the computer program 72, the functions of each module / unit in the above-mentioned point cloud data processing device embodiments are realized, for example Fig.11 Functions of modules 710 to 730 are shown.

[0090] The computer device 7 may be a computing device such as a desktop computer, a notebook, a PDA, or a cloud server. The computer device may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will appreciate that Fig.12 It is only an example of computer device 7 and does not constitute a limitation of computer device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include an input sending device, a network access device, a bus, etc.

[0091] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0092] In some embodiments, the memory 71 may be an internal storage unit of the terminal device 7, such as a hard disk or memory of the terminal device 7. The memory 71 may also be an external storage device of the terminal device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the terminal device 7. Further, the memory 71 may also include both an internal storage unit and an external storage device of the terminal device 7. The memory 71 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory 71 may also be used to temporarily store data that has been sent or is to be sent.

[0093] In addition, each functional module in each embodiment of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of software functional unit.

[0094] An embodiment of the present application also provides a computer device, which includes at least one memory, at least one processor, and a computer program stored in the at least one memory and executable on the at least one processor. When the processor executes the computer program, the terminal device implements the steps in the above-mentioned embodiment of the method for estimating any point cloud normal vector.

[0095] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0096] An embodiment of the present application provides a computer program product. When the computer program product is run on a computer device, the computer device can implement the steps in the above-mentioned point cloud normal vector estimation method embodiments when executing the computer device.

[0097] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned point cloud normal vector estimation method embodiments. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium.

[0098] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0099] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for estimating a point cloud normal vector, characterized in that: The method for estimating the point cloud normal vector includes: Acquire point cloud data collected by the first device and a color image corresponding to the point cloud data collected by the second device; Associating each coordinate point in a target point cloud region in the point cloud data with the color image to determine image feature information corresponding to each coordinate point in the target point cloud region; the target point cloud region is any point cloud region in the point cloud data; Fusing the coordinate information of each coordinate point in the target point cloud area with its corresponding image feature information, and determining the normal vector of the target point cloud area according to the fusion result; Fusing the coordinate information of each coordinate point in the target point cloud area with the corresponding image feature information, and determining the normal vector of the target point cloud area according to the fusion result, including: splicing the image feature information of each coordinate point in the target point cloud area with the coordinate information of the coordinate point to obtain a first splicing vector corresponding to each coordinate point; Performing a mean pooling operation on the first splicing vectors corresponding to all coordinate points in the target point cloud area to obtain a second splicing vector corresponding to the target point cloud area; splicing the first splicing vector of each coordinate point in the target point cloud region with the second splicing vector, or splicing the image feature information of each coordinate point in the target point cloud region with the second splicing vector, or splicing the coordinate information of each coordinate point in the target point cloud region with the second splicing vector, to determine a splicing fusion vector; The normal vector of the target point cloud area is determined according to the splicing fusion vector.

2. A method for estimating a point cloud normal vector as claimed in claim 1, characterized in that: The step of associating each coordinate point in the target point cloud area in the point cloud data with the color image to determine image feature information corresponding to each coordinate point in the target point cloud area includes: Projecting each coordinate point in the target point cloud region onto the camera imaging plane where the color image is located, to obtain a projection point corresponding to each coordinate point in the target point cloud region; The image feature information corresponding to the position of the projection point in the color image is extracted to obtain the image feature information of each coordinate point in the target point cloud area.

3. A method for estimating a point cloud normal vector as claimed in claim 1, characterized in that: Determining the normal vector of the target point cloud area according to the splicing fusion vector includes: According to the spliced ​​fusion vector, the normal vector of the target point cloud area is determined through a normal vector estimation network model.

4. A method for estimating a point cloud normal vector as claimed in claim 3, characterized in that: The normal vector estimation network model is trained using training data, and the training data is obtained based on the following method: Obtain the number and distribution area of ​​sample projection points of each coordinate point in the sample point cloud data projected on the camera imaging plane; Determine a virtual point cloud projection map according to the number and distribution area of ​​the sample projection points, wherein the virtual point cloud projection map is a two-dimensional map in which the sample projection points are evenly distributed; Determine the depth value of each sample projection point in the virtual point cloud projection image; According to the depth value, each sample projection point in the virtual point cloud projection map is projected into a spatial coordinate system, the three-dimensional coordinates of each sample projection point in the virtual point cloud projection map are determined, and the obtained three-dimensional coordinates corresponding to each sample projection point are used as the training data.

5. A method for estimating a point cloud normal vector as claimed in claim 1, characterized in that: Before associating each coordinate point in the target point cloud region in the point cloud data with the color image, the point cloud normal vector estimation method further includes: generating a plurality of point cloud regions from the point cloud data; The point cloud data is used to generate a number of point cloud regions, including: Each coordinate point in the point cloud data is taken as a center point, and coordinate points within a spherical range are searched according to a specified radius. Each center point and coordinate points within the spherical range thereof constitute a point cloud region.

6. A method for estimating a point cloud normal vector as claimed in claim 5, characterized in that: The number of coordinate points contained in each point cloud area in the point cloud data is a specified value; Generating a plurality of point cloud regions from the point cloud data also includes: Compare the number of coordinate points searched within the spherical range with the specified value; When the number of coordinate points searched within the spherical range is less than the specified value, points consistent with the coordinates of the center point are added to the point cloud area so that the number of coordinate points contained in the point cloud area reaches the specified value.

7. A point cloud normal vector estimation device, characterized in that: The point cloud normal vector estimation device comprises: An acquisition module, used to acquire point cloud data collected by the first device and a color image corresponding to the point cloud data collected by the second device; an association module, used to associate each coordinate point in a target point cloud region in the point cloud data with the color image, and determine image feature information corresponding to each coordinate point in the target point cloud region; the target point cloud region is any point cloud region in the point cloud data; A normal vector determination module, used to fuse the coordinate information of each coordinate point in the target point cloud area with its corresponding image feature information, and determine the normal vector of the target point cloud area according to the fusion result; Fusing the coordinate information of each coordinate point in the target point cloud area with the corresponding image feature information, and determining the normal vector of the target point cloud area according to the fusion result, including: splicing the image feature information of each coordinate point in the target point cloud area with the coordinate information of the coordinate point to obtain a first splicing vector corresponding to each coordinate point; Performing a mean pooling operation on the first splicing vectors corresponding to all coordinate points in the target point cloud area to obtain a second splicing vector corresponding to the target point cloud area; splicing the first splicing vector of each coordinate point in the target point cloud region with the second splicing vector, or splicing the image feature information of each coordinate point in the target point cloud region with the second splicing vector, or splicing the coordinate information of each coordinate point in the target point cloud region with the second splicing vector, to determine a splicing fusion vector; The normal vector of the target point cloud area is determined according to the splicing fusion vector.

8. A computer device, characterized in that: The computer device includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the steps of the method for estimating the point cloud normal vector according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for estimating a point cloud normal vector according to any one of claims 1 to 6 are implemented.

Images