A method and system for extracting panoramic fusion elements of point cloud

Abstract

An embodiment of the present invention provides a method and system for extracting point cloud panoramic fusion elements. By using data information such as images, point clouds, and poses obtained by hardware devices, combined with point cloud processing algorithms and point cloud image registration algorithms, it is possible to achieve fully automatic extraction of point cloud elements and realize an integrated extraction method. Since the targets in the panoramic image have different degrees of deformation, in order to facilitate subsequent segmentation processing, the panoramic image is divided into frames. According to the mapping relationship between the framed images and the panoramic image, and then using the registration relationship between the panoramic image and the point cloud data, the mapping relationship between the framed images and the point cloud data can be obtained. By generating a depth map matrix from the point cloud data and using the image coordinates of the framed image mask, the corresponding point cloud data can be obtained, reducing the difficulty of manually collecting point cloud elements and improving the efficiency.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of high-precision maps, and particularly to a method and system for extracting point cloud panoramic fusion elements. Background Art

[0002] With the continuous development of science and technology, the development of smart cities is inseparable from the diversification and intelligence of information. As an indispensable part of the construction of smart cities, the construction of three-dimensional high-precision maps is becoming increasingly important. Traditional topographic surveys mostly use total stations and GNSS RTKs for measurement. Traditional measurement methods have low upfront investment and are easy to operate, but require a large number of personnel and a long operation cycle in the later stage. In today's high personnel costs, this undoubtedly increases the production cost. Different from traditional surveys, three-dimensional laser measurement technology is a new concept of surveying and mapping technology. The data measured by the vehicle-mounted three-dimensional laser scanning system is a high-precision point cloud model, with high measurement efficiency, intuitive results, clear identification of ground objects, convenient element extraction and urban component extraction, saving the investment in time cost and labor cost for the construction of smart cities, and having an important economic effect on social production.

[0003] In the prior art, there is a method for extracting feature points of ground objects based on a spatial grid. It performs stitching processing on the original point cloud data to unify the coordinate system, then organizes the spatial grid data, sets operator parameters according to the spatial grid characteristics of the ground object features, visually judges and analyzes the extraction results. If the complete point cloud of the ground object to be extracted is obtained, it passes. If the extraction result is not obvious, the relevant operator parameters are appropriately adjusted and then the extraction and analysis are carried out again. The disadvantages of this method are high spatial complexity, slow speed, the need for certain experience accumulation in operator design, low accuracy in extracting areas with similar morphological features of ground objects, and low automation, requiring manual assistance. There is also a high-precision traffic element target extraction method based on image-point cloud fusion in the prior art. It uses the features of images and point cloud data for registration, assigns attributes to the corresponding point cloud data according to the registration result to obtain fused data, uses each independent traffic element in the image to train the image based on deep learning to obtain target detection models for each classification, uses each detection model to detect each traffic element in the image, saves the point cloud data corresponding to the identification signal, uses the Kalman filter to predict the current position of the traffic elements in the image, then associates the targets of the detection boxes through the Hungarian algorithm, removes duplicates of the same target for multiple front and back trajectory points, retains the target image with the largest field of view, and finally performs deep learning semantic segmentation on the above-mentioned identified and detected target images, maps the segmentation result of the image to the point cloud data, and extracts the traffic element targets. The disadvantages of this method are as follows: 1. Large workload. Training various target detection models for various targets, this method has a large workload and low accuracy. 2. Complicated steps. The semantic segmentation based on deep learning is actually an extension of target detection. This method first uses a deep learning-based target detection network to obtain the rectangular frame position of the target image, and then performs deep learning-based semantic segmentation on this image, with complicated steps. 3. Low accuracy in removing duplicates of target elements. The method uses a two-dimensional image to remove duplicates of the same target for multiple front and back trajectory points, uses the Kalman filter to predict the current position of the traffic elements in the image, then associates the targets of the detection boxes through the Hungarian algorithm, and retains the target image with the largest field of view. There are errors between the predicted current position and the associated detection box. 4. Low accuracy in extracting target elements. This method directly maps the image after semantic segmentation back to the point cloud as the target element. Because the targets after semantic segmentation may be incomplete, redundant, etc., if no secondary processing is performed, the accuracy is low. Summary of the Invention

[0004] An embodiment of the present invention provides a method and system for extracting panoramic fusion elements of point clouds to solve the problems of complicated steps, low efficiency, and high labor intensity in the prior art for extracting point cloud elements.

[0005] In a first aspect, an embodiment of the present invention provides a method for extracting panoramic fusion elements of point clouds, including:

[0006] S1. Obtain the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image according to the attitude information and position information of the data acquisition device; wherein, the data acquisition device at least includes a lidar for collecting three-dimensional point cloud data and a panoramic camera for collecting two-dimensional panoramic images;

[0007] S2. Convert the two-dimensional panoramic image into a panoramic sphere in the three-dimensional image, and take the center of the panoramic sphere as the view point, and obtain the tiled images within a specified area based on the central projection principle;

[0008] S3. Extract the preset classification and segmentation target masks in the tiled images;

[0009] S4. Determine the target point cloud data corresponding to the classification and segmentation target masks based on the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image.

[0010] Preferably, it further includes:

[0011] S5. Denoise the target point cloud data based on the statistical filtering method, and perform duplicate removal processing on the denoised target point cloud data.

[0012] Preferably, the S1 specifically includes:

[0013] S11. Obtain the three-dimensional point cloud data of the target area according to the lidar, and obtain the two-dimensional panoramic image of the target area based on the panoramic camera; obtain the attitude information and position information of the lidar and the panoramic camera based on the positioning and attitude system;

[0014] S12. Obtain the registration relationship between the two-dimensional panoramic image and the three-dimensional point cloud data according to the registration method:

[0015] Construct multiple homologous feature pairs according to the homologous feature points of the panoramic image and the corresponding three-dimensional point cloud data;

[0016] According to the attitude information and position information of the panoramic camera at the exposure moment, convert the absolute coordinates of the homologous feature pairs into the vehicle coordinates with the panoramic camera as the origin;

[0017] According to the initial exterior orientation elements of the panoramic camera, convert the vehicle coordinates of the homologous feature pairs into the camera coordinates;

[0018] According to the interior orientation elements of the panoramic camera, obtain the image coordinates of the homologous feature pairs on the two-dimensional panoramic image, and calculate the corresponding residual values;

[0019] According to the principle that the object point, the image point and the center of the panoramic sphere are collinear, solve the exterior orientation elements of the panoramic camera by the least squares method;

[0020] S13. Repeat the above step S12 until the exterior orientation elements of the panoramic camera to be solved meet the preset requirements.

[0021] Preferably, the S2 specifically includes:

[0022] S21. Convert the image coordinates of the two-dimensional panoramic image into a horizontal angle θ and a vertical angle The horizontal angle distribution range is 0° to 360°, and the vertical angle distribution range is -90° to +90°; where:

[0023]

[0024]

[0025] In the above formula, u is the column coordinate and v is the row coordinate;

[0026] S22. In the panoramic coordinate system, set the distance R from the origin of the coordinate system after converting the image coordinates of the two-dimensional panoramic image into the panoramic coordinate system. According to the horizontal angle θ, vertical angle and R of the pixel point, convert the image coordinates of the two-dimensional panoramic image into three-dimensional coordinates in the panoramic coordinate system;

[0027] S23. According to the position of the target sub-image and the required size of the sub-image, set the orientation of the projection plane in the panoramic coordinate system and the distance from the center of the sphere;

[0028] S24. With the center of the sphere as the projection center, project the spherical surface within the required angle range onto the projection plane in the manner of central projection to obtain the sub-image.

[0029] Preferably, the S4 specifically includes:

[0030] S41. Convert the image coordinates in the sub-image into the image coordinates in the two-dimensional panoramic image;

[0031] S411. Convert the two-dimensional panoramic image into a panoramic sphere in the panoramic coordinate system, and the radius is the set distance R when the two-dimensional panoramic image is converted into the sub-image;

[0032] S412. According to the position of the sub-image, set the position of the projection plane, and convert the image coordinates of the classified and segmented target mask into three-dimensional coordinates in the panoramic coordinate system;

[0033] S413. Use the intersection point of the spatial ray formed by the center of the sphere and the three-dimensional coordinates where the classified and segmented target mask is located and the panoramic sphere as the image coordinates of the sub-image in the two-dimensional panoramic image;

[0034] S42. Convert the three-dimensional coordinates of the three-dimensional point cloud data into vehicle coordinates, and according to the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image, convert the vehicle coordinates into coordinates in the panoramic coordinate system;

[0035] Determine a matrix of the same size as the size of the two-dimensional panoramic image, convert the three-dimensional coordinates in the panoramic coordinate system into image coordinates of the two-dimensional panoramic image, and store the distance value of the corresponding point cloud from the center of the panoramic sphere in the matrix as an index to generate a depth map matrix;

[0036] S43. Convert the image coordinates of the classified and segmented target mask in the two-dimensional panoramic image into three-dimensional point cloud data.

[0037] Preferably, in the step S5, the target point cloud data is denoised based on a statistical filtering method, which specifically includes:

[0038] Traverse each point cloud in the three-dimensional point cloud data, and calculate the average distance between each point cloud and its nearest K neighbor point clouds;

[0039] Calculate the mean μ and standard deviation σ of all the average distances. The distance threshold dmax is dmax = μ + α * σ, where α is a pre-obtained proportionality coefficient;

[0040] Traverse the point clouds again, and remove the points whose average distance from the K neighbor point clouds is greater than dmax.

[0041] Preferably, in the step S5, the denoised target point cloud data is de-duplicated, which specifically includes:

[0042] Use the position of the vehicle at the exposure time of the two-dimensional panoramic image as the station; obtain the point cloud targets of the same target in the two-dimensional panoramic images corresponding to the previous station, the current station, and the next station;

[0043] Determine a point cloud cube according to the point cloud target of the target, construct a set H for storing candidate point cloud cubes to be processed, and initialize it to contain all N point cloud cubes; construct a set M for storing the best point cloud cubes, and initialize it to an empty set;

[0044] Sort all the point cloud cubes in the set H by volume, select the point cloud cube m with the largest volume, and move it from the set H to the set M;

[0045] Traverse the point cloud cubes in the set H, and calculate the 3DIou value with m respectively. If it is higher than 0.2, it is considered that the point cloud cube overlaps with m, and this point cloud cube is removed from the set H;

[0046] Repeat the above steps until the set H is empty. The point cloud data in the set M is the de-duplicated target point cloud data.

[0047] In a second aspect, an embodiment of the present invention provides a point cloud panoramic fusion feature extraction system, including:

[0048] A data acquisition module, which obtains the mapping relationship between three-dimensional point cloud data and two-dimensional panoramic images according to the attitude information and position information of a data acquisition device; wherein, the data acquisition device at least includes a lidar for collecting three-dimensional point cloud data and a panoramic camera for collecting two-dimensional panoramic images;

[0049] A feature extraction module, which converts the two-dimensional panoramic image into a panoramic sphere in a three-dimensional image, and uses the center of the panoramic sphere as the viewpoint to obtain a tiled image within a specified area based on the central projection principle;

[0050] Extract a preset classification and segmentation target mask in the tiled image;

[0051] Based on the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image, determine the target point cloud data corresponding to the classification and segmentation target mask.

[0052] In a third aspect, an embodiment of the present invention provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the point cloud panoramic fusion feature extraction method described in the first aspect embodiment of the present invention.

[0053] In a fourth aspect, an embodiment of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the point cloud panoramic fusion feature extraction method described in the first aspect embodiment of the present invention.

[0054] The point cloud panoramic fusion feature extraction method and system provided by the embodiments of the present invention utilize data information such as images, point clouds, and poses obtained by hardware devices, and combine point cloud processing algorithms and point cloud image registration algorithms to achieve automatic extraction of point cloud features and an integrated extraction method; the targets in the panoramic image are deformed to varying degrees. For the convenience of subsequent segmentation processing, the panoramic image is tiled; according to the mapping relationship between the tiled image and the panoramic image, and then using the registration relationship between the panoramic image and the point cloud data, the mapping relationship between the tiled image and the point cloud data can be obtained, generating a depth map matrix from the point cloud data, and using the image coordinates of the tiled image mask, the corresponding point cloud data can be obtained, reducing the difficulty of manually collecting point cloud features and improving efficiency. Description of the Drawings

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the accompanying drawings required in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can also be obtained based on these drawings.

[0056] Figure 1 It is a flowchart of the method for extracting point cloud panoramic fusion elements according to an embodiment of the present invention;

[0057] Figure 2 It is a schematic diagram of the registration principle of two-dimensional panoramic images and three-dimensional point cloud data according to an embodiment of the present invention;

[0058] Figure 3 It is a schematic diagram of the central projection principle according to an embodiment of the present invention;

[0059] Figure 4 It is a flowchart of the method for registering two-dimensional panoramic images and three-dimensional point cloud data according to an embodiment of the present invention;

[0060] Figure 5 It is a flowchart of obtaining the mapping relationship between the image and point cloud data after tile processing according to an embodiment of the present invention;

[0061] Figure 6 It is a two-dimensional panoramic image before tiling according to an embodiment of the present invention;

[0062] Figure 7 It is a schematic diagram of converting the image coordinates of a two-dimensional panoramic image into three-dimensional coordinates in a panoramic coordinate system according to an embodiment of the present invention;

[0063] Figure 8 It is a panoramic image after tiling according to an embodiment of the present invention;

[0064] Figure 9 It is a flowchart of semantic segmentation using the DeepLabV3+ network structure according to an embodiment of the present invention;

[0065] Figure 10 It is a flowchart of saving the point cloud data mapped back by the mask with trajectory points according to an embodiment of the present invention;

[0066] Figure 11 It is a schematic diagram of the calculation principle of 3DIou according to an embodiment of the present invention;

[0067] Figure 12 It is a block diagram of the system structure for extracting point cloud panoramic fusion elements according to an embodiment of the present invention;

[0068] Figure 13 It is a processing flowchart of the system for extracting point cloud panoramic fusion elements according to an embodiment of the present invention;

[0069] Figure 14 It is a schematic structural diagram of an electronic device according to an embodiment of the present invention. Detailed implementation manners

[0070] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0071] Referring to "embodiments" herein means that the specific features, structures, or characteristics described in connection with the embodiments may be included in at least one embodiment of the present application. The phrase appears in various places in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein may be combined with other embodiments.

[0072] Figure 1 The present invention provides an extraction method for point cloud panoramic fusion elements according to an embodiment of the present invention, which can be applied to the extraction of point clouds of traffic element targets and includes:

[0073] S1. Obtain the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image according to the attitude information and position information of the data acquisition device; wherein, the data acquisition device at least includes a lidar for collecting three-dimensional point cloud data and a panoramic camera for collecting two-dimensional panoramic images;

[0074] Specifically, a three-dimensional laser scanner (lidar, panoramic camera, GNSS, INS) integrating multiple sensors not only obtains the point cloud data and image data of spatial objects, but also can obtain the mapping relationship between the point cloud and the image through information such as the attitude and position of the sensors. Currently, deep learning technology has excellent recognition effects in the field of images. First, use deep learning technology to obtain the target of interest in the image, and then obtain the point cloud data of interest according to the mapping relationship between the image and the point cloud.

[0075] As Figure 2 shown in, S1 specifically includes:

[0076] S11. Obtain the three-dimensional point cloud data of the target area according to the lidar, and obtain the two-dimensional panoramic image of the target area based on the panoramic camera; obtain the attitude information and position information of the lidar and the panoramic camera based on the positioning and attitude system;

[0077] Using the hardware settings of 3D laser scanning to obtain relevant data required by the embodiments of the present invention, such as 3D point cloud data, 2D panoramic images, POS and other data information.

[0078] S12. Obtain the registration relationship between the 2D panoramic image and the 3D point cloud data according to the registration method:

[0079] Construct multiple corresponding feature pairs based on the corresponding feature points of the panoramic image and the corresponding 3D point cloud data;

[0080] According to the attitude information and position information at the exposure moment of the panoramic camera, convert the absolute coordinates of the corresponding feature pairs into vehicle coordinates with the panoramic camera as the origin;

[0081] According to the initial exterior orientation elements of the panoramic camera, convert the vehicle coordinates of the corresponding feature pairs into camera coordinates;

[0082] According to the interior orientation elements of the panoramic camera, obtain the image coordinates of the corresponding feature pairs on the 2D panoramic image and calculate the corresponding residual values;

[0083] According to the principle that the object point, image point and the center of the panoramic sphere are collinear, solve the exterior orientation elements of the panoramic camera by the least squares method;

[0084] Specifically, as Figure 3 shown in, in the figure, the image coordinates of the 2D panoramic image are (u, v), the 3D coordinates of the center of the panoramic sphere are (x0, y0, z0), the 3D coordinates of the object point are (X, Y, Z), the intersection coordinates of the light ray and the panoramic sphere are (x, y, z), the position of the panoramic camera is (X c , Y c , Z c ), the object point, image point and the center of the panoramic sphere are collinear. From the image point on the single spherical panoramic image, the following collinearity equation can be obtained:

[0085]

[0086] In the above formula:

[0087]

[0088]

[0089] Expanding the above formula gives:

[0090]

[0091] In the above formula, a1, a2, a3, b1, b2, b3, c1, c2, c3 are the relevant coefficients in the rotation and translation matrix R respectively; Let:

[0092]

[0093] Then there is:

[0094]

[0095] Let:

[0096]

[0097] Then the error equation is:

[0098]

[0099] After linearization and expansion according to the Taylor series, we get:

[0100]

[0101] Then the error equation is:

[0102] V = AH - L

[0103] Where:

[0104] V = [v α v β T

[0105]

[0106]

[0107] L = [α - (α) β - (β)] T

[0108] Two equations can be listed for one control point. At least three known points are required to solve six unknowns. After calculating these six parameters, a mapping relationship is established between the three-dimensional point cloud data and the two-dimensional panoramic image coordinates.

[0109] S13. Repeat the above step S12 until the exterior orientation elements of the panoramic camera obtained by the solution meet the preset requirements.

[0110] S2. Convert the two-dimensional panoramic image into the panoramic sphere in the three-dimensional image. Taking the center of the panoramic sphere as the viewpoint, obtain the mosaicked images within the specified area based on the central projection principle;

[0111] Convert the panoramic image into the panoramic sphere in the three-dimensional image. Taking the center of the sphere as the viewpoint, the mosaicked images within the specified range can be obtained using the central projection principle. The principle is shown Figure 4 as follows.

[0112] ​Due to the unconventional deformation of the targets in the two-dimensional panoramic image, which is not conducive to the subsequent processing of related images, a panoramic image sub-framing processing algorithm is used to obtain sub-framed images within the ranges of 0-45° to the left, front, and right of the station centered on the station. At the same time, according to the registration relationship between the two-dimensional panoramic image and the three-dimensional point cloud data, the mapping relationship between the sub-framed image and the point cloud data is obtained.

[0113] As Figure 5 shown, the S2 specifically includes:

[0114] S21. Convert the image coordinates of the two-dimensional panoramic image into the horizontal angle θ and the vertical angle The horizontal angle distribution range is 0° to 360°, and the vertical angle distribution range is -90° to +90; where:

[0115]

[0116]

[0117] In the above formula, u is the column coordinate, v is the row coordinate, and h is the size height of the panoramic image; the two-dimensional panoramic image before sub-framing is as Figure 6 shown.

[0118] S22. In the panoramic coordinate system, set the distance R from the origin of the coordinate after converting the image coordinates of the two-dimensional panoramic image into the panoramic coordinate system. According to the horizontal angle θ, the vertical angle of the pixel point and R, convert the image coordinates of the two-dimensional panoramic image into three-dimensional coordinates in the panoramic coordinate system; the conversion formula is:

[0119]

[0120]

[0121]

[0122] The conversion of the two-dimensional panoramic image coordinates into three-dimensional coordinates in the panoramic coordinate system is as Figure 7 shown.

[0123] S23. According to the position of the target sub-framed image and the required size of the sub-framed image, set the orientation of the projection plane in the panoramic coordinate system and the distance from the center of the sphere.

[0124] S24. With the center of the sphere as the projection center, project the spherical surface of the required angular range onto the projection plane in the manner of central projection to obtain the sub-framed image. The panoramic image after sub-framing is as Figure 8 shown.

[0125] S3. Extract the preset classification and segmentation target mask in the sub-framed image;

[0126] Specifically, for the processed tiled images, a well-trained deep learning semantic segmentation model is used for semantic segmentation processing to obtain the mask image coordinates of various targets to be extracted. The DeepLabV3+ network structure is used in the present invention, and the overall process schematic diagram is as shown in Figure 9 shown below.

[0127] S4. Based on the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image, determine the target point cloud data corresponding to the classified and segmented target mask.

[0128] In step S3, the input end of the model is the image to be processed, and the output end of the model is the processed mask image. The mask image coordinates are mapped back to the point cloud;

[0129] Since the panoramic image and the point cloud are registered, and the relationship between the tiled image and the panoramic image is also known, the mask image coordinates can be mapped back to the point cloud to obtain the corresponding point cloud data of various targets to be extracted.

[0130] The specific steps of S4 include:

[0131] S41. Convert the image coordinates in the tiled image to the image coordinates in the two-dimensional panoramic image;

[0132] S411. Convert the two-dimensional panoramic image into a panoramic sphere in the panoramic coordinate system, with a radius of the set distance R when the two-dimensional panoramic image is converted into a tiled image;

[0133] S412. According to the position of the tiled image, set the position of the projection plane (the 0th tiled image corresponds to 0° - 45°, the 1st tiled image corresponds to 45° - 135°, and the 2nd tiled image corresponds to 135° - 225°), and convert the image coordinates of the classified and segmented target mask into three-dimensional coordinates in the panoramic coordinate system so that its position on the projection plane is consistent with that on the two-dimensional image;

[0134] S413. Take the intersection point of the space ray formed by the center of the sphere and the three-dimensional coordinates where the classified and segmented target mask is located and the panoramic sphere as the image coordinates of the tiled image in the two-dimensional panoramic image; the center of the sphere and the three-dimensional coordinates where the mask is located can form a space ray, and the intersection point of the space ray and the panoramic sphere is obtained in turn. The three-dimensional coordinates of the intersection point are converted into the image coordinates of the panoramic image, which are the image coordinates of the tiled image in the panoramic image.

[0135] S42. Convert the three-dimensional coordinates of the three-dimensional point cloud data into vehicle coordinates, and according to the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image, convert the vehicle coordinates into coordinates in the panoramic coordinate system;

[0136] Determine a matrix of the same size as the size of the two-dimensional panoramic image, convert the three-dimensional coordinates in the panoramic coordinate system into the image coordinates of the two-dimensional panoramic image, and store the distance value of the corresponding point cloud from the center of the panoramic sphere into the matrix as an index to generate a depth map matrix;

[0137] S43. Convert the image coordinates of the classified and segmented target mask in the two-dimensional panoramic image into three-dimensional point cloud data.

[0138] Since the panoramic image and the point cloud have been registered, the positional relationship between the panoramic image and the depth map matrix is consistent at this time. According to the traversed image coordinates of the mask, the corresponding distance value d of the depth map matrix is taken out. First, convert the image coordinates to the horizontal angle (θ) and the vertical angle Then convert the horizontal angle and the vertical angle into three-dimensional coordinates to obtain the point cloud data corresponding to the mask. The formula used is as follows:

[0139]

[0140]

[0141]

[0142]

[0143]

[0144] S5. Denoise the target point cloud data based on the statistical filtering method, and remove duplicates from the denoised target point cloud data.

[0145] Specifically, in S5, denoising the target point cloud data based on the statistical filtering method specifically includes:

[0146] Traverse each point cloud in the three-dimensional point cloud data, and calculate the average distance between each point cloud and its K nearest neighbor point clouds;

[0147] Calculate the mean μ and standard deviation σ of all average distances. The distance threshold dmax is dmax = μ + α * σ, where α is a pre-obtained proportionality coefficient;

[0148] Traverse the point cloud again and remove the points whose average distance from the K nearest neighbor point clouds is greater than dmax.

[0149] Specifically, in S5, removing duplicates from the denoised target point cloud data specifically includes:

[0150] Use the position where the carrier is located at the exposure time of the two-dimensional panoramic image as the station; obtain the point cloud targets of the same target in the two-dimensional panoramic images corresponding to the previous station, the current station, and the next station;

[0151] Determine a point cloud cube according to the point cloud target of the object, construct a set H for storing candidate point cloud cubes to be processed, and initialize it to contain all N point cloud cubes; calculate the 3DIou value of two intersecting point cloud targets, as Figure 11 shown in, where the 3DIou value calculation formula is as follows:

[0152] 3DIou = V abcdefgh / (V M + V N - V abcdefgh )

[0153] In the formula, V represents the volume of the corresponding cube;

[0154] a. Construct a set M for storing the best point cloud cubes, and initialize it to an empty set;

[0155] b. Sort all the point cloud cubes in the set H by volume, select the point cloud cube m with the largest volume, and move it from the set H to the set M;

[0156] c. Traverse the point cloud cubes in the set H, calculate the 3DIou value with m respectively. If it is higher than 0.2, it is considered that the point cloud cube overlaps with m, and this point cloud cube is removed from the set H;

[0157] d. Repeat the above steps b and c until the set H is empty. The point cloud data in the set M is the target point cloud data after duplicate removal.

[0158] An embodiment of the present invention also provides a point cloud panoramic fusion element extraction system, based on the point cloud panoramic fusion element extraction method in the above embodiments, as Figure 12 、 Figure 13 shown in, including:

[0159] A data acquisition module, according to the attitude information and position information of the data acquisition device, obtains the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image; wherein, the data acquisition device at least includes a lidar for collecting three-dimensional point cloud data and a panoramic camera for collecting two-dimensional panoramic images;

[0160] An element extraction module, converts the two-dimensional panoramic image into a panoramic sphere in the three-dimensional image, uses the center of the panoramic sphere as the viewpoint, and obtains the tiled images in the specified area based on the central projection principle;

[0161] Extract the preset classification and segmentation target mask in the tiled images;

[0162] Based on the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image, determine the target point cloud data corresponding to the classification and segmentation target mask.

[0163] Based on the same concept, an embodiment of the present invention also provides a schematic structural diagram of an electronic device, as shown in Figure 14 As shown, the server may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840. Among them, the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logical instructions in the memory 830 to execute the steps of the point cloud panoramic fusion element extraction method described in the above embodiments. For example, it includes:

[0164] S1. Obtain the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image according to the attitude information and position information of the data acquisition device; wherein, the data acquisition device at least includes a lidar for collecting three-dimensional point cloud data and a panoramic camera for collecting two-dimensional panoramic images;

[0165] S2. Convert the two-dimensional panoramic image into a panoramic sphere in the three-dimensional image, and take the center of the panoramic sphere as the viewpoint, and obtain the tiled images in the specified area based on the central projection principle;

[0166] S3. Extract the preset classification and segmentation target mask in the tiled images;

[0167] S4. Determine the target point cloud data corresponding to the classification and segmentation target mask based on the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image;

[0168] S5. Denoise the target point cloud data based on the statistical filtering method, and perform duplicate removal processing on the denoised target point cloud data.

[0169] In addition, when the logical instructions in the above-mentioned memory 830 are implemented in the form of software functional units and sold or used as an independent product, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, may be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. And the aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0170] Based on the same concept, an embodiment of the present invention further provides a non-transitory computer-readable storage medium. The computer-readable storage medium stores a computer program, and the computer program includes at least one segment of code. The at least one segment of code can be executed by a main control device to control the main control device to implement the steps of the point cloud panoramic fusion element extraction method described in the above embodiments. For example, it includes:

[0171] S1. According to the attitude information and position information of the data acquisition device, obtain the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image; wherein, the data acquisition device at least includes a lidar for collecting three-dimensional point cloud data and a panoramic camera for collecting two-dimensional panoramic images;

[0172] S2. Convert the two-dimensional panoramic image into a panoramic sphere in the three-dimensional image, and use the center of the panoramic sphere as the viewing point to obtain the tiled images within a specified area based on the central projection principle;

[0173] S3. Extract the preset classification and segmentation target masks in the tiled images;

[0174] S4. Based on the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image, determine the target point cloud data corresponding to the classification and segmentation target masks;

[0175] S5. Perform denoising processing on the target point cloud data based on a statistical filtering method, and perform duplicate removal processing on the denoised target point cloud data.

[0176] Based on the same technical concept, an embodiment of the present application further provides a computer program. When the computer program is executed by a main control device, it is used to implement the above method embodiments.

[0177] The program can be stored in whole or in part on a storage medium packaged together with the processor, or can be stored in whole or in part on a memory not packaged together with the processor.

[0178] Based on the same technical concept, an embodiment of the present application further provides a processor. The processor is used to implement the above method embodiments. The above processor can be a chip.

[0179] In summary, a method and system for extracting point cloud panoramic fusion elements provided by the embodiments of the present invention utilize data information such as images, point clouds, and poses obtained by hardware devices, and combine point cloud processing algorithms and point cloud image registration algorithms to achieve fully automatic extraction of point cloud elements and an integrated extraction method; the targets in the panoramic image have different degrees of deformation. In order to facilitate subsequent segmentation processing, the panoramic image is divided into frames; according to the mapping relationship between the framed image and the panoramic image, and then using the registration relationship between the panoramic image and the point cloud data, the mapping relationship between the framed image and the point cloud data can be obtained. The point cloud data is used to generate a depth map matrix, and by using the image coordinates of the framed image mask, the corresponding point cloud data can be obtained, reducing the difficulty of manually collecting point cloud elements and improving efficiency.

[0180] The various embodiments of the present invention can be combined arbitrarily to achieve different technical effects.

[0181] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by wire (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that the computer can access or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid-state drive SolidStateDisk), etc.

[0182] Those of ordinary skill in the art can understand all or part of the processes in the methods of the above embodiments. These processes can be completed by instructing relevant hardware with a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The foregoing storage medium includes: various media such as ROM or random access memory RAM, magnetic disk, or optical disk that can store program codes.

[0183] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for extracting point cloud panoramic fusion elements, characterized in that, Including: S1. Obtain the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image according to the attitude information and position information of the data acquisition device; wherein, the data acquisition device at least includes a lidar for collecting three-dimensional point cloud data and a panoramic camera for collecting two-dimensional panoramic images; S2. Convert the two-dimensional panoramic image into a panoramic sphere in the three-dimensional image, and use the center of the panoramic sphere as the viewpoint to obtain the tiled images within the specified area based on the central projection principle; S3. Extract the preset classification and segmentation target masks in the tiled images; S4. Determine the target point cloud data corresponding to the classification and segmentation target masks based on the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image; The specific steps of S2 include: S21. Convert the image coordinates of the two-dimensional panoramic image into a horizontal angle θ and a vertical angle ; the horizontal angle distribution range is 0° to 360°, and the vertical angle distribution range is -90° to +90°; where: In the above formula, u is the column coordinate, v is the row coordinate, h is the size height of the panoramic image; S22. In the panoramic coordinate system, after setting the image coordinates of the two-dimensional panoramic image to be converted into the panoramic coordinate system, the distance R from the coordinate origin, according to the horizontal angle θ and vertical angle of the pixel point, as well as R, convert the image coordinates of the two-dimensional panoramic image into three-dimensional coordinates in the panoramic coordinate system; S23. Set the orientation of the projection plane in the panoramic coordinate system and the distance from the center of the sphere according to the position of the target tiled image and the required size of the tiled image; S24. Use the center of the sphere as the projection center and project the spherical surface within the required angular range onto the projection plane in the manner of central projection to obtain the tiled images; The specific steps of S4 include: S41. Convert the image coordinates in the tiled images into the image coordinates in the two-dimensional panoramic image; S411. Convert the two-dimensional panoramic image into a panoramic sphere in the panoramic coordinate system, and the radius is the set distance R when the two-dimensional panoramic image is converted into the tiled images; S412. Set the position of the projection plane according to the position of the tiled image, and convert the image coordinates of the classification and segmentation target masks into three-dimensional coordinates in the panoramic coordinate system; S413. Take the intersection point of the spatial ray formed by the center of the sphere and the three-dimensional coordinates where the classification and segmentation target masks are located with the panoramic sphere as the image coordinates of the tiled images in the two-dimensional panoramic image; S42. Convert the three-dimensional coordinates of the three-dimensional point cloud data into the carrier coordinates, and convert the carrier coordinates into the coordinates in the panoramic coordinate system according to the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image; Determine a matrix of the same size as the size of the two-dimensional panoramic image, convert the three-dimensional coordinates in the panoramic coordinate system into the image coordinates of the two-dimensional panoramic image, and store the distance value of the corresponding point cloud from the center of the panoramic sphere in the matrix as an index to generate a depth map matrix; S43. Convert the image coordinates of the classification and segmentation target masks in the two-dimensional panoramic image into the three-dimensional point cloud data.

2. The method for extracting point cloud panoramic fusion elements according to claim 1, wherein It further includes: S5. Denoise the target point cloud data based on the statistical filtering method, and perform duplicate removal processing on the denoised target point cloud data.

3. The method for extracting point cloud panoramic fusion elements according to claim 1, wherein The specific steps of S1 include: S11. Obtain the three-dimensional point cloud data of the target area according to the lidar, and obtain the two-dimensional panoramic image of the target area based on the panoramic camera; obtain the attitude information and position information of the lidar and the panoramic camera based on the positioning and attitude system; S12. Obtain the registration relationship between the two-dimensional panoramic image and the three-dimensional point cloud data according to the registration method: Construct multiple corresponding feature pairs according to the corresponding feature points of the panoramic image and the corresponding three-dimensional point cloud data; Convert the absolute coordinates of the corresponding feature pairs into the carrier coordinates with the panoramic camera as the origin according to the attitude information and position information at the exposure moment of the panoramic camera; Convert the vehicle coordinates of the homologous feature pairs into camera coordinates according to the initial exterior orientation elements of the panoramic camera; Obtain the image coordinates of the homologous feature pairs on the two-dimensional panoramic image according to the interior orientation elements of the panoramic camera, and calculate the corresponding residual values; Solve the exterior orientation elements of the panoramic camera by the least squares method according to the principle that the object point, the image point and the center of the panoramic sphere are collinear; S13. Repeat the above step S12 until the exterior orientation elements of the solved panoramic camera meet the preset requirements.

4. The method for extracting point cloud panoramic fusion elements according to claim 2, characterized in that, In the above S5, denoise the target point cloud data based on the statistical filtering method, which specifically includes: Traverse each point cloud in the three-dimensional point cloud data, and calculate the average distance between each point cloud and its nearest K neighbor point clouds; Calculate the mean μ and standard deviation σ of all average distances, and the distance threshold dmax is dmax = μ + α * σ, where α is a pre-obtained proportionality coefficient; Traverse the point clouds again, and remove the points whose average distance from the K neighbor point clouds is greater than dmax.

5. The method for extracting point cloud panoramic fusion elements according to claim 2, wherein In the above S5, and perform duplicate removal processing on the denoised target point cloud data, which specifically includes: Use the position of the vehicle at the exposure moment of the two-dimensional panoramic image as the station; obtain the point cloud targets of the same target in the two-dimensional panoramic images corresponding to the previous station, the current station and the next station; Determine the point cloud cube according to the point cloud target of the target, construct a set H for storing candidate point cloud cubes to be processed, and initialize it to contain all N point cloud cubes; construct a set M for storing the best point cloud cubes, and initialize it to an empty set; Sort all the point cloud cubes in the set H by volume, select the point cloud cube m with the largest volume, and move it from the set H to the set M; Traverse the point cloud cubes in the set H, and calculate the 3DIou value with m respectively. If it is higher than 0.2, it is considered that the point cloud cube overlaps with m, and remove this point cloud cube from the set H; Repeat the above steps until the set H is empty, and the point cloud data in the set M is the target point cloud data after duplicate removal.

6. A point cloud panoramic fusion feature extraction system for performing a point cloud panoramic fusion feature extraction method as described in claim 1, characterized in that, Including: A data acquisition module, which obtains the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image according to the attitude information and position information of the data acquisition device; wherein, the data acquisition device at least includes a lidar for collecting three-dimensional point cloud data and a panoramic camera for collecting two-dimensional panoramic images; A feature extraction module, which converts the two-dimensional panoramic image into a panoramic sphere in the three-dimensional image, uses the center of the panoramic sphere as the viewpoint, and obtains the tiled images in the specified area based on the central projection principle; Extract the preset classification and segmentation target masks in the tiled images; Based on the mapping relationship between the three-dimensional point cloud data and the two-dimensional panoramic image, determine the target point cloud data corresponding to the classification and segmentation target masks.

7. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the point cloud panoramic fusion feature extraction method according to any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the point cloud panoramic fusion feature extraction method according to any one of claims 1 to 5.

Images