Point cloud data fusion method and device, computer readable storage medium and vehicle

Abstract

The invention belongs to the technical field of intelligent driving, and particularly relates to a point cloud data fusion method and device, a computer readable storage medium and a vehicle. The method comprises the following steps: acquiring first point cloud data in a global map to be fused and second point cloud data in a local map; converting the second point cloud data to obtain third point cloud data; searching matching points corresponding to the points in the first point cloud data in the third point cloud data; taking points of which the corresponding matching points are not searched in the first point cloud data as fourth point cloud data; and performing fusion processing on the third point cloud data and the fourth point cloud data to obtain fused point cloud data. Through the method, point cloud data fusion can be carried out according to the simple search matching result between the point cloud data, updating of the point cloud data is realized, and compared with the prior art, the calculation amount is effectively reduced, and the updating efficiency of the point cloud data is improved.

Description

Technical Field

[0001] The present application belongs to the field of intelligent driving technology, and in particular, relates to a point cloud data fusion method, device, computer-readable storage medium and vehicle. Background Art

[0002] With the development of intelligent driving technology, various types of vehicles can automatically drive in different operating scenarios to perform operating tasks. For example, vehicles can automatically drive to mining areas to perform mining tasks. During the operation of the vehicle, the operating environment will gradually change. In order to operate accurately, it is necessary to regularly update the point cloud data of the global map of the operating environment.

[0003] In the prior art, a point cloud data fusion method of a local map and a global map is often used to update the global map. However, the calculation process of the existing point cloud data fusion method is relatively complicated, resulting in low efficiency in updating the point cloud data. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a point cloud data fusion method, device, computer-readable storage medium and vehicle to solve the problem that the existing point cloud data fusion method has a complex calculation process, resulting in low efficiency in updating point cloud data.

[0005] A first aspect of an embodiment of the present application provides a point cloud data fusion method, which may include:

[0006] Acquire first point cloud data and second point cloud data to be fused; wherein the first point cloud data is point cloud data in a global map, and the second point cloud data is point cloud data in a local map;

[0007] Converting the second point cloud data into the coordinate system of the global map to obtain converted third point cloud data;

[0008] Searching the third point cloud data for matching points corresponding to points in the first point cloud data;

[0009] taking points in the first point cloud data for which no corresponding matching points are found as fourth point cloud data;

[0010] The third point cloud data and the fourth point cloud data are fused to obtain fused point cloud data.

[0011] In a specific implementation of the first aspect, searching the third point cloud data for a matching point corresponding to a point in the first point cloud data may include:

[0012] Performing planarization processing on the third point cloud data to obtain planarized point cloud data;

[0013] Performing scattered point edge extraction on the planarized point cloud data to obtain boundaries and holes corresponding to the planarized point cloud data;

[0014] Using the first point cloud data within the boundary as in-bounds point cloud data;

[0015] The planarized point cloud data is searched for matching points corresponding to points in the in-bounds point cloud data.

[0016] In a specific implementation of the first aspect, searching the planarized point cloud data for matching points corresponding to points in the in-bound point cloud data may include:

[0017] Searching for the nearest neighboring point of the in-bounds target point in the planarized point cloud data; wherein the in-bounds target point is any point in the in-bounds point cloud data;

[0018] Calculate the distance between the target point within the boundary and the nearest point;

[0019] If the distance between the in-bounds target point and the nearest point is less than a preset first distance threshold, the nearest point is determined as a matching point corresponding to the in-bounds target point;

[0020] If the distance between the in-bounds target point and the nearest point is greater than or equal to the first distance threshold, it is determined that no matching point corresponding to the in-bounds target point has been found.

[0021] In a specific implementation of the first aspect, after taking the points in the first point cloud data for which no corresponding matching points are searched as the fourth point cloud data, the method may further include:

[0022] Using the fourth point cloud data in the hole as in-hole point cloud data;

[0023] Determining points in the visually visible area in the in-hole point cloud data;

[0024] The points in the visually visible area are removed from the fourth point cloud data.

[0025] In a specific implementation of the first aspect, determining the points in the in-hole point cloud data that are in a visually visible area may include:

[0026] Searching for the nearest point set of the target point in the hole on the hole; wherein the target point in the hole is any point in the point cloud data in the hole;

[0027] Constructing a simulated plane corresponding to the nearest point set;

[0028] Determining whether the target point in the hole is located above the simulation plane;

[0029] If the in-hole target point is located above the simulation plane, it is determined that the in-hole target point is in the visually visible area.

[0030] In a specific implementation of the first aspect, constructing a simulated plane corresponding to the nearest point set may include:

[0031] Select three points from the nearest point set, the distances between any two of which are greater than a preset second distance threshold;

[0032] The simulation plane is constructed according to the three selected points.

[0033] In a specific implementation of the first aspect, after converting the second point cloud data into the coordinate system of the global map to obtain the converted third point cloud data, the method may further include:

[0034] The outlier points in the third point cloud data are removed.

[0035] A second aspect of an embodiment of the present application provides a point cloud data fusion device, which may include:

[0036] A data acquisition module, used to acquire first point cloud data and second point cloud data to be fused; wherein the first point cloud data is point cloud data in a global map, and the second point cloud data is point cloud data in a local map;

[0037] A data conversion module, used for converting the second point cloud data into the coordinate system of the global map to obtain converted third point cloud data;

[0038] A data search module, used to search the third point cloud data for matching points corresponding to points in the first point cloud data;

[0039] a data determination module, configured to use points in the first point cloud data for which no corresponding matching points are searched as fourth point cloud data;

[0040] The data fusion module is used to fuse the third point cloud data and the fourth point cloud data to obtain fused point cloud data.

[0041] In a specific implementation of the second aspect, the data search module may include:

[0042] A planarization processing unit, used for performing planarization processing on the third point cloud data to obtain planarized point cloud data;

[0043] An edge extraction unit, used for performing scattered point edge extraction on the planarized point cloud data to obtain boundaries and holes corresponding to the planarized point cloud data;

[0044] a point cloud data determining unit, configured to use the first point cloud data within the boundary as in-bounds point cloud data;

[0045] A data search unit is used to search the planarized point cloud data for matching points corresponding to points in the in-bound point cloud data.

[0046] In a specific implementation of the second aspect, the data search unit may include:

[0047] A nearest point search subunit is used to search the nearest point of the in-bounds target point in the planarized point cloud data; wherein the in-bounds target point is any point in the in-bounds point cloud data;

[0048] A distance calculation subunit, used for calculating the distance between the target point within the boundary and the nearest point;

[0049] A first determining subunit is used to determine the nearest point as a matching point corresponding to the in-bounds target point if the distance between the in-bounds target point and the nearest point is less than a preset first distance threshold;

[0050] The second determining subunit is used to determine that no matching point corresponding to the in-bounds target point has been searched if the distance between the in-bounds target point and the nearest point is greater than or equal to the first distance threshold.

[0051] In a specific implementation of the second aspect, the point cloud data fusion device may further include:

[0052] A first data determination module, configured to use the fourth point cloud data in the hole as in-hole point cloud data;

[0053] A second data determination module is used to determine points in the visually visible area in the in-hole point cloud data;

[0054] A data elimination module is used to eliminate the points in the visually visible area from the fourth point cloud data.

[0055] In a specific implementation of the second aspect, the second data determination module may include:

[0056] A point set search unit, used for searching the nearest point set of the target point in the hole on the hole; wherein the target point in the hole is any point in the point cloud data in the hole;

[0057] A plane construction unit, used for constructing a simulation plane corresponding to the nearest point set;

[0058] A judging unit, used for judging whether the target point in the hole is located above the simulation plane;

[0059] The area determination unit is used to determine that the in-hole target point is in a visually visible area if the in-hole target point is located above the simulation plane.

[0060] In a specific implementation of the second aspect, the plane construction unit may include:

[0061] A point set selection subunit, used for selecting three points from the nearest point set, the distances between any two of which are greater than a preset second distance threshold;

[0062] The plane construction subunit is used to construct the simulation plane according to the three selected points.

[0063] In a specific implementation of the second aspect, the point cloud data fusion device may further include:

[0064] The data removal module is used to remove outliers in the third point cloud data.

[0065] A third aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the above-mentioned point cloud data fusion methods are implemented.

[0066] A fourth aspect of an embodiment of the present application provides a vehicle, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any one of the above-mentioned point cloud data fusion methods when executing the computer program.

[0067] A fifth aspect of an embodiment of the present application provides a computer program product. When the computer program product is run on a vehicle, the vehicle executes the steps of any one of the above-mentioned point cloud data fusion methods.

[0068] Compared with the prior art, the embodiments of the present application have the following beneficial effects: the embodiments of the present application obtain the first point cloud data in the global map to be fused and the second point cloud data in the local map; convert the second point cloud data to obtain the third point cloud data; search the third point cloud data for matching points corresponding to the points in the first point cloud data; use the points in the first point cloud data for which the corresponding matching points are not found as the fourth point cloud data; fuse the third point cloud data and the fourth point cloud data to obtain fused point cloud data. Through the embodiments of the present application, point cloud data fusion can be performed based on simple search and matching results between point cloud data to achieve point cloud data update, which effectively reduces the amount of calculation and improves the efficiency of point cloud data update compared with the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0070] Figure 1 Schematic diagram of the die casting process of the upper mold;

[0071] Figure 2 is a schematic diagram of the die casting process of the mold below;

[0072] Figure 3 It is a schematic diagram of die casting stop;

[0073] Figure 4 This is a flow chart of an embodiment of a point cloud data fusion method in an embodiment of the present application;

[0074] Figure 5 A schematic flow chart of the matching point search;

[0075] Figure 6 This is a schematic diagram of the planarization process;

[0076] Figure 7 Schematic diagram of the boundaries and holes corresponding to the planarized point cloud data;

[0077] Figure 8 This is a top-down view of the map;

[0078] Fig. 9 This is a front view of the map;

[0079] Fig.10 It is a schematic diagram of the process of determining the point cloud data in the hole;

[0080] Fig.11 A schematic flow chart for determining whether a target point in a hole is within a visually visible area;

[0081] Fig.12 is a schematic diagram of the simulation plane;

[0082] Fig.13 This is a structural diagram of an embodiment of a point cloud data fusion device in an embodiment of the present application;

[0083] Fig.14 A schematic block diagram of a vehicle in an embodiment of the present application. DETAILED DESCRIPTION

[0084] In order to make the purpose, features, and advantages of the invention of this application more obvious and easy to understand, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the embodiments described below are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0085] It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0086] It should also be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0087] It should be further understood that the term “and / or” used in the specification and appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0088] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0089] In addition, in the description of the present application, the terms "first", "second", "third", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0090] With the development of intelligent driving technology, various types of vehicles can automatically drive in different operating scenarios to perform operating tasks. For example, vehicles can automatically drive to mining areas to perform mining tasks. During the operation of the vehicle, the operating environment will gradually change. In order to operate accurately, it is necessary to regularly update the point cloud data of the global map of the operating environment.

[0091] In the prior art, a point cloud data fusion method of a local map and a global map is often used to update the global map. However, the calculation process of the existing point cloud data fusion method is relatively complicated, resulting in low updating efficiency of the point cloud data.

[0092] In view of this, the embodiments of the present application provide a point cloud data fusion method, device, computer-readable storage medium and vehicle to solve the problem that the existing point cloud data fusion method has a complex calculation process, resulting in low efficiency in updating point cloud data. In the embodiments of the present application, point cloud data fusion can be performed based on simple search and matching results between point cloud data to achieve point cloud data update, which effectively reduces the amount of calculation and improves the efficiency of point cloud data update compared to the existing technology.

[0093] It should be noted that the execution subject of the embodiment of the present application can be a vehicle, including but not limited to mining trucks, trucks, tractors, dump trucks and other common industrial vehicles in the prior art. The point cloud data fusion method provided in the embodiment of the present application can be applied to various actual operation scenarios. For example, in a mining operation environment, the global map can be locally updated by the point cloud data fusion method, which is conducive to the precise operation of the operation vehicle.

[0094] In the embodiment of the present application, the local map can be analogized to the same upper mold and lower mold, and the global map can be analogized to the object to be cast. The upper mold is pressed vertically from the position of positive infinity height, such as Figure 1 As shown, the lower mold is lifted vertically from the negative infinite height position, as shown in Figure 2 When the two molds reach the preset height, they stop. Figure 3 As shown, at this time, the die-casting object point in the plane space where each point in the mold is located will be die-cast to the height where the mold point is located. In the embodiment of the present application, no matter whether the point of the global map in the plane is floating or sinking in the local map, it will be die-cast into the shape of the mold (local map), ensuring the integrity of the local map and the neatness of the die-casting area.

[0095] For details, please refer to Figure 4 , an embodiment of a point cloud data fusion method in the embodiment of the present application may include:

[0096] Step S401: Acquire first point cloud data and second point cloud data to be fused.

[0097] In an embodiment of the present application, the first point cloud data and the second point cloud data to be fused can be obtained through a preset laser radar, wherein the first point cloud data is the point cloud data in the global map, which can be recorded as g_map; the second point cloud data is the point cloud data in the local map, which can be recorded as l_map.

[0098] It should be noted that the global map in the embodiment of the present application is usually generated before the local map, and the local map can contain the local map information in the global map, so the point cloud data in the global map can be locally updated through the point cloud data in the local map.

[0099] In a possible embodiment, the first point cloud data may be generated for the first time when the map of the working environment is constructed, and stored in a preset memory module or server, and may be obtained from the memory module or server when point cloud data fusion is required; and the second point cloud data may be generated in real time when point cloud data fusion is required, or may be generated and stored in the memory module or server before point cloud data fusion is performed, and may be obtained from the memory module or server when point cloud data fusion is required.

[0100] It is understandable that in order to obtain point cloud data in a timely manner, a laser radar can be pre-installed on the vehicle. When the latest point cloud data needs to be obtained, the laser radar can be used to detect the working environment and collect point cloud data.

[0101] Step S402: convert the second point cloud data into the coordinate system of the global map to obtain converted third point cloud data.

[0102] After acquiring the second point cloud data, the second point cloud data can be converted into the coordinate system of the global map through the conversion relationship T between the global coordinate system and the local coordinate system to obtain the converted third point cloud data, which can be recorded as l_map_t, where l_map_t=l_map*T.

[0103] Among them, the conversion relationship between the global coordinate system and the local coordinate system can be obtained by combining various navigation devices, or can also be obtained by any SLAM method in the prior art, and the embodiments of the present application do not make specific limitations on this.

[0104] In a possible embodiment, the conversion relationship between the global coordinate system and the local coordinate system can be obtained by combining various navigation devices. For example, the conversion relationship between the global coordinate system and the local coordinate system can be obtained by combining various sensors such as visual sensors, radar sensors, positioning sensors, and inertial measurement units.

[0105] In another possible embodiment, the transformation relationship between the global coordinate system and the local coordinate system can be obtained by any common simultaneous localization and mapping (SLAM) method, including but not limited to Cartographer method, Karto method, LOAM series methods, VINS-Fusion method and other common SLAM methods in the prior art.

[0106] In another possible embodiment, in order to prevent the interference data in the third point cloud data from affecting the fusion of the point cloud data, outlier detection may be performed on the third point cloud data, and the outliers in the third point cloud data may be removed. For example, the StatisticalOutlierRemoval filter may be used to remove outliers in the third point cloud data. For another example, a cluster analysis may be performed on the third point cloud data, and points that do not belong to any class may be regarded as outliers, and the outliers may be removed from the third point cloud data. For another example, the local outlier factor in the point cloud data may be calculated based on the density of the point cloud data, and then the point with the largest degree of outlier may be regarded as an outlier, and the outlier may be removed from the third point cloud data.

[0107] Step S403: Search the third point cloud data for matching points corresponding to points in the first point cloud data.

[0108] In an embodiment of the present application, matching points corresponding to points in the first point cloud data may be searched in the third point cloud data based on the distance between points in the first point cloud data and points in the third point cloud data.

[0109] Specifically, see Figure 5 , step S403 may include the following process:

[0110] Step S4031, planarize the third point cloud data to obtain planarized point cloud data.

[0111] In an embodiment of the present application, in order to speed up the search and matching, the third point cloud data can be planarized, the height of the points in the third point cloud data (ie, the z coordinate) can be ignored, and the points in the third point cloud data can be gathered on the same plane to obtain planarized point cloud data, which can be recorded as l_map_plane.

[0112] Specifically, see Figure 6 , the z coordinate value in the three-dimensional coordinates (x, y, z) of the points in the third point cloud data can be set to 0, and the points in the third point cloud data can be gathered on the plane of z=0 to obtain the planarized point cloud data of the third point cloud data on the two-dimensional plane.

[0113] In the embodiment of the present application, after obtaining the planarized point cloud data, a corresponding kd tree (k-dimensional tree) can also be constructed according to the planarized point cloud data. Here, since the planarized point cloud data is two-dimensional data, the planarized point cloud data can be divided according to one of the dimensions in the planarized point cloud data, and then the point cloud data obtained by the previous division can be divided according to another dimension in the planarized point cloud data, and the replacement and division of the two dimensions are repeated until the point cloud data is divided, and the kd tree corresponding to the planarized point cloud data can be obtained.

[0114] Step S4032: extract scattered point edges from the planarized point cloud data to obtain boundaries and holes corresponding to the planarized point cloud data.

[0115] In an embodiment of the present application, a preset scattered point edge extraction algorithm can be used to perform scattered point edge extraction on the planar point cloud data to obtain boundaries (denoted as l_shape) and holes (denoted as l_holes) corresponding to the planar point cloud data.

[0116] In the embodiment of the present application, a simple and effective Alpha Shapes algorithm can be preferably used to outline the following Figure 7 Boundaries and holes corresponding to the planarized point cloud data shown.

[0117] It is understandable that in addition to the Alpha Shapes algorithm, any scattered point edge extraction algorithm in the prior art may be used, including but not limited to the Gift Wrapping algorithm, the Graham Scan algorithm, and the Quickhull algorithm, etc. This application does not make any specific limitation on this.

[0118] Step S4033: Use the first point cloud data within the boundary as the in-boundary point cloud data.

[0119] In the embodiment of the present application, the first point cloud data within the boundary can be selected and used as the in-boundary point cloud data.

[0120] It can be understood that since the planarized point cloud data is obtained based on the third point cloud data, and the third point cloud data and the first point cloud data are in the same coordinate system, at this time, we can only focus on the two-dimensional coordinates of the first point cloud data, and regard the first point cloud data and the planar point cloud data as point cloud data located in the same plane. Based on this, the first point cloud data within the boundary can be selected.

[0121] Specifically, the value of the z coordinate of the point in the first point cloud data can also be regarded as 0. At this time, the first point cloud data and the plane point cloud data can both be regarded as point cloud data on the z=0 plane. Afterwards, the first point cloud data within the boundary can be selected and the selected first point cloud data can be used as the in-boundary point cloud data.

[0122] Step S4034: Search the planarized point cloud data for matching points corresponding to points in the in-bounds point cloud data.

[0123] Since the point cloud data in the local map only exists within the boundary, in order to reduce unnecessary search and matching processes and improve the efficiency of point cloud data fusion, only the points in the point cloud data within the boundary (that is, the first point cloud data located within the boundary) can be matched and searched, while the first point cloud data outside the boundary can directly determine that the corresponding matching points cannot be searched.

[0124] Specifically, for the in-bounds point cloud data, points that match the points in the in-bounds point cloud data can be searched in the planarized point cloud data. If a matching point pair can be successfully searched, it can be considered that the matching point pair is actually the same point in the two maps.

[0125] For ease of description, step S4034 will be explained below by taking any point in the in-bounds point cloud data (recorded as the in-bounds target point) as an example.

[0126] In an embodiment of the present application, the nearest neighbor point of the in-bounds target point can be searched in the planarized point cloud data. Specifically, a nearest neighbor search can be performed on the constructed kd tree corresponding to the planarized point cloud data to obtain the nearest neighbor point of the in-bounds target point; thereafter, the distance between the in-bounds target point and the nearest neighbor point can be calculated, and then based on the distance between the in-bounds target point and the nearest neighbor point, it can be determined whether the nearest neighbor point matches the in-bounds target point.

[0127] It can be understood that if the distance between the in-bounds target point and the nearest neighbor point is small, it can be considered that the in-bounds target point and the nearest neighbor point are actually the same points in the two maps; if the distance between the in-bounds target point and the nearest neighbor point is large, it can be considered that the in-bounds target point and the nearest neighbor point are different points in the two maps.

[0128] Specifically, in the embodiment of the present application, if the distance between the in-bounds target point and the nearest neighbor point is less than a preset first distance threshold, the nearest neighbor point can be determined as a matching point corresponding to the in-bounds target point; if the distance between the in-bounds target point and the nearest neighbor point is greater than or equal to the first distance threshold, it can be determined that no matching point corresponding to the in-bounds target point has been searched. The value of the first distance threshold can be set according to actual needs, and the embodiment of the present application does not specifically limit this.

[0129] It can be understood that by traversing each point in the in-bound point cloud data and repeating the above steps, the search for matching points corresponding to the points in the in-bound point cloud data can be completed in the planarized point cloud data.

[0130] Step S404: taking points in the first point cloud data for which no corresponding matching points are found as fourth point cloud data.

[0131] In an embodiment of the present application, after determining matching points corresponding to points in the in-bounds point cloud data in the planarized point cloud data, points in the first point cloud data for which corresponding matching points have not been searched are used as fourth point cloud data, and the fourth point cloud data can be recorded as g_map_f.

[0132] Step S405: Fusing the third point cloud data and the fourth point cloud data to obtain fused point cloud data.

[0133] In the embodiment of the present application, after obtaining the third point cloud data and the fourth point cloud data, the third point cloud data and the fourth point cloud data can be fused to obtain fused point cloud data. Specifically, the third point cloud data and the fourth point cloud data can be merged to obtain fused point cloud data.

[0134] It is understandable that after obtaining the fused point cloud data, the latest global map can be obtained based on the fused point cloud data, and can be used for operation planning such as vehicle navigation.

[0135] It is understandable that there may be holes in the map during construction, that is, visual blind spots and sparse point cloud areas. For details, please refer to Figure 8 The map shown is a top view and Fig. 9 The front view of the map shown in the figure, the square box is the visual blind area, and the circle box is the point cloud sparse area. When the global map is constructed, due to the difference in angles, point cloud data in the holes of the local map may be collected. At this time, it can also be determined whether the point cloud data in the holes can be used for point cloud data fusion.

[0136] Therefore, in another embodiment provided by the present application, after the fourth point cloud data is determined in step S404, it is further possible to determine whether the fourth point cloud data located in the hole can be used for point cloud data fusion.

[0137] It can be understood that if a point in the fourth point cloud data in the hole is in the visual blind spot of the local map, the point can be retained; if a point in the fourth point cloud data in the hole is in the visually visible area of ​​the local map, it means that the point cloud data should exist in the local map, but there is no point cloud data in the local map. At this time, the point can be considered as an interference point and removed from the fourth point cloud data.

[0138] Specifically, the fourth point cloud data in the hole can be used as the in-hole point cloud data, and the points in the visually visible area can be determined from the in-hole point cloud data. Then, the points in the visually visible area can be eliminated from the fourth point cloud data.

[0139] It is understandable that, since there is no point cloud data outside the boundary in the local map, in order to reduce unnecessary processing, the point cloud data inside the hole can be determined here only based on the fourth point cloud data inside the boundary. For any point in the fourth point cloud data inside the boundary, it is possible to determine whether the point is inside the hole by using the ray method. Specifically, for any point in the fourth point cloud data inside the boundary, a ray in any direction can be drawn. If the number of intersections between the drawn ray and the hole is not equal to 1, such as Fig.10 (a), then it can be considered that the point is not located in the hole; if the number of intersections between the ray and the hole is equal to 1, such as Fig.10 As shown in (b), it can be considered that the point is located inside the hole and can be used as the point cloud data inside the hole.

[0140] Afterwards, you can refer to Fig.11 The steps shown take any point in the hole point cloud data (recorded as the hole target point) as an example to determine whether the hole target point is in the visually visible area:

[0141] Step S1101, searching the hole for the nearest point set of the target point in the hole.

[0142] In the embodiment of the present application, the kd tree corresponding to the constructed planarized point cloud data can be used to perform a nearest neighbor search to obtain the nearest point set of the target point in the hole.

[0143] It can be understood that the nearest point set of the target point in the hole can include more than three points that are closest to the target point in the hole, and the specific number of points in the nearest point set can be set to any value in [3, N] according to actual conditions, where N is the number of points of the polygon that makes up the hole.

[0144] In this embodiment, three points may be selected from the set of nearest points of the target point in the hole, and a simulation plane may be constructed according to the three selected points, and the simulation plane may be recorded as plane_i.

[0145] It is understandable that if the distance between the two closest points of the simulated plane is too small, it can be considered that the result of visually visible area judgment based on the simulated plane may have a large deviation. Therefore, in the embodiment of the present application, three points whose distances between each other are greater than the preset second distance threshold can be selected from the nearest points of the target point in the hole. Among them, the second distance threshold can be set according to the actual situation, and the embodiment of the present application does not specifically limit this.

[0146] Step S1102: construct a simulated plane corresponding to the nearest point set.

[0147] In this embodiment, a simulation plane can be constructed according to the three selected points.

[0148] Specifically, the equation of the simulated plane can be set to Ax+By+Cz+D=0, and then the three-dimensional coordinates of the three selected points are substituted into the equation to solve the three unknown coefficients A, B, C and the unknown constant D to obtain the equation of the simulated plane.

[0149] Step S1103: According to the simulated plane, determine whether the target point in the hole is located above the simulated plane.

[0150] In this embodiment, it can be determined whether the target point in the hole is located above the simulation plane based on the constructed simulation plane and the target point in the hole.

[0151] Specifically, the three-dimensional coordinates of the target point in the hole can be substituted into the equation of the simulation plane, and the value of the equation can be calculated. If the value of the equation is less than or equal to 0, such as Fig.12 As shown in (a), it can be considered that the target point in the hole is below the simulation plane. At this time, it can be determined that the target point in the hole is not in the visually visible area; if the value of this equation is greater than 0, such as Fig.12 As shown in (b), it can be considered that the target point in the hole is located above the simulation plane, that is, it can be determined that the target point in the hole is in the visually visible area. At this time, the target point in the hole can be removed from the fourth point cloud data.

[0152] It can be understood that by traversing the in-hole point cloud data and repeatedly executing the above process, points (interference points) in the visually visible area can be removed from the fourth point cloud data.

[0153] After removing the interference points in the fourth point cloud data, the third point cloud data and the fourth point cloud data can be fused according to step S405 to obtain fused point cloud data. Since the points in the visually visible area of ​​the local map can be removed from the fourth point cloud data, the influence of the interference points on the point cloud data fusion can be reduced, making the result of the point cloud data fusion more realistic, which is conducive to improving the accuracy of the point cloud data fusion.

[0154] In summary, the embodiment of the present application obtains the first point cloud data in the global map to be fused and the second point cloud data in the local map; converts the second point cloud data to obtain the third point cloud data; searches for matching points corresponding to the points in the first point cloud data in the third point cloud data; uses the points in the first point cloud data for which the corresponding matching points are not found as the fourth point cloud data; and fuses the third point cloud data and the fourth point cloud data to obtain fused point cloud data. Through the embodiment of the present application, point cloud data fusion can be performed based on simple search and matching results between point cloud data to achieve point cloud data update. Compared with the prior art, the amount of calculation is effectively reduced and the efficiency of point cloud data update is improved.

[0155] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0156] Corresponding to the point cloud data fusion method described in the above embodiment, Fig.13 A structural diagram of an embodiment of a point cloud data fusion device provided in an embodiment of the present application is shown.

[0157] In this embodiment, a point cloud data fusion device may include:

[0158] The data acquisition module 1301 is used to acquire first point cloud data and second point cloud data to be fused; wherein the first point cloud data is point cloud data in the global map, and the second point cloud data is point cloud data in the local map;

[0159] A data conversion module 1302 is used to convert the second point cloud data into the coordinate system of the global map to obtain converted third point cloud data;

[0160] A data search module 1303 is used to search the third point cloud data for matching points corresponding to points in the first point cloud data;

[0161] A data determination module 1304 is used to use points in the first point cloud data for which no corresponding matching points are found as fourth point cloud data;

[0162] The data fusion module 1305 is used to fuse the third point cloud data and the fourth point cloud data to obtain fused point cloud data.

[0163] In a specific implementation of the embodiment of the present application, the data search module may include:

[0164] A planarization processing unit, used for performing planarization processing on the third point cloud data to obtain planarized point cloud data;

[0165] An edge extraction unit, used for performing scattered point edge extraction on the planarized point cloud data to obtain boundaries and holes corresponding to the planarized point cloud data;

[0166] a point cloud data determining unit, configured to use the first point cloud data within the boundary as in-bounds point cloud data;

[0167] A data search unit is used to search the planarized point cloud data for matching points corresponding to points in the in-bound point cloud data.

[0168] In a specific implementation of the embodiment of the present application, the data search unit may include:

[0169] A nearest point search subunit is used to search the nearest point of the in-bounds target point in the planarized point cloud data; wherein the in-bounds target point is any point in the in-bounds point cloud data;

[0170] A distance calculation subunit, used for calculating the distance between the target point within the boundary and the nearest point;

[0171] A first determining subunit is used to determine the nearest point as a matching point corresponding to the in-bounds target point if the distance between the in-bounds target point and the nearest point is less than a preset first distance threshold;

[0172] The second determining subunit is used to determine that no matching point corresponding to the in-bounds target point has been searched if the distance between the in-bounds target point and the nearest point is greater than or equal to the first distance threshold.

[0173] In a specific implementation of the embodiment of the present application, the point cloud data fusion device may further include:

[0174] A first data determination module, configured to use the fourth point cloud data in the hole as in-hole point cloud data;

[0175] A second data determination module is used to determine points in the visually visible area in the in-hole point cloud data;

[0176] A data elimination module is used to eliminate the points in the visually visible area from the fourth point cloud data.

[0177] In a specific implementation of the embodiment of the present application, the second data determination module may include:

[0178] A point set search unit, used for searching the nearest point set of the target point in the hole on the hole; wherein the target point in the hole is any point in the point cloud data in the hole;

[0179] A plane construction unit, used for constructing a simulation plane corresponding to the nearest point set;

[0180] A judging unit, used for judging whether the target point in the hole is located above the simulation plane;

[0181] The area determination unit is used to determine that the in-hole target point is in a visually visible area if the in-hole target point is located above the simulation plane.

[0182] In a specific implementation of the embodiment of the present application, the plane construction unit may include:

[0183] A point set selection subunit, used for selecting three points from the nearest point set, the distances between any two of which are greater than a preset second distance threshold;

[0184] The plane construction subunit is used to construct the simulation plane according to the three selected points.

[0185] In a specific implementation of the embodiment of the present application, the point cloud data fusion device may further include:

[0186] The data removal module is used to remove outliers in the third point cloud data.

[0187] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices, modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0188] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0189] Fig.14 A schematic block diagram of a vehicle provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0190] like Fig.14 As shown, the vehicle 14 of this embodiment includes: a processor 140, a memory 141, and a computer program 142 stored in the memory 141 and executable on the processor 140. When the processor 140 executes the computer program 142, the steps in the above-mentioned point cloud data fusion method embodiments are implemented, for example Figure 4 Alternatively, when the processor 140 executes the computer program 142, the functions of each module / unit in the above-mentioned device embodiments are realized, for example Fig.13 The functions of modules 1301 to 1305 are shown.

[0191] Exemplarily, the computer program 142 may be divided into one or more modules / units, which are stored in the memory 141 and executed by the processor 140 to complete the present application. The one or more modules / units may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 142 in the vehicle 14.

[0192] Those skilled in the art will understand that Fig.14 It is only an example of the vehicle 14 and does not constitute a limitation of the vehicle 14. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the vehicle 14 may also include input and output devices, network access devices, buses, etc.

[0193] The processor 140 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0194] The memory 141 may be an internal storage unit of the vehicle 14, such as a hard disk or memory of the vehicle 14. The memory 141 may also be an external storage device of the vehicle 14, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card (FlashCard), etc. equipped on the vehicle 14. Further, the memory 141 may also include both an internal storage unit and an external storage device of the vehicle 14. The memory 141 is used to store the computer program and other programs and data required by the vehicle 14. The memory 141 may also be used to temporarily store data that has been output or is to be output.

[0195] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0196] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0197] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0198] In the embodiments provided in the present application, it should be understood that the disclosed devices / vehicles and methods can be implemented in other ways. For example, the device / vehicle embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0199] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0200] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0201] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable storage medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable storage media do not include electric carrier signals and telecommunication signals.

[0202] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A point cloud data fusion method, characterized in that: include: Acquire first point cloud data and second point cloud data to be fused; wherein the first point cloud data is point cloud data in a global map, and the second point cloud data is point cloud data in a local map; Converting the second point cloud data into the coordinate system of the global map to obtain converted third point cloud data; Searching the third point cloud data for matching points corresponding to points in the first point cloud data; taking points in the first point cloud data for which no corresponding matching points are found as fourth point cloud data; The third point cloud data and the fourth point cloud data are fused to obtain fused point cloud data.

2. The point cloud data fusion method according to claim 1, characterized in that: The searching the third point cloud data for matching points corresponding to points in the first point cloud data comprises: Performing planarization processing on the third point cloud data to obtain planarized point cloud data; Performing scattered point edge extraction on the planarized point cloud data to obtain boundaries and holes corresponding to the planarized point cloud data; Using the first point cloud data within the boundary as in-bounds point cloud data; The planarized point cloud data is searched for matching points corresponding to points in the in-bounds point cloud data.

3. The point cloud data fusion method according to claim 2, characterized in that: The step of searching the planarized point cloud data for matching points corresponding to points in the in-bound point cloud data comprises: Searching for the nearest neighboring point of the in-bounds target point in the planarized point cloud data; wherein the in-bounds target point is any point in the in-bounds point cloud data; Calculate the distance between the target point within the boundary and the nearest point; If the distance between the in-bounds target point and the nearest point is less than a preset first distance threshold, the nearest point is determined as a matching point corresponding to the in-bounds target point; If the distance between the in-bounds target point and the nearest point is greater than or equal to the first distance threshold, it is determined that no matching point corresponding to the in-bounds target point has been found.

4. The point cloud data fusion method according to claim 2, characterized in that: After taking the points in the first point cloud data for which no corresponding matching points are searched as the fourth point cloud data, the method further includes: Using the fourth point cloud data in the hole as in-hole point cloud data; Determining points in the visually visible area in the in-hole point cloud data; The points in the visually visible area are removed from the fourth point cloud data.

5. The point cloud data fusion method according to claim 4, characterized in that: Determining points in the visually visible area in the in-hole point cloud data includes: Searching for the nearest point set of the target point in the hole on the hole; wherein the target point in the hole is any point in the point cloud data in the hole; Constructing a simulated plane corresponding to the nearest point set; Determining whether the target point in the hole is located above the simulation plane; If the in-hole target point is located above the simulation plane, it is determined that the in-hole target point is in the visually visible area.

6. The point cloud data fusion method according to claim 5, characterized in that: The constructing a simulated plane corresponding to the nearest point set includes: Select three points from the nearest point set, the distances between any two of which are greater than a preset second distance threshold; The simulation plane is constructed according to the three selected points.

7. The point cloud data fusion method according to any one of claims 1 to 6, characterized in that: After converting the second point cloud data into the coordinate system of the global map to obtain the converted third point cloud data, the method further includes: The outlier points in the third point cloud data are removed.

8. A point cloud data fusion device, characterized in that: include: A data acquisition module, used to acquire first point cloud data and second point cloud data to be fused; wherein the first point cloud data is point cloud data in a global map, and the second point cloud data is point cloud data in a local map; A data conversion module, used for converting the second point cloud data into the coordinate system of the global map to obtain converted third point cloud data; A data search module, used to search the third point cloud data for matching points corresponding to points in the first point cloud data; A data determination module, configured to use points in the first point cloud data for which no corresponding matching points are searched as fourth point cloud data; The data fusion module is used to fuse the third point cloud data and the fourth point cloud data to obtain fused point cloud data.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the point cloud data fusion method according to any one of claims 1 to 7 are implemented.

10. A vehicle comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the point cloud data fusion method according to any one of claims 1 to 7 are implemented.

Images