Point cloud map construction method and device, excavator and computer readable storage medium

Abstract

The invention relates to a point cloud map construction method and device, an excavator and a computer readable storage medium, and relates to the technical field of engineering machinery. The point cloud map construction method comprises the following steps: acquiring point cloud data of a first radar, a second radar and a third radar of an excavator; combining the point cloud data of the first radar, the second radar and the third radar; removing the point cloud data corresponding to the boom system from the merged point cloud data to obtain material point cloud data; and constructing a point cloud map according to the material point cloud data.

Description

Technical Field

[0001] The present disclosure relates to the technical field of engineering machinery, and in particular to a point cloud map construction method and device, an excavator, and a computer-readable storage medium. Background Art

[0002] Excavators are important engineering machinery, widely used in construction, mining and other engineering construction industries. However, excavators often face harsh environmental conditions and high-intensity operating requirements during operation, which not only easily leads to safety hazards, but also makes it difficult and costly to recruit operators, thus limiting the further development of the industry. Therefore, the research and development of excavators with unmanned operation functions has huge industry demand and practical significance.

[0003] As one of the key technologies to realize unmanned operation of excavators, point cloud mapping is mainly used to build a three-dimensional point cloud map of the real scene. By building a point cloud map, unmanned excavators can carry out more accurate task planning, including excavation and loading operations, thereby improving work efficiency and safety. Summary of the invention

[0004] According to some embodiments of the present disclosure, a point cloud map construction method is provided, comprising:

[0005] Obtaining point cloud data of a first radar, a second radar, and a third radar of the excavator, wherein the first radar and the second radar are located on the top of the cab of the excavator, the third radar is located in an adjacent area of ​​the cab, an angle is formed between an installation direction of the first radar and an installation direction of the second radar, an angle between an installation direction of the first radar and an operating direction of the excavator is less than 90 degrees, an angle between an installation direction of the second radar and an operating direction of the excavator is less than 90 degrees, and an angle between an installation direction of the third radar and an operating direction of the excavator is less than 90 degrees;

[0006] Merging the point cloud data of the first radar, the second radar and the third radar;

[0007] From the merged point cloud data, remove the point cloud data corresponding to the boom system to obtain the material point cloud data;

[0008] Build a point cloud map based on the material point cloud data.

[0009] In some embodiments, the merging of the point cloud data of the first radar, the second radar, and the third radar includes:

[0010] converting the point cloud data of the first radar, the second radar and the third radar into the same coordinate system;

[0011] In the same coordinate system, the point cloud data of the first radar, the second radar and the third radar are merged.

[0012] In some embodiments, converting the point cloud data of the first radar, the second radar, and the third radar into the same coordinate system includes:

[0013] The point cloud data of the second radar and the point cloud data of the third radar are converted into the coordinate system of the first radar by using the first parameter.

[0014] In some embodiments, the point cloud map construction method further includes:

[0015] Before converting the point cloud data of the second radar and the point cloud data of the third radar into the coordinate system of the first radar using the first parameter, displaying the point cloud data of the first radar, the point cloud data of the second radar, and the point cloud data of the third radar in the same view;

[0016] In the view, the first parameter is determined by aligning the point cloud of the bucket in the point cloud data of the first radar, the second radar, and the third radar.

[0017] In some embodiments, removing the point cloud data corresponding to the boom system from the merged point cloud data to obtain the material point cloud data includes:

[0018] Using the second parameter, the merged point cloud data is converted from the coordinate system of the first radar to the coordinate system of the vehicle;

[0019] In the vehicle coordinate system, the point cloud data corresponding to the boom system is removed from the merged point cloud data to obtain the material point cloud data.

[0020] In some embodiments, constructing a point cloud map based on material point cloud data includes:

[0021] Get turntable attitude sensor data;

[0022] Time synchronization of material point cloud data and turntable attitude sensor data;

[0023] A point cloud map is constructed based on the time-synchronized material point cloud data and turntable attitude sensor data.

[0024] In some embodiments, constructing a point cloud map based on the time-synchronized material point cloud data and turntable attitude sensor data includes:

[0025] Acquire point cloud data of a fourth radar and a fifth radar of the excavator, wherein the fourth radar and the fifth radar are located on both sides of the excavator, and an angle between an installation direction of the fourth radar and an operating direction of the excavator is greater than an angle between an installation direction of any one of the first radar, the second radar, and the third radar and the operating direction of the excavator, and an angle between an installation direction of the fifth radar and the operating direction of the excavator is greater than an angle between an installation direction of any one of the first radar, the second radar, and the third radar and the operating direction of the excavator;

[0026] Using the third parameter, the point cloud data of the fourth radar and the point cloud data of the fifth radar are converted into the vehicle coordinate system;

[0027] In the vehicle coordinate system, time-synchronize the point cloud data of the fourth radar and the point cloud data of the fifth radar with the turntable attitude sensor data;

[0028] A point cloud map is constructed based on the time-synchronized material point cloud data, the point cloud data of the fourth radar, the point cloud data of the fifth radar, and the turntable attitude sensor data.

[0029] In some embodiments, the point cloud map construction method further includes:

[0030] Calculate the registration score of the point cloud map,

[0031] The second parameter and / or the third parameter is adjusted according to the registration score of the point cloud map.

[0032] In some embodiments, a point cloud map is constructed based on the time-synchronized material point cloud data, the point cloud data of the fourth radar, the point cloud data of the fifth radar, and the turntable attitude sensor data, including:

[0033] Convert the time-synchronized material point cloud data, the fourth radar point cloud data, and the fifth radar point cloud data from the vehicle-on-board coordinate system to the vehicle-off-board coordinate system;

[0034] Construct a point cloud map based on the point cloud data converted to the get-off coordinate system.

[0035] In some embodiments, the point cloud data corresponding to the boom system is removed from the merged point cloud data to obtain the material point cloud data, including:

[0036] According to the structural parameters of the boom system, the point cloud data corresponding to the boom system is removed from the merged point cloud data to obtain the material point cloud data.

[0037] In some embodiments, the angle between the installation direction of the first radar and the ground is less than 90 degrees, and the angle between the installation direction of the second radar and the ground is greater than 90 degrees; the imaging areas of the first radar, the second radar and the third radar partially overlap.

[0038] The present disclosure provides a point cloud map construction device, comprising:

[0039] an acquisition module, configured to acquire point cloud data of a first radar, a second radar, and a third radar of an excavator, wherein the first radar and the second radar are located on top of a cab of the excavator, the third radar is located in an adjacent area of ​​the cab, an angle is formed between an installation direction of the first radar and an installation direction of the second radar, an angle between an installation direction of the first radar and an operating direction of the excavator is less than 90 degrees, an angle between an installation direction of the second radar and an operating direction of the excavator is less than 90 degrees, and an angle between an installation direction of the third radar and an operating direction of the excavator is less than 90 degrees;

[0040] a merging module configured to merge the point cloud data of the first radar, the second radar and the third radar;

[0041] A removal module is configured to remove the point cloud data corresponding to the boom system from the merged point cloud data to obtain material point cloud data;

[0042] The construction module is configured to construct a point cloud map based on the material point cloud data.

[0043] The present disclosure provides an electronic device, comprising:

[0044] Memory; and

[0045] A processor coupled to the memory, the processor being configured to execute the point cloud map construction method according to some embodiments of the present disclosure based on instructions stored in the memory.

[0046] The present disclosure provides a computer-readable storage medium having computer program instructions stored thereon. When the instructions are executed by a processor, the point cloud map construction method described in some embodiments of the present disclosure is implemented.

[0047] The present disclosure provides a computer program product, including computer program instructions, which, when executed by a processor, implement the point cloud map construction method according to some embodiments of the present disclosure.

[0048] The present disclosure provides an excavator, comprising:

[0049] A point cloud map constructing device according to some embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0051] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0052] Figure 1 A flowchart showing a method for constructing a point cloud map according to some embodiments of the present disclosure is shown;

[0053] Figure 2 A schematic diagram showing a radar installation according to some embodiments of the present disclosure;

[0054] Figure 3 A schematic diagram showing calibration parameters according to some embodiments of the present disclosure;

[0055] Figure 4 A schematic diagram showing time synchronization according to some embodiments of the present disclosure;

[0056] Figure 5 A schematic diagram showing time synchronization according to other embodiments of the present disclosure;

[0057] Figure 6 A schematic diagram showing construction of a point cloud map according to some embodiments of the present disclosure;

[0058] Figure 7 A block diagram showing a point cloud map construction device according to some embodiments of the present disclosure;

[0059] Figure 8 A block diagram showing an electronic device according to some other embodiments of the present disclosure;

[0060] Fig. 9 A block diagram of a computer system for implementing some embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0061] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0062] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0063] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0064] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0065] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0066] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0067] A typical working condition of an unmanned excavator is to perform rotary excavation and loading operations on a relatively fixed platform. In the related art, the boom is not specially processed during the mapping process. The boom will continue to move during the excavation and loading process. If it is not separated from the point cloud, the point cloud data of the boom will be mixed with the point cloud data of the material. This will cause the data in the point cloud map to be impure, and it is difficult to accurately distinguish which point clouds belong to static materials and which belong to dynamic booms.

[0068] Since the boom is dynamic, its point cloud data changes frequently. If the boom data is not filtered out correctly, the system may mistakenly regard the boom as a dynamic object, which will affect the recognition of other real dynamic objects (such as pedestrians, vehicles, etc.) and trigger unnecessary alarms or take incorrect obstacle avoidance measures. At the same time, since the boom's point cloud data is mixed with the material, the system may not be able to accurately identify the position and shape of the material, thus affecting the efficiency of excavation and loading.

[0069] The present disclosure provides a cloud map construction method and device, an excavator, and a computer-readable storage medium, which improve the working efficiency and safety of the unmanned excavator.

[0070] Figure 1 A flowchart of a point cloud map construction method according to some embodiments of the present disclosure is shown.

[0071] like Figure 1 As shown, the point cloud map construction method includes steps S1 to S4. In some embodiments, the point cloud map construction method is executed by a domain controller.

[0072] In step S1, point cloud data of a first radar, a second radar and a third radar of an excavator are obtained, wherein the first radar and the second radar are located on the top of a cab of the excavator, the third radar is located in an adjacent area of ​​the cab, there is an angle between the installation direction of the first radar and the installation direction of the second radar, the angle between the installation direction of the second radar and the operating direction of the excavator is less than 90 degrees, the angle between the installation direction of the third radar and the operating direction of the excavator is less than 90 degrees, and the angle between the installation direction of the first radar and the operating direction of the excavator is less than 90 degrees; in step S2, the point cloud data of the first radar, the second radar and the third radar are merged; in step S3, the point cloud data corresponding to the boom system is removed from the merged point cloud data to obtain material point cloud data; in step S4, a point cloud map is constructed according to the material point cloud data.

[0073] The radar used in some embodiments of the present disclosure is, for example, a petal-scanning laser radar.

[0074] The operating direction is, for example, the front of the cab, that is, the front direction of the vehicle. The installation direction of the radar refers to the direction of the radar antenna relative to the carrier (or fixed platform).

[0075] The arm system includes, for example, a boom and a dipper rod. The point cloud of the bucket can be removed by using a method similar to that of removing the point cloud of the arm system.

[0076] The three forward-looking radars (first radar 1, second radar 2 and third radar 3) are all installed at the front of the excavator (front of the vehicle). Although the angles are not exactly the same, they are all facing forward and have a certain common view area. These radars can simultaneously observe the boom system and / or bucket at a specific position. The point clouds collected by these three radars are merged, and then the point cloud of the boom system is identified and extracted, so that the point cloud of the dynamic execution part of the device (boom system and / or bucket) is not included in the final constructed point cloud map.

[0077] Figure 2 A schematic diagram showing a radar installation according to some embodiments of the present disclosure.

[0078] like Figure 2 As shown, the first radar 1 and the second radar 2 are both installed on the top of the cab 6, that is, the first radar 1 and the second radar 2 are installed at the front position on the cab side of the excavator, and both are facing forward. Because the top of the cab is at a high position, the first radar 1 and the second radar 2 are set at a high position to significantly increase their visual range, thereby expanding the coverage of the radar, allowing the radar to detect targets at a longer distance, and can reduce the shielding of the radar signal by the ground terrain.

[0079] The adjacent area of ​​the cab is, for example, the boom system 7 and the area connected to the excavator body. Taking the operating direction of the excavator as the front, if the cab 6 is on the left, the right side of the cab 6 is the adjacent area of ​​the cab. In other words, the third radar 3 is located at the front of the right side of the excavator (i.e., the other side relative to the cab). In addition, the installation direction of the third radar 3 is also forward.

[0080] The turntable attitude sensor 9 is installed at the turntable of the upper vehicle, and is used to measure the rotation angle, rotation angular velocity, pitch angle and roll angle of the turntable relative to the lower vehicle. Among them, the rotation angle of the turntable relative to the lower vehicle refers to the rotation angle of the upper structure of the excavator (i.e., the turntable) relative to the lower walking mechanism (i.e., the lower vehicle). The turntable attitude sensor data is the Euler angle (rpy), that is, the roll angle (roll), pitch angle (pitch), rotation angle (yaw), and rotation speed (yaw rate).

[0081] The domain controller 8 is a computing device for data collection and computational processing. The radar data is a point cloud, and the radar data is linked to the domain controller via the User Datagram Protocol (UDP) protocol link. The turntable attitude sensor data is linked to the domain controller via the Controller Area Network (CAN) protocol. The domain controller 8 can simultaneously receive and process multiple radar point cloud data (e.g., including frame start time, frame length, frame point cloud) of low frequency (e.g., 10 Hz) and data (e.g., time, rotation angle, pitch angle, roll angle, and rotation speed) of the turntable attitude sensor 9 of high frequency (e.g., 100 Hz). High-frequency turntable attitude sensor data refers to the attitude information collected and output by the sensor at a higher frequency (e.g., 100 Hz). The frame start time refers to the timestamp when a frame of point cloud data starts to be collected. The frame length refers to the duration of a frame of point cloud data, that is, the time interval from the frame start time to the end of the frame. Frame point cloud refers to all point cloud data collected within a certain frame time.

[0082] In some embodiments, the angle between the installation direction of the first radar and the ground is less than 90 degrees, and the angle between the installation direction of the second radar and the ground is greater than 90 degrees; the imaging areas of the first radar, the second radar, and the third radar partially overlap. For example, the installation directions of the first radar 1 and the second radar 2 deviate from the horizontal direction to a certain extent. Although both are roughly forward, in comparison, the first radar 1 scans obliquely downward, and the second radar 2 scans obliquely upward.

[0083] The main working area of ​​the excavator is usually in the front, including tasks such as digging, loading and moving materials. The first radar 1, the second radar 2 and the third radar 3 are forward-looking radars that work together to scan forward from the left and right positions respectively, providing information on the terrain and material conditions in front of the excavator.

[0084] The three forward-looking radars (first radar 1, second radar 2 and third radar 3) have a certain common viewing area and can simultaneously observe the bucket or boom at a specific position. By merging the forward-looking radars and extracting the boom, the moving boom point cloud can be efficiently removed, and an effective construction area point cloud map can be established, which improves the working efficiency and safety of the unmanned excavator.

[0085] In some embodiments, merging the point cloud data of the first radar, the second radar, and the third radar includes: converting the point cloud data of the first radar, the second radar, and the third radar to the same coordinate system; and merging the point cloud data of the first radar, the second radar, and the third radar in the same coordinate system.

[0086] For example, spatial synchronization is first performed, and the point cloud data of different radars are converted to the same coordinate system through parameter transformation, so that in the same coordinate system, the point cloud data of different radars are spatially aligned.

[0087] Furthermore, the point cloud data of the first radar, the second radar, and the third radar can be aligned in time according to their respective timestamps. The point cloud data of the first radar, the second radar, and the third radar can also be directly synchronized with the turntable attitude sensor, and then the point cloud data of the boom system can be extracted. The order of time synchronization, space synchronization, and extraction of the boom point cloud can be adjusted according to actual needs.

[0088] In some embodiments, converting the point cloud data of the first radar, the second radar, and the third radar into the same coordinate system includes: using a first parameter to convert the point cloud data of the second radar and the point cloud data of the third radar into the coordinate system of the first radar.

[0089] In the excavator construction environment, the side wall materials sometimes only appear in a certain direction. Due to the oblique upward angle of the second radar, when the excavator rotates to other directions, the point cloud of the second radar will sometimes be empty. In contrast, the first radar is not only installed at a high position, but also installed at an oblique downward angle, so that it can usually collect a more comprehensive point cloud of the construction area during the excavation operation of the excavator. Therefore, the first radar is determined as the main radar. The remaining forward-looking radars are converted from their respective coordinate systems to the coordinate system of the main radar for spatial synchronization. The second radar (front upper left radar) is recorded as Lflu, and the third radar (front lower right radar) is recorded as Lfrd.

[0090] The first parameter converted from the coordinate system of the second radar to the coordinate system of the first radar is recorded as TLflu2Lfld, and the first parameter converted from the coordinate system of the third radar to the coordinate system of the first radar is recorded as TLfrd2Lfld.

[0091] Because the side wall materials only appear in specific directions, the forward-looking radar point cloud data will not increase the point cloud data volume too much after merging.

[0092] In some embodiments, the point cloud map construction method also includes: before using the first parameter to convert the point cloud data of the second radar and the point cloud data of the third radar into the coordinate system of the first radar, displaying the point cloud data of the first radar, the point cloud data of the second radar, and the point cloud data of the third radar in the same view; in the view, determining the first parameter by aligning the point cloud of the bucket in the point cloud data of the first radar, the second radar, and the third radar.

[0093] For example, before converting the point cloud data of the second radar and the point cloud data of the third radar into the coordinate system of the first radar, parameters for the conversion are first determined.

[0094] Figure 3 A schematic diagram showing calibration parameters according to some embodiments of the present disclosure.

[0095] like Figure 3 As shown, before synchronizing to the coordinate system of the main lidar, the point cloud data of the remaining two forward-looking radars (the second radar and the third radar) are displayed in the same view as the point cloud data of the first radar. Because it has not been synchronized yet, the buckets sensed by the three forward-looking radars appear in different positions. At this time, you can select the second radar, and then drag the adjustment slider to adjust the parameters of x, y, z, roll angle (roll), pitch angle (pitch), and yaw angle (yaw) to align the bucket point cloud of the second radar with the bucket point cloud of the main radar. Similarly, align the bucket point cloud of the third radar with the bucket point cloud of the main radar. Through the above alignment method, manual calibration of the external parameters of the common view radar is achieved. The bucket point cloud can also be replaced by similar point clouds of other relatively stable objects. By aligning the point clouds of relatively stable objects and performing parameter calibration, the accuracy of the point cloud map can be improved.

[0096] In some embodiments, removing the point cloud data corresponding to the boom system from the merged point cloud data to obtain material point cloud data includes: using a second parameter to convert the merged point cloud data from the coordinate system of the first radar to the vehicle coordinate system; in the vehicle coordinate system, removing the point cloud data corresponding to the boom system from the merged point cloud data to obtain material point cloud data.

[0097] For example, the spatially aligned forward-looking radar point cloud is converted from the coordinate system of the first radar to the vehicle coordinate system through the second parameter (TLfd2up). By converting the point clouds of several forward-looking radars to the vehicle coordinate system, the point cloud data of the forward-looking radar and the boom are in the same coordinate system, thereby realizing the extraction of the boom point cloud.

[0098] In some embodiments, constructing a point cloud map based on material point cloud data includes: acquiring turntable attitude sensor data; time synchronizing the material point cloud data with the turntable attitude sensor data; and constructing a point cloud map based on the time-synchronized material point cloud data and turntable attitude sensor data.

[0099] The frequency of the material point cloud data can be adjusted according to the frequency of the turntable attitude sensor data. In addition, the forward-looking radar point cloud can be spatially aligned first, and then time-aligned with the turntable attitude sensor data; or the forward-looking radar point cloud can be time-aligned with the turntable attitude sensor data first, and then the radar can be spatially aligned.

[0100] Figure 4 A schematic diagram showing time synchronization according to some embodiments of the present disclosure.

[0101] If the frequency of the on-board attitude sensor is higher than the radar frequency, each radar frame is split to form high-frequency short-frame data. For example, each frame of radar point cloud data is split into 10 parts, and the radar point cloud data frequency changes from 10hz to 100hz, which is the same as the turntable attitude sensor data frequency. Finally, the split data is pushed into the radar frame queue, and the split frames in the queue are sorted according to the time size (for example, from time max to time min in the figure) to obtain the forward-looking radar frame queue sorted by time. The start time of the three forward-looking radars may be different, so all the short frames formed after the split can be sorted in time and pushed into the radar frame queue. The turntable attitude sensor data is also pushed into the rotation data queue, so as to align the time with the forward-looking radar frame queue of the same frequency.

[0102] In some embodiments, a point cloud map is constructed based on the time-synchronized material point cloud data and turntable attitude sensor data, including: obtaining point cloud data of a fourth radar and a fifth radar of the excavator, wherein the fourth radar and the fifth radar are located on both sides of the excavator, and the angle between the installation direction of the fourth radar and the operating direction of the excavator is greater than the angle between the installation direction of any one of the first radar, the second radar and the third radar and the operating direction of the excavator, and the angle between the installation direction of the fifth radar and the operating direction of the excavator is greater than the angle between the installation direction of any one of the first radar, the second radar and the third radar and the operating direction of the excavator; using a third parameter, converting the point cloud data of the fourth radar and the point cloud data of the fifth radar to a vehicle coordinate system; in the vehicle coordinate system, time-synchronizing the point cloud data of the fourth radar and the point cloud data of the fifth radar with the turntable attitude sensor data; constructing a point cloud map based on the time-synchronized material point cloud data, the point cloud data of the fourth radar, the point cloud data of the fifth radar and the turntable attitude sensor data.

[0103] Please refer back to Figure 2 , the fourth radar 4 and the fifth radar 5 (on the other side of the excavator, represented by a dotted filled square in the figure) are installed on both sides of the excavator, one installation direction is facing left, and the other installation direction is facing right, respectively, for collecting material information on the left and right sides of the excavator. The angle between the fourth radar and the installation direction of the first radar 1 is, for example, close to 90 degrees, for example, the first radar 1 scans forward, and the fourth radar 4 scans to the right. Similarly, the angle between the fifth radar and the installation direction of the first radar 1 is, for example, close to 90 degrees, for example, the fifth radar scans to the left.

[0104] The third parameter includes, for example, parameters TLl2up and TLr2up. TLr2up represents a parameter for converting the coordinate system of the fourth radar 4 on the right side to the vehicle coordinate system, and TLl2up represents a parameter for converting the coordinate system of the fifth radar 5 on the left side to the vehicle coordinate system.

[0105] Figure 5 A schematic diagram showing time synchronization according to other embodiments of the present disclosure.

[0106] like Figure 5 As shown, the point cloud data of the fourth radar (right-viewing radar) and the point cloud data of the fifth radar (left-viewing radar) are synchronized with the turntable attitude sensor data in time, and Figure 4 The forward-looking radar and the turntable attitude sensor data are time-synchronized in a similar manner. In addition, the point cloud data of the forward-looking radar, the point cloud data of the fourth radar, and the point cloud data of the fifth radar may also be time-synchronized with the turntable attitude sensor data.

[0107] The fourth radar, fifth radar, and forward-looking radar data are split, and the splitting frequency is determined by the radar frequency and the frequency of the turntable attitude sensor data, so that the split point cloud data and the turntable attitude sensor data are in the same frequency. The split data can be compared by timestamp to find the data with a time difference within the allowable range (that is, the time difference between the left, right, and forward-looking laser data is within the preset allowable range), so as to achieve time synchronization between the radar point cloud data and the turntable attitude sensor data. The allowable range refers to a pre-set time interval range.

[0108] Based on the petal-type scanning radar, the turntable attitude sensor data can be used and multi-threading can be adopted to build a map of the excavator working environment. The amount of map data is small and the map building is highly real-time.

[0109] In some embodiments, the point cloud map construction method further includes: calculating a registration score of the point cloud map, and adjusting the second parameter and / or the third parameter according to the registration score of the point cloud map.

[0110] The registration process is, for example:

[0111] Step 1, extracting the non-excavation area point clouds in the front view, left and right radar sub-maps from the constructed point cloud map respectively;

[0112] Step 2: Extract point cloud features. For example, point and surface features can be extracted for registration;

[0113] Step 3, register the left and right radar point clouds with the forward-looking radar point cloud respectively;

[0114] Step 4, perform registration through the iterative closest point (icp) registration algorithm to obtain the registration score. The registration score and the external parameter TcalLx2gnd obtained through the registration can be obtained through multiple iterative registrations. The registration process is performed, for example, in the off-vehicle coordinate system. Because the point clouds of the excavation area and the unloading area will change in real time with the excavation and loading process, the point cloud of the non-excavation and loading area (point cloud data that does not belong to the excavation area and the unloading area) is used during registration to improve the registration accuracy. If the registration effect is good and the sub-map overlap is high, the registration score is high. Taking the third parameter as an example, TcalLx2gnd, for example, includes the transformation matrix TcalLl2gnd from the coordinate system of the left-view radar to the off-vehicle coordinate system and the transformation matrix TcalLr2gnd from the coordinate system of the right-view radar to the off-vehicle coordinate system.

[0115] Step 5: Adjust the radar extrinsic parameters according to the registration score. For example, the above steps may be iteratively performed, and if the registration score converges to a certain value, the radar extrinsic parameters are automatically adjusted.

[0116] Taking the third parameter (the parameter for converting the point cloud data of the fourth radar and the point cloud data of the fifth radar into the vehicle coordinate system) as an example, the automatic adjustment of the radar external parameter can be calculated according to the following formula.

[0117] TcalLx2up = TcalLx2gnd / Tup2gnd.

[0118] Tup2gnd is the transformation matrix from the vehicle coordinate system to the vehicle coordinate system. "x" in TcalLx2up and TcalLx2gnd represents left or right. TcalLx2up includes parameters for transforming the right-looking radar coordinate system to the vehicle coordinate system and the left-looking radar coordinate system to the vehicle coordinate system.

[0119] In some embodiments, a point cloud map is constructed based on the time-synchronized material point cloud data, the point cloud data of the fourth radar, the point cloud data of the fifth radar, and the turntable attitude sensor data, including: converting the time-synchronized material point cloud data, the point cloud data of the fourth radar, and the point cloud data of the fifth radar from the on-vehicle coordinate system to the off-vehicle coordinate system; and constructing a point cloud map based on the point cloud data converted to the off-vehicle coordinate system.

[0120] By spatially transforming the material point cloud extracted from the on-vehicle coordinate system to the excavator's off-vehicle coordinate system (assuming that the excavator is not moving and the off-vehicle coordinate system is a static coordinate system), the generated point cloud is automatically spliced ​​in the static coordinate system as the excavator moves, thereby constructing a point cloud map of the material in the excavator's construction area.

[0121] For example, the radar data that have been time-synchronized are transformed from the vehicle coordinate system to the vehicle coordinate system to obtain radar sub-maps. Then, each sub-map is merged according to a specific data structure organization to form a hybrid map, which is the final point cloud map.

[0122] The transformation parameters (TLflu2Lfld, TLfrd2Lfld) of each forward-looking radar coordinate system to the main radar can be calibrated by the common view area external parameter manual calibration method.

[0123] The parameters (TLfd2up, TLr2up, TLl2up) of each radar transformed to the vehicle coordinate system can be automatically calibrated by sub-image registration. Among them, TLfd2up represents the parameters of the main radar coordinate system transformed to the vehicle coordinate system, TLr2up represents the parameters of the right-view radar coordinate system transformed to the vehicle coordinate system, and TLl2up represents the parameters of the left-view radar coordinate system transformed to the vehicle coordinate system.

[0124] The parameter Tup2gnd of each radar transforming from the on-vehicle coordinate system to the off-vehicle coordinate system can be determined according to the on-vehicle posture rpy combined with the rotation speed compensation.

[0125] In some embodiments, the point cloud data corresponding to the boom system is removed from the merged point cloud data to obtain material point cloud data, including: according to the structural parameters of the boom system, the point cloud data corresponding to the boom system is removed from the merged point cloud data to obtain material point cloud data.

[0126] For example, after the forward-looking radar point cloud data is transformed into the vehicle coordinate system for spatial synchronization, it is in the same vehicle coordinate system as the boom. According to the structural parameters of the boom, a motion model of the boom can be established (such as constraints on joint activity angles and range of motion), or the key point method or point-to-surface distance method can be used to identify and extract the boom point cloud and predict the position and posture of the boom in different states. By comparing these predictions with the actual point cloud data, the point cloud of the boom can be identified. Machine learning models can also be used to identify or remove point cloud data corresponding to the boom system. The extracted boom point cloud is a separate data and is no longer saved in the map.

[0127] Figure 6 A schematic diagram showing construction of a point cloud map according to some embodiments of the present disclosure is shown.

[0128] like Figure 6As shown, the three forward-looking radars, the second radar (front upper left Lflu), the first radar (front lower left Lfld), and the third radar (front lower right Lfrd), are first spatially synchronized in the coordinate system of the main radar (first radar), and then the synchronized radars are converted to the vehicle coordinate system. In the vehicle coordinate system, the point cloud extraction of the boom system can be performed. In addition, the fourth radar (right Lr) and the fifth radar (left Lr) can also be transformed to the vehicle coordinate system and then synchronized in time. The radar point cloud data after time synchronization can be transformed to the off-vehicle coordinate system to construct a hybrid map.

[0129] Figure 7 A block diagram of a point cloud map constructing device according to some embodiments of the present disclosure is shown.

[0130] like Figure 7 As shown, the point cloud map construction device 2 includes an acquisition module 71, a merging module 72, a removal module 73, and a construction module 74.

[0131] The acquisition module 71 is configured to acquire point cloud data of a first radar, a second radar, and a third radar of the excavator, wherein the first radar and the second radar are located on the top of the cab of the excavator, the third radar is located in an adjacent area of ​​the cab, there is an angle between the installation direction of the first radar and the installation direction of the second radar, the angle between the installation direction of the first radar and the operating direction of the excavator is less than 90 degrees, the angle between the installation direction of the second radar and the operating direction of the excavator is less than 90 degrees, and the angle between the installation direction of the third radar and the operating direction of the excavator is less than 90 degrees. For example, the following steps are performed: Figure 1 Step S1 is shown.

[0132] The merging module 72 is configured to merge the point cloud data of the first radar, the second radar and the third radar, for example, by performing the following steps: Figure 1 Step S2 is shown.

[0133] The removal module 73 is configured to remove the point cloud data corresponding to the boom system from the merged point cloud data to obtain the material point cloud data, for example, by performing the following steps: Figure 1 Step S3 shown.

[0134] The construction module 74 is configured to construct a point cloud map according to the material point cloud data, for example, by performing the following steps: Figure 1 Step S4 is shown.

[0135] Some embodiments of the present disclosure provide a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the point cloud map construction method according to any of the embodiments.

[0136] Some embodiments of the present disclosure provide a computer program product, including computer program instructions, which, when executed by a processor, implement the point cloud map construction method according to any of the embodiments.

[0137] Some embodiments of the present disclosure provide an excavator, including: a point cloud map building device according to any of the embodiments.

[0138] Figure 8 A block diagram of an electronic device according to some other embodiments of the present disclosure is shown.

[0139] like Figure 8 As shown, the electronic device 8 includes a memory 81; and a processor 82 coupled to the memory 81, the memory 81 is used to store instructions for executing the point cloud map construction method. The processor 82 is configured to execute the point cloud map construction method in any of the embodiments of the present disclosure based on the instructions stored in the memory 81.

[0140] Fig. 9 A block diagram of a computer system for implementing some embodiments of the present disclosure is shown.

[0141] like Fig. 9 As shown, the computer system 90 may be embodied in the form of a general-purpose computing device. The computer system 90 includes a memory 910, a processor 920, and a bus 900 that connects the various system components.

[0142] The memory 910 may include, for example, a system memory, a non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application, a boot loader, and other programs. The system memory may include a volatile storage medium, such as a random access memory (RAM) and / or a cache memory. The non-volatile storage medium may store, for example, instructions for executing the point cloud map construction method in any of the embodiments of the present disclosure. The non-volatile storage medium includes, but is not limited to, a disk memory, an optical memory, a flash memory, etc.

[0143] The processor 920 can be implemented by a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistors, etc. Discrete hardware components. Accordingly, each module such as the judgment module and the determination module can be implemented by a central processing unit (CPU) running instructions in a memory that execute corresponding steps, or can be implemented by a dedicated circuit that executes corresponding steps.

[0144] The bus 900 may use any of a variety of bus architectures, including, but not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, and a Peripheral Component Interconnect (PCI) bus.

[0145] The computer system 90 may also include an input / output interface 930, a network interface 940, a storage interface 950, etc. These interfaces 930, 940, 950, the memory 910, and the processor 920 may be connected via a bus 900. The input / output interface 930 may provide a connection interface for input / output devices such as a display, a mouse, and a keyboard. The network interface 940 may provide a connection interface for various networked devices. The storage interface 950 may provide a connection interface for external storage devices such as a floppy disk, a USB flash drive, and an SD card.

[0146] Here, various aspects of the present disclosure are described with reference to flowcharts and / or block diagrams of methods, devices, and computer program products according to embodiments of the present disclosure. It should be understood that each frame of the flowchart and / or block diagram and the combination of frames can be implemented by computer-readable program instructions.

[0147] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable device to produce a machine, so that the processor executes the instructions to produce means for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0148] These computer-readable program instructions can also be readable and stored in a computer-readable memory, which cause the computer to work in a specific manner to produce a product, including instructions for implementing the functions specified in one or more blocks in the flowchart and / or block diagram.

[0149] The present disclosure can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects.

[0150] So far, the point cloud map construction method and device, and computer-readable storage medium according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed here.

Claims

1. A point cloud map construction method, comprising: Obtaining point cloud data of a first radar, a second radar, and a third radar of the excavator, wherein the first radar and the second radar are located on the top of the cab of the excavator, the third radar is located in an adjacent area of ​​the cab, an angle is formed between an installation direction of the first radar and an installation direction of the second radar, an angle between an installation direction of the first radar and an operating direction of the excavator is less than 90 degrees, an angle between an installation direction of the second radar and an operating direction of the excavator is less than 90 degrees, and an angle between an installation direction of the third radar and an operating direction of the excavator is less than 90 degrees; Merging the point cloud data of the first radar, the second radar and the third radar; From the merged point cloud data, remove the point cloud data corresponding to the boom system to obtain the material point cloud data; Build a point cloud map based on the material point cloud data.

2. The point cloud map construction method according to claim 1, wherein: The point cloud data obtained by combining the first radar, the second radar and the third radar includes: converting the point cloud data of the first radar, the second radar and the third radar into the same coordinate system; In the same coordinate system, the point cloud data of the first radar, the second radar and the third radar are merged.

3. The point cloud map construction method according to claim 2, wherein: The step of converting the point cloud data of the first radar, the second radar, and the third radar into the same coordinate system includes: The point cloud data of the second radar and the point cloud data of the third radar are converted into the coordinate system of the first radar by using the first parameter.

4. The point cloud map construction method according to claim 3, further comprising: Before converting the point cloud data of the second radar and the point cloud data of the third radar into the coordinate system of the first radar using the first parameter, displaying the point cloud data of the first radar, the point cloud data of the second radar, and the point cloud data of the third radar in the same view; In the view, the first parameter is determined by aligning the point cloud of the bucket in the point cloud data of the first radar, the second radar, and the third radar.

5. The point cloud map construction method according to claim 3, wherein: The step of removing the point cloud data corresponding to the boom system from the merged point cloud data to obtain the material point cloud data includes: Using the second parameter, the merged point cloud data is converted from the coordinate system of the first radar to the coordinate system of the vehicle; In the vehicle coordinate system, the point cloud data corresponding to the boom system is removed from the merged point cloud data to obtain the material point cloud data.

6. The point cloud map construction method according to claim 5, wherein: The step of constructing a point cloud map based on the material point cloud data includes: Get turntable attitude sensor data; Time synchronization of material point cloud data and turntable attitude sensor data; A point cloud map is constructed based on the time-synchronized material point cloud data and turntable attitude sensor data.

7. The method for constructing a point cloud map according to claim 6, wherein the step of constructing a point cloud map based on the time-synchronized material point cloud data and turntable attitude sensor data comprises: Acquire point cloud data of a fourth radar and a fifth radar of the excavator, wherein the fourth radar and the fifth radar are located on both sides of the excavator, and an angle between an installation direction of the fourth radar and an operating direction of the excavator is greater than an angle between an installation direction of any one of the first radar, the second radar, and the third radar and the operating direction of the excavator, and an angle between an installation direction of the fifth radar and the operating direction of the excavator is greater than an angle between an installation direction of any one of the first radar, the second radar, and the third radar and the operating direction of the excavator; Using the third parameter, the point cloud data of the fourth radar and the point cloud data of the fifth radar are converted into the vehicle coordinate system; In the vehicle coordinate system, time-synchronize the point cloud data of the fourth radar and the point cloud data of the fifth radar with the turntable attitude sensor data; A point cloud map is constructed based on the time-synchronized material point cloud data, the point cloud data of the fourth radar, the point cloud data of the fifth radar, and the turntable attitude sensor data.

8. The point cloud map construction method according to claim 7, further comprising: Calculate the registration score of the point cloud map, The second parameter and / or the third parameter is adjusted according to the registration score of the point cloud map.

9. The point cloud map construction method according to claim 7, wherein: According to the time-synchronized material point cloud data, the point cloud data of the fourth radar, the point cloud data of the fifth radar and the turntable attitude sensor data, a point cloud map is constructed, including: Convert the time-synchronized material point cloud data, the fourth radar point cloud data, and the fifth radar point cloud data from the vehicle-on-board coordinate system to the vehicle-off-board coordinate system; Construct a point cloud map based on the point cloud data converted to the get-off coordinate system.

10. The point cloud map construction method according to claim 1, wherein: From the merged point cloud data, remove the point cloud data corresponding to the boom system to obtain the material point cloud data, including: According to the structural parameters of the boom system, the point cloud data corresponding to the boom system is removed from the merged point cloud data to obtain the material point cloud data.

11. The point cloud map construction method according to claim 1, wherein: The angle between the installation direction of the first radar and the ground is less than 90 degrees, and the angle between the installation direction of the second radar and the ground is greater than 90 degrees; The imaging areas of the first radar, the second radar, and the third radar partially overlap.

12. A point cloud map construction device, comprising: an acquisition module, configured to acquire point cloud data of a first radar, a second radar, and a third radar of an excavator, wherein the first radar and the second radar are located on top of a cab of the excavator, the third radar is located in an adjacent area of ​​the cab, an angle is formed between an installation direction of the first radar and an installation direction of the second radar, an angle between an installation direction of the first radar and an operating direction of the excavator is less than 90 degrees, an angle between an installation direction of the second radar and an operating direction of the excavator is less than 90 degrees, and an angle between an installation direction of the third radar and an operating direction of the excavator is less than 90 degrees; a merging module configured to merge the point cloud data of the first radar, the second radar and the third radar; A removal module is configured to remove the point cloud data corresponding to the boom system from the merged point cloud data to obtain material point cloud data; The construction module is configured to construct a point cloud map based on the material point cloud data.

13. An electronic device, comprising: Memory; as well as A processor coupled to the memory, wherein the processor is configured to execute the point cloud map construction method according to any one of claims 1 to 11 based on instructions stored in the memory.

14. A computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, implement the point cloud map construction method according to any one of claims 1 to 11.

15. A computer program product, comprising computer program instructions, which, when executed by a processor, implement the point cloud map construction method according to any one of claims 1 to 11.

16. An excavator, comprising: The point cloud map constructing device according to claim 12.

Images