Construction environment map generation method and device, engineering machinery and storage medium

Abstract

The invention provides a construction environment map generation method and device, engineering machinery, a computer readable storage medium and a computer program product, and the method comprises the steps: determining a first rotation matrix between a currently collected image and a key image determined last time, determining a first transformation matrix and a first three-dimensional space rotation angle between the currently collected point cloud frame and the point cloud local map; determining whether the point cloud frame is a key point cloud frame according to the first rotation matrix, the first transformation matrix and the first three-dimensional space rotation angle; and under the condition that the point cloud frame is the key point cloud frame, fusing the currently collected point cloud frame with the point cloud local map to generate a map. The method can effectively reduce the loss of key point cloud frame data, improves the robustness of construction environment mapping, improves the precision of a map, and improves the operation safety of engineering machinery. Environment mapping can be carried out under the condition that high-precision inertial navigation equipment is not adopted, the calculation complexity is low, and the real-time performance is high.

Description

Technical Field

[0001] The present application relates to the technical field of engineering machinery, and in particular to a method and device for generating a construction environment map, engineering machinery, a computer-readable storage medium, and a computer program product. Background Art

[0002] Construction machinery is a vital component of the equipment industry and comes in many forms, such as underground loaders. Construction machinery is used in a variety of scenarios, including underground construction, such as tunneling and ore mining. Underground construction environments often face conditions such as poor lighting, confined spaces, and complex terrain. They may also contain various obstacles, pipes, and cables, increasing the difficulty and risk of construction. Mapping underground construction environments, such as these, facilitates construction route planning and provides real-time location information for personnel and equipment, preventing collisions and other safety incidents. However, due to the unique characteristics of underground spaces, satellite positioning signals are not effectively covered, making traditional satellite-based positioning methods (such as GPS) ineffective. Currently, mapping of construction environments is typically done using sensors such as LiDAR. However, due to LiDAR's inherent performance limitations and noise interference, key point cloud frames used for map construction may be lost during the mapping process, resulting in poor robustness and low map accuracy. Summary of the Invention

[0003] The present disclosure provides a method and device for generating a construction environment map, engineering machinery, a computer-readable storage medium, and a computer program product.

[0004] According to a first aspect of the present disclosure, a method for generating a construction environment map is provided, comprising: acquiring an image captured by an image acquisition device for the construction environment and a point cloud captured by a radar device for the construction environment, and performing time synchronization and coordinate system conversion processing on the captured image and point cloud; determining a first rotation matrix between a currently captured image and a key image determined last time, and determining a first transformation matrix and a first three-dimensional space rotation angle between the currently captured point cloud and a point cloud local map; determining whether the currently captured point cloud frame is a key point cloud frame based on the first rotation matrix, the first transformation matrix and the first three-dimensional space rotation angle; wherein, if the currently captured point cloud frame is determined to be a key point cloud frame, determining that the currently captured image is a key image; and in the case that the currently captured point cloud frame is a key point cloud frame, fusing the currently captured point cloud with the point cloud local map to generate a map corresponding to the construction environment.

[0005] Optionally, determining whether the currently acquired point cloud frame is a key point cloud frame based on the first rotation matrix, the first transformation matrix and the first three-dimensional space rotation angle includes: determining the second transformation matrix between the currently acquired point cloud and the point cloud local map based on the first rotation matrix, the first transformation matrix and the first three-dimensional space rotation angle; determining transformation matrix difference information based on the second transformation matrix and the first transformation matrix; and determining whether the currently acquired point cloud frame is a key point cloud frame based on the transformation matrix difference information.

[0006] Optionally, determining the second transformation matrix between the currently acquired point cloud and the point cloud local map based on the first rotation matrix, the first transformation matrix and the first three-dimensional space rotation angle includes: converting the first rotation matrix into a second rotation matrix in the radar device coordinate system based on the calibration rotation matrix between the image acquisition device and the radar device, and determining the second three-dimensional space rotation angle corresponding to the second rotation matrix; performing weighted averaging processing on the first three-dimensional space rotation angle and the second three-dimensional space rotation angle to obtain a third three-dimensional space rotation angle; and determining the second transformation matrix based on the displacement vector of the first transformation matrix and the third three-dimensional space rotation angle.

[0007] Optionally, determining the transformation matrix difference information based on the second transformation matrix and the first transformation matrix includes: calculating the product of the inverse matrix of the first transformation matrix and the second transformation matrix as the transformation matrix difference information.

[0008] Optionally, determining whether the currently acquired point cloud frame is a key point cloud frame based on the transformation matrix difference information includes: calculating the sum of the absolute values ​​of the displacements in the three axis directions in the transformation matrix difference information as shift information; calculating the sum of the absolute values ​​of the three spatial rotation angles in the transformation matrix difference information as rotation angle information; and determining that the currently acquired point cloud frame is a key point cloud frame when the shift information is greater than a shift threshold and / or the rotation angle information is greater than a rotation angle threshold.

[0009] Optionally, the fusing of the currently collected point cloud with the point cloud local map includes: transforming the currently collected point cloud according to the second transformation matrix; and fusing the currently collected point cloud that has undergone the transformation into the point cloud local map.

[0010] Optionally, determining the first rotation matrix between the currently acquired image and the last determined key image includes: estimating the posture of the image acquisition device based on the epipolar geometry constraint, determining the first rotation matrix between the currently acquired image and the last determined key image, and the three-dimensional space rotation angle corresponding to the first rotation matrix.

[0011] Optionally, determining the first transformation matrix and the first three-dimensional space rotation angle between the currently collected point cloud and the point cloud local map includes: determining the first transformation matrix and the second three-dimensional space rotation angle between the currently collected point cloud and the point cloud local map through a normal distribution transform NDT algorithm.

[0012] Optionally, the image acquisition device periodically acquires images based on a preset sampling time interval; and the radar device periodically acquires point clouds based on the sampling time interval.

[0013] Optionally, the time synchronization and coordinate system conversion processing of the collected images and point clouds includes: using the Network Time Protocol NTP server to perform time calibration processing on the image acquisition device and the radar device to perform time synchronization processing on the collected images and point clouds; using the PNP algorithm to determine the transformation matrix between the image acquisition device and the radar device to perform coordinate system conversion processing on the collected images and point clouds.

[0014] According to a second aspect of the present disclosure, a device for generating a construction environment map is provided, comprising: a data acquisition module for acquiring images captured by an image acquisition device for the construction environment and point clouds captured by a radar device for the construction environment, and performing time synchronization and coordinate system conversion processing on the captured images and point clouds; an information determination module for determining a first rotation matrix between a currently captured image and a key image determined last time, and determining a first transformation matrix and a first three-dimensional space rotation angle between a currently captured point cloud and a point cloud local map; a key data determination module for determining whether the currently captured point cloud frame is a key point cloud frame based on the first rotation matrix, the first transformation matrix and the first three-dimensional space rotation angle; wherein, if the currently captured point cloud frame is a key point cloud frame, the currently captured image is determined to be a key image; a map generation module for fusing the currently captured point cloud frame with the point cloud local map when the currently captured point cloud frame is a key point cloud frame, so as to generate a map corresponding to the construction environment.

[0015] According to a third aspect of the present disclosure, a device for generating a construction environment map is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the method described above based on instructions stored in the memory.

[0016] According to a fourth aspect of the present disclosure, there is provided a construction machine comprising: the device for generating a construction environment map as described above.

[0017] According to a fifth aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the steps of the method described above are implemented.

[0018] According to a sixth aspect of the present disclosure, a computer program product is provided, comprising a computer program, which implements the steps of the method described above when executed by a processor.

[0019] The disclosed method, device, engineering machinery, computer-readable storage medium, and computer program product for generating a construction environment map can determine whether a collected point cloud frame is a key point cloud frame based on images and point clouds collected of the construction environment, and generate a map corresponding to the construction environment through the key point cloud frame; can determine the key point cloud frame based on the point cloud frame and the image, effectively reducing the loss of key point cloud frame data, improving the robustness of construction environment mapping, improving the accuracy of the map, and improving the safety of engineering machinery operation; can perform environmental mapping without using high-precision inertial navigation equipment, with low computational complexity and high real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other purposes, features and advantages of the present disclosure will become more apparent by describing the embodiments of the present disclosure in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the present disclosure and do not constitute a limitation of the present disclosure. The above and other purposes and advantages of the present disclosure will be further described below in conjunction with specific embodiments and with reference to the accompanying drawings. In the accompanying drawings, the same or corresponding technical features or components will be represented by the same or corresponding figure marks.

[0021] Figure 1 1 is a flowchart of some embodiments of the method for generating a construction environment map according to the present disclosure;

[0022] Figure 2 Schematic diagram of the relationship between the camera coordinate system, lidar coordinate system and image plane;

[0023] Figure 31 is a schematic diagram of a process for determining key point cloud frames in some embodiments of the method for generating a construction environment map according to the present disclosure;

[0024] Figure 4 1 is a schematic diagram of a process for determining a second transformation matrix between a point cloud and a point cloud local map in some embodiments of the method for generating a construction environment map according to the present disclosure;

[0025] Figure 5 Schematic diagram of a process for determining whether a point cloud frame is a key point cloud frame in other embodiments of the method for generating a construction environment map according to the present disclosure;

[0026] Figure 6 Schematic diagram of a process for determining whether a point cloud frame is a key point cloud frame in still other embodiments of the method for generating a construction environment map according to the present disclosure;

[0027] Figure 7 1 is a schematic diagram of a process for fusing a point cloud into a point cloud local map in some embodiments of the method for generating a construction environment map according to the present disclosure;

[0028] Figure 8 Schematic diagram of modules of some embodiments of the apparatus for generating a construction environment map according to the present disclosure;

[0029] Figure 9 Schematic diagrams of modules of other embodiments of the device for generating a construction environment map according to the present disclosure. DETAILED DESCRIPTION

[0030] Exemplary embodiments of the present disclosure will be described below with reference to the accompanying drawings. For the sake of clarity and conciseness, not all features of the embodiments are described in the specification. However, it should be understood that many implementation-specific settings must be made in the process of implementing the embodiments in order to achieve the developer's specific goals, such as meeting those restrictions related to equipment and services, and these restrictions may vary depending on the implementation. In addition, it should be understood that although the development work may be very complex and time-consuming, it is only a routine task for those skilled in the art who benefit from the contents of this disclosure.

[0031] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

[0032] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present disclosure are only used to distinguish different steps, devices or modules, and do not represent any specific technical meanings, nor do they indicate a necessary logical order between them.

[0033] It should also be understood that in the embodiments of the present disclosure, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two, or more than two.

[0034] It should also be understood that any component, data or structure mentioned in the embodiments of the present disclosure can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0035] In addition, the term "and / or" in this disclosure is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this disclosure generally indicates that the related objects are in an "or" relationship.

[0036] It should also be understood that the description of the various embodiments in this disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0037] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0038] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0039] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0040] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0041] In addition, in order to avoid obscuring the present disclosure with unnecessary details, only the processing steps and / or device structures that are closely related to at least the solution according to the present disclosure are shown in the drawings, while other details that are not closely related to the present disclosure are omitted. It should also be noted that similar reference numerals and letters in the drawings indicate similar items, and therefore once an item is defined in one drawing, it does not need to be discussed again for subsequent drawings.

[0042] Figure 1 FIG. 1 is a flow chart of some embodiments of the method for generating a construction environment map according to the present disclosure. Figure 1 As shown, the method for generating a construction environment map includes steps S101-S104.

[0043] Step S101: obtain images of the construction environment collected by the image acquisition device and point clouds of the construction environment collected by the radar device, and perform time synchronization and coordinate system conversion processing on the collected images and point clouds.

[0044] Image acquisition equipment can be various cameras, such as cameras; radar equipment can be lidar, such as laser radar. Construction machinery can be, for example, an underground scraper, which can be equipped with image acquisition equipment and radar equipment to capture images and point clouds of the same area within a construction environment. Construction environments include tunnel construction, ore mining, and other diverse environments. Images captured by the image acquisition equipment for the construction environment can be image frames, and point clouds captured by the radar equipment can be point cloud frames.

[0045] Step S102: Determine a first rotation matrix between the currently captured image and the previously determined key image, and determine a first transformation matrix between the currently captured point cloud and the point cloud local map, as well as a first three-dimensional rotation angle. The three-dimensional rotation angle is the rotation angle around the x, y, and z axes of the coordinate system.

[0046] Step S103, determining whether the currently acquired point cloud frame is a key point cloud frame based on the first rotation matrix, the first transformation matrix and the first three-dimensional space rotation angle; wherein, if the currently acquired point cloud frame is determined to be a key point cloud frame, then determining the currently acquired image to be a key image.

[0047] Key point cloud frames are representative point cloud frames selected from the collected point cloud that can effectively represent changes in the construction scene and can balance mapping efficiency and accuracy. A variety of methods can be used to determine key point cloud frames. When a point cloud frame is determined to be a key point cloud frame, the corresponding image captured by the image acquisition device is determined to be a key image.

[0048] Step S104 , when the currently acquired point cloud frame is a key point cloud frame, the currently acquired point cloud is fused with the point cloud local map to generate a map corresponding to the construction environment.

[0049] When the radar device first captures a point cloud frame, it is identified as a key point cloud frame, and a local point cloud map is constructed based on this point cloud. The first image captured by the image acquisition device is also identified as a key image. The radar device's starting position can be used as the coordinate origin of the entire environment map, and key point cloud frames can be continuously integrated into the existing local point cloud map to generate a map corresponding to the construction environment. The map corresponding to the construction environment can be a three-dimensional point cloud map, for example. After the image acquisition device and radar device complete the image and point cloud acquisition of the construction environment, a comprehensive point cloud map of the construction environment is generated.

[0050] The method for generating a construction environment map disclosed herein determines whether the collected point cloud frame is a key point cloud frame based on the image collected by the image acquisition device and the point cloud collected by the radar device, and generates a map corresponding to the construction environment through the key point cloud frame. In view of the possibility that key point cloud frame data used to construct the map may be lost during the mapping process, the key point cloud frame is determined based on the point cloud frame and the image, which effectively reduces the loss of key point cloud frame data, improves the robustness of the construction environment mapping, and improves the accuracy of the map. It can perform environmental mapping without using high-precision inertial navigation equipment, with low calculation complexity and high real-time performance.

[0051] In some embodiments, the image acquisition device periodically acquires images based on a preset sampling time interval; the radar device periodically acquires point clouds based on a sampling time interval; the sampling time interval can be set, for example, to 100ms.

[0052] For example, a camera and a lidar can be used to periodically collect image data img and point cloud data scan, respectively, with a sampling interval of 100ms. Because the camera and lidar have different operating frequencies, start times, and coordinate systems, time synchronization and coordinate conversion algorithms can be used to achieve precise alignment of the time dimensions of image and point cloud data and a unified coordinate system. The camera and lidar output a frame of image and point cloud every 100ms.

[0053] Various methods can be used to synchronize the time and coordinate system of captured images and point clouds. Using a Network Time Protocol (NTP) server, image acquisition devices and radar equipment can be time-calibrated to synchronize the captured images and point clouds. For example, an NTP server can send time synchronization messages to image acquisition devices and radar equipment to align the two devices and ensure that the captured image data and point cloud data are collected at the same time.

[0054] The PNP (Perspective-n-Point) algorithm is used to determine the transformation matrix between the image acquisition device and the radar device. This matrix is ​​used to transform the acquired images and point clouds into different coordinate systems. The transformation matrix includes a rotation matrix and a translation vector. The existing PNP algorithm can be used to calibrate the rotation matrix R and translation vector t in the transformation matrix between the radar and image acquisition device coordinate systems, thus achieving the coordinate system transformation from the radar device to the image acquisition device.

[0055] For example, the image acquisition device is a camera and the radar device is a lidar. In the process of calibrating the transformation matrix between the camera and the lidar, it is necessary to make the camera and the lidar scan the same target, obtain the three-dimensional coordinates of the target in the lidar coordinate system through the lidar, and obtain the pixel coordinates of the same target on the image plane through the image taken by the camera.

[0056] The coordinate system of the camera is also called the camera coordinate system. The relationship between the camera coordinate system, the lidar coordinate system and the image plane is as follows: Figure 2 As shown. Figure 2 In the figure, point Pw is a point on the target in the LiDAR coordinate system, which can be a rectangular box or other target. Pi is the point corresponding to point Pw in the image plane. Oc is the origin of the camera's 3D coordinate system, with the Xc, Yc, and Zc axes being the three perpendicular axes of the camera coordinate system. Oi is the origin of the image's 2D coordinate system, with the u and v axes being the two perpendicular axes of the image coordinate system.

[0057] Ow1 is the origin of the laser radar's three-dimensional coordinate system. The Xw, Yw, and Zw axes are the three mutually perpendicular coordinate axes of the laser radar coordinate system. The straight line connecting points Pw, Pi, and Oc is the projection line of the three-dimensional point Pw in the laser radar coordinate system on the image plane. The projection line is obtained by connecting points Pw and Oc. The intersection of the projection line and the image plane is the imaging point of point Pw on the image. The transformation relationship is shown in the following equation (1-1), where the transformation relationship between the three-dimensional coordinates (xw, yw, zw) of point Pw in the laser three-dimensional coordinate system and the two-dimensional coordinates (ui, vi) in the image coordinate system is shown in the following equation:

[0058] (1-1);

[0059] Among them, Z is the scale factor; f is the focal length of the camera; dX, dY represent the physical length of a pixel on the photosensitive plate in the X-axis and Y-axis directions respectively; (u0, v0) represent the coordinates of the center of the camera photosensitive plate in the pixel coordinate system respectively; R is the rotation matrix from the three-dimensional coordinate system of the lidar to the three-dimensional coordinate system of the camera; t is the translation vector from the three-dimensional coordinate system of the lidar to the three-dimensional coordinate system of the camera; K is the intrinsic parameter matrix of the camera; Z in formula (1-1) is the Z coordinate of point Pw in the camera coordinate system. Multiplying 1 / Z and the three-dimensional coordinate of point Pw gives the normalized coordinate of P in the camera coordinate system, which is located on the plane with Z=1 in front of the camera.

[0060] Since the operating frequencies, start times, etc. of image acquisition devices and radar devices may be different, time synchronization of the acquired images and point clouds can ensure that the data of the two correspond in the time dimension; and the coordinate systems of different sensors of image acquisition devices and radar devices are different. Coordinate system conversion of the acquired images and point clouds can unify the data into the same coordinate system, which is convenient for subsequent processing; through time synchronization and coordinate system conversion, image and point cloud data can be effectively integrated.

[0061] Figure 3 FIG. 1 is a flow chart of determining key point cloud frames in some embodiments of the method for generating a construction environment map according to the present disclosure, such as Figure 3 As shown:

[0062] Step S301 : determining a second transformation matrix between the currently collected point cloud and the point cloud local map according to the first rotation matrix, the first transformation matrix and the first three-dimensional space rotation angle.

[0063] Step S302: Determine transformation matrix difference information according to the second transformation matrix and the first transformation matrix.

[0064] Step S303: Determine whether the currently acquired point cloud frame is a key point cloud frame based on the transformation matrix difference information.

[0065] A variety of methods can be used to determine the first rotation matrix between the currently captured image and the previously determined key image. For example, the pose of the image capture device can be estimated based on existing epipolar geometry constraints, and the first rotation matrix between the currently captured image and the previously determined key image, as well as the three-dimensional rotation angle corresponding to the first rotation matrix, can be determined.

[0066] Epipolar geometry is a geometric model in computer vision that describes the projection relationship between the same spatial point in two images from different perspectives. Epipolar geometry constraints require no 3D scene information and can be established solely using the camera pose and image pixel coordinates. For the time-synchronized currently captured image frame img and the most recent keyframe image kimg (the last determined key image), the camera pose is estimated based on epipolar geometry constraints. The rotation matrix rimg (the first rotation matrix) between the currently captured image frame img and the most recent keyframe image kimg is calculated, and the corresponding rotation angles around the x, y, and z axes (the three-dimensional rotation angles corresponding to the first rotation matrix) are output.

[0067] Various methods can be used to determine the first transformation matrix and the first 3D rotation angle between the currently acquired point cloud and the local point cloud map. For example, the Normal Distribution Transform (NDT) algorithm can be used to determine the first transformation matrix and the second 3D rotation angle between the currently acquired point cloud and the local point cloud map. The NDT algorithm achieves point cloud alignment by modeling point cloud data as a multidimensional normal distribution and calculating the matching degree based on a statistical model.

[0068] Figure 4 FIG. 1 is a flow chart of determining a second transformation matrix between a point cloud and a point cloud local map in some embodiments of the method for generating a construction environment map according to the present disclosure, such as Figure 4 As shown:

[0069] Step S401: Convert a first rotation matrix into a second rotation matrix in the radar device coordinate system according to the calibration rotation matrix between the image acquisition device and the radar device, and determine a second three-dimensional space rotation angle corresponding to the second rotation matrix.

[0070] Step S402 : performing weighted averaging processing on the first three-dimensional space rotation angle and the second three-dimensional space rotation angle to obtain a third three-dimensional space rotation angle.

[0071] Step S403: Determine a second transformation matrix according to the displacement vector of the first transformation matrix and the third three-dimensional space rotation angle.

[0072] In some embodiments, the first rotation matrix rimg is converted to a second rotation matrix Ri2l in the radar device coordinate system using a calibration rotation matrix R between the image acquisition device and the radar device. A weighted average is performed by multiplying the three xyz-axis rotation angles corresponding to the second rotation matrix Ri2l (the second three-dimensional rotation angles) by α, and the three xyz-axis rotation angles corresponding to the first transformation matrix pose (the first three-dimensional rotation angles) by β, to obtain a third three-dimensional rotation angle. According to various existing methods, a new transformation matrix pose (the second transformation matrix) is formed based on the third three-dimensional rotation angles and the displacement vectors in the first transformation matrix pose, where α and β are adjustable values ​​that satisfy 0 ≤ α, β ≤ 1, and α + β = 1.

[0073] Figure 5 FIG. 1 is a flow chart of determining whether a point cloud frame is a key point cloud frame in another embodiment of the method for generating a construction environment map according to the present disclosure. Figure 5 As shown:

[0074] Step S501 : Calculate the product of the inverse matrix of the first transformation matrix and the second transformation matrix as transformation matrix difference information.

[0075] Step S502 : Calculate the sum of the absolute values ​​of the displacements in the three axis directions in the transformation matrix difference information as the shift information.

[0076] Step S503: Calculate the sum of the absolute values ​​of the three spatial rotation angles in the transformation matrix difference information as rotation angle information.

[0077] Step S504 : When the displacement information is greater than the displacement threshold and / or the rotation angle information is greater than the rotation angle threshold, determining that the currently acquired point cloud frame is a key point cloud frame.

[0078] In some embodiments, the image acquisition device is a camera, and the radar device is a lidar. Through time synchronization and coordinate system conversion, an association is established between the point cloud and the image, with each image frame corresponding to a point cloud frame. pose is the transformation matrix (first transformation matrix) between the current point cloud captured by the lidar and the latest local point cloud map. scans_part_map consists of a series of elements, each of which is a scan of point cloud data collected by the lidar and determined to be a key point cloud frame. scan_world is the pose transformation of the point cloud frame scanned by the lidar, resulting in a single point cloud frame fused with the global map. direct_kf_pose is the transformation matrix generated by registering the last key point cloud frame with the latest local point cloud map. part_map is the local point cloud map fused from all elements in scans_part_map.

[0079] Figure 6 FIG. 1 is a flow chart of determining whether a point cloud frame is a key point cloud frame in some other embodiments of the method for generating a construction environment map according to the present disclosure, such as Figure 6 As shown:

[0080] Step S601: The camera collects image data imgs (collecting one frame of image every 100 ms).

[0081] In step S602, the laser radar collects point cloud data, collecting one frame of point cloud every 100ms.

[0082] Step S603, determine whether the pointer of part_map is empty. If the pointer of part_map is empty, execute step S604, otherwise execute step S605.

[0083] In step S604, the currently acquired point cloud data scan is assigned to part_map (equivalent to initializing the local map); the unit pose is assigned to direct_kf_pose (used to record the initial state of the keyframe pose); and part_map is assigned to the target point cloud in the NDT algorithm (for subsequent point cloud matching).

[0084] Step S605 : If the part_map pointer is not empty, the currently collected point cloud data scan is assigned to the source point cloud of the NDT algorithm, that is, as input data to be matched with the target point cloud.

[0085] Step S606: Extract the two most recent poses T1 and T2 from the sequence variable est_pose of the point cloud pose. These two poses will be used as references for calculating the initial pose.

[0086] Step S607: Generate the initial pose of the NDT algorithm (init_pose = T1 * (T2.inv() * T1). This initial pose is the initial value for the pose between the subsequent precise calculation of the point cloud and the local point cloud map.

[0087] In the NDT algorithm, the initial point cloud pose (init_pose) is generated as follows: T1 = est_poses[est_poses length - 1]; T2 = est_poses[est_poses length - 2]; init_pose = T1 * (T2.inverse() * T1); where T1 and T2 represent the pose of the point cloud 100ms and 200ms apart from the current frame, respectively. The initial pose for registering the current frame with the local map is calculated as T1 * (T2.inverse() * T1).

[0088] Step S608: Obtain the precise pose between the point cloud scan and the point cloud local map part_map through the NDT algorithm, determine the position and posture of the current point cloud in the local map, and output the rotation angle of the point cloud scan relative to the part_map around the three axes xyz.

[0089] Step S609: retain the image imgs corresponding to scan.

[0090] Step S610: If the image key frame sequence is empty, assign imgs corresponding to scan to the image key frame sequence.

[0091] Step S611: output the most recent frame kimg in the image key frame sequence for subsequent image-based analysis.

[0092] Step S612: Calculate the image rotation matrix. Perform camera pose estimation based on epipolar geometry constraints and calculate the rotation matrix rimg between the current output image frame imgs and the most recent image key frame kimg.

[0093] In step S613 , the three rotation angles of Rimg and the three rotation angles corresponding to pose are fused, and the displacement vector in pose is used to form a new pose.

[0094] The rotation matrix rimg (first rotation matrix) is converted to the lidar coordinate system using the rotation matrix R (calibration rotation matrix) between the lidar and camera coordinate systems to generate the rotation matrix Ri2l (second rotation matrix). The three xyz-axis rotation angles (second 3D rotation angles) corresponding to the rotation matrix Ri2l are multiplied by α. The sum of the three xyz-axis rotation angles (first 3D rotation angles) corresponding to pose (first 3D rotation angles) is then multiplied by β to obtain the minimum rotation angle. This is combined with the displacement vector in pose to form the new pose (second transformation matrix). α and β are adjustable values ​​and must satisfy the conditions 0 ≤ α, β ≤ 1 and α + β = 1, for example, α = 0.2 and β = 0.8. The new pose is assigned to the est_pose sequence.

[0095] Step S614: Perform pose transformation on the point cloud scan to scan_world.

[0096] Step S615: Determine whether scan is a point cloud keyframe based on pose.

[0097] Based on the transformed pose obtained by fusion of the point cloud data and image data collected by the LiDAR and camera, determine whether the current frame is a key frame. If the current frame is a key frame, save the current frame scan to the key frame sequence scans_part_map after pose transformation. Otherwise, continue to receive the next point cloud scan frame and the next image frame from the LiDAR and camera.

[0098] Calculate the difference term Tdelta of the transformation matrix:

[0099] Tdelta = direct_kf_pose_.inverse() * current_pose (1-2);

[0100] Among them, direct_kf_pose_ is the transformation matrix corresponding to the key frame image or point cloud of the last frame (first transformation matrix); current_pose is the latest transformation matrix (second transformation matrix) after the fusion of image transformation and point cloud transformation parameters; Tdelta is the difference between the transformation matrix of the nearest frame point cloud relative to the local map and the transformation matrix of the current frame point cloud relative to the local map (transformation matrix difference information).

[0101] Determine the sum of the absolute values ​​of the displacements in the x / y / z directions in Tdelta (displacement information), as well as the sum of the absolute values ​​of the rotation angles (sd) (rotation angle information). If the sum of the absolute values ​​of the displacements st is greater than the fixed displacement threshold of 20 cm, the current frame point cloud is determined to be a key frame, otherwise it is determined that the current frame point cloud is not a key frame; if the sum of the absolute values ​​of the rotation angles sd is greater than the fixed rotation angle threshold of 3 degrees, the current frame point cloud is determined to be a key frame, otherwise it is determined that the current frame point cloud is not a key frame.

[0102] Step S616: Check whether the current point cloud or image frame is a key frame. If not, proceed to steps S601 and S602. If yes, proceed to step S617.

[0103] Step S617: perform point cloud fusion processing.

[0104] In some embodiments, a variety of methods can be used to fuse the currently acquired point cloud with the local point cloud map. The currently acquired point cloud can be transformed according to a second transformation matrix; the transformed currently acquired point cloud can be fused into the local point cloud map, i.e., the point cloud data after keyframe determination can be fused into the existing point cloud map.

[0105] For example, through algorithms such as NDT point cloud registration, new point cloud data is integrated with previously accumulated point cloud maps, allowing the point cloud maps to be continuously updated and improved, more accurately reflecting the actual environment. Each time a key frame of point cloud data is determined, it is fused into the point cloud map.

[0106] If the construction environment scan is complete, a 3D point cloud map is output. If the scan is not complete, the next frame of image and point cloud acquisition begins again. When the construction environment scan is complete (i.e., the scheduled scan task is completed or the set end condition is met), the final 3D point cloud map of the construction environment is output. This map can be used for subsequent applications such as navigation, environmental modeling, and target recognition.

[0107] Figure 7 FIG. 1 is a flow chart of fusing point clouds into a point cloud local map in some embodiments of the method for generating a construction environment map according to the present disclosure, such as Figure 7 As shown:

[0108] Step S701: Determine whether the current frame is a keyframe. The current frame is the currently captured point cloud frame, and a keyframe is a key point cloud frame. A keyframe is a representative frame in the data processing process that provides an important reference for subsequent operations. In point cloud map construction, key point cloud frames can better reflect environmental characteristics.

[0109] Step S702: Assign the pose of the current frame to direct_kf_pose. The pose contains position and attitude information, which is used as a reference for subsequent inter-frame relationship calculations and map construction. When calculating point cloud transformations, the relative motion of point clouds can be determined based on the poses of adjacent keyframes.

[0110] Step S703: Integrate the current keyframe into the global map. The global map is a 3D point cloud map of the entire scene environment. Integrating keyframes into the local map allows the map to be continuously updated and continuously mapped to a larger area in the environment.

[0111] Step S704: Assign scan_world (the point cloud data transformed to the coordinate system of the global point cloud map) to the last element of the key frame sequence array scans_part_map. This step is to store the key point cloud frame data in an orderly manner to facilitate subsequent operations such as building a local map.

[0112] Step S705 , check whether the number of point cloud key frames in scans_part_map is greater than a specified threshold. If yes, execute step S706 ; if not, execute step S707 .

[0113] By setting a threshold, you can balance data volume and processing efficiency. Too much data may lead to a surge in subsequent processing calculations and slow operation; too little data may not accurately express environmental characteristics. For example, specify a threshold of 20.

[0114] In step S706, if the number of point clouds in scans_part_map exceeds the threshold, the first frame of point cloud is discarded. This step simplifies the data volume while ensuring that the environmental characteristics can be effectively reflected, so that the system can process useful data more efficiently.

[0115] Step S707: Combine all elements in scans_part_map into a local map part_map. The local map focuses on the current area of ​​interest and is constructed from keyframe point cloud data, providing a more targeted map model for operations such as point cloud matching and pose estimation.

[0116] In step S708, part_map is set as the target point cloud for NDT (normal distribution transform, an algorithm commonly used for point cloud matching) point cloud registration. This step allows subsequent point cloud data to be matched using the NDT algorithm with part_map as a reference, calculating the pose and other information of the new point cloud within the local map.

[0117] In step S709, new point clouds are collected through the camera and lidar to obtain new environmental perception data, and a new round of processing is started, so that the entire system can continuously perceive environmental changes and update maps and posture information.

[0118] The point cloud data of the current frame after keyframe determination is integrated into the existing point cloud map. Each time a keyframe point cloud data is determined, it is integrated into the point cloud map; by integrating the sum of the transformations of the new point cloud data with the previously accumulated point cloud map, the point cloud map is continuously updated and improved, more accurately reflecting the actual environment.

[0119] The method for generating a construction environment map in the above embodiment is based on the image captured by the image acquisition device of the construction environment and the point cloud captured by the radar device of the construction environment, and the rotation matrix and transformation matrix between the point cloud and the image key frame are fused to determine whether the captured point cloud frame is a key point cloud frame, and a map corresponding to the construction environment is generated through the key point cloud frame; in view of the fact that key point cloud frame data used to construct the map may be lost during the mapping process, the key point cloud frame is determined based on the point cloud and the image, which effectively reduces the loss of key point cloud frame data, improves the robustness of the construction environment mapping, and improves the accuracy of the map; it can perform environmental mapping without using high-precision inertial navigation equipment, with low calculation complexity and high real-time performance; the starting position of the laser radar can be used as the coordinate origin of the map, and the point cloud collected by the laser radar during movement is aligned with the local map, and the camera is used to collect images for synchronous key frame detection and fusion, effectively reducing the probability of missed key frame detection and improving the robustness of environmental mapping.

[0120] In some embodiments, as Figure 8 As shown, the present disclosure provides a construction environment map generation device 80, comprising a data acquisition module 801, an information determination module 802, a key data determination module 803, and a map generation module 804. The data acquisition module 801 acquires images of the construction environment captured by an image acquisition device and point clouds of the construction environment captured by a radar device, and performs time synchronization and coordinate system conversion on the captured images and point clouds.

[0121] The information determination module 802 determines a first rotation matrix between the currently acquired image and the last determined key image, and determines a first transformation matrix and a first three-dimensional space rotation angle between the currently acquired point cloud and the point cloud local map.

[0122] The key data determination module 803 determines whether the currently acquired point cloud frame is a key point cloud frame based on the first rotation matrix, the first transformation matrix, and the first three-dimensional space rotation angle; if the currently acquired point cloud frame is a key point cloud frame, then the currently acquired image is determined to be a key image;

[0123] When the currently acquired point cloud frame is a key point cloud frame, the map generation module 804 fuses the currently acquired point cloud frame with the point cloud local map to generate a map corresponding to the construction environment.

[0124] In some embodiments, the data acquisition module 801 uses the Network Time Protocol (NTP) server to perform time calibration processing on the image acquisition device and the radar device, so as to perform time synchronization processing on the acquired images and point clouds; the data acquisition module 801 uses the PNP algorithm to determine the transformation matrix between the image acquisition device and the radar device, so as to perform coordinate system conversion processing on the acquired images and point cloud frames.

[0125] The information determination module 802 estimates the pose of the image acquisition device based on the epipolar geometry constraint, and determines a first rotation matrix between the currently acquired image and the last determined key image and a three-dimensional space rotation angle corresponding to the first rotation matrix.

[0126] The information determination module 802 determines a first transformation matrix and a second three-dimensional space rotation angle between the currently collected point cloud and the point cloud local map through a normal distribution transform (NDT) algorithm.

[0127] The key data determination module 803 determines the second transformation matrix between the currently acquired point cloud and the point cloud local map based on the first rotation matrix, the first transformation matrix and the first three-dimensional space rotation angle; the key data determination module 803 determines the transformation matrix difference information based on the second transformation matrix and the first transformation matrix; the key data determination module 803 determines whether the currently acquired point cloud frame is a key point cloud frame based on the transformation matrix difference information.

[0128] For example, the key data determination module 803 converts the first rotation matrix into a second rotation matrix in the radar device coordinate system based on the calibration rotation matrix between the image acquisition device and the radar device, and determines the second three-dimensional space rotation angle corresponding to the second rotation matrix; the key data determination module 803 performs weighted averaging processing on the first three-dimensional space rotation angle and the second three-dimensional space rotation angle to obtain the third three-dimensional space rotation angle; the key data determination module 803 determines the second transformation matrix based on the displacement vector of the first transformation matrix and the third three-dimensional space rotation angle.

[0129] The key data determination module 803 calculates the product of the inverse matrix of the first transformation matrix and the second transformation matrix as the transformation matrix difference information. The key data determination module 803 calculates the sum of the absolute values ​​of the displacements in the three axial directions in the transformation matrix difference information as the shift information. The key data determination module 803 calculates the sum of the absolute values ​​of the three spatial rotation angles in the transformation matrix difference information as the rotation angle information. If the shift information is greater than the shift threshold and / or the rotation angle information is greater than the rotation angle threshold, the key data determination module 803 determines that the currently acquired point cloud frame is a key point cloud frame.

[0130] The map generation module 804 performs a transformation process on the currently collected point cloud according to the second transformation matrix, and fuses the currently collected point cloud that has undergone the transformation process into a point cloud local map.

[0131] In some embodiments, as Figure 9 As shown, the apparatus for generating a construction environment map may include a memory 901, a processor 902, a communication interface 903, and a bus 904. The memory 901 is used to store instructions, and the processor 902 is coupled to the memory 901. The processor 902 is configured to execute the above-mentioned method for generating a construction environment map based on the instructions stored in the memory 901.

[0132] Memory 901 can be high-speed RAM, non-volatile memory, or a memory array. Memory 901 can also be divided into blocks, which can be combined into virtual volumes according to certain rules. Processor 902 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the disclosed method for generating a construction environment map or the method for predicting atmospheric pollution information.

[0133] In some embodiments, the present disclosure provides a construction machine, including the device for generating a construction environment map as described in any of the above embodiments. The construction machine may be an underground scraper or the like.

[0134] In some embodiments, the present disclosure provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the method for generating a construction environment map as described in any of the above embodiments.

[0135] Computer-readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or components, or any combination thereof. More specific examples (non-exhaustive) of readable storage media can include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0136] The embodiments of the present disclosure may also be a computer program product, which includes computer program instructions. When the computer program instructions are executed by a processor, the processor executes the steps of the method according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0137] The basic principles of the present disclosure have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in this disclosure are merely illustrative and not restrictive, and should not be construed as necessarily possessed by each embodiment of the present disclosure. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, rather than as limitations. These details do not limit the present disclosure to necessarily being implemented using these specific details.

[0138] Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar parts between the various embodiments are sufficient. For system embodiments, since they largely correspond to method embodiments, their description is relatively simple. For relevant parts, references to the description of the method embodiments are sufficient.

[0139] The block diagrams of the devices, devices, equipment, and systems involved in this disclosure are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, devices, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "include," "comprise," "have," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0140] It should also be noted that in the apparatus, device, and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0141] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0142] The above description has been provided for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, it will be understood by those skilled in the art that the above embodiments are merely illustrative and do not limit the scope of the present disclosure. It will be understood by those skilled in the art that the above embodiments may be combined, modified, or replaced without departing from the scope and essence of the present disclosure.

Claims

1. A method for generating a construction environment map, comprising: Acquire images of the construction environment collected by the image acquisition device and point clouds of the construction environment collected by the radar device, and perform time synchronization and coordinate system conversion processing on the collected images and point clouds; Determine a first rotation matrix between the currently acquired image and the last determined key image, and determine a first transformation matrix and a first three-dimensional spatial rotation angle between the currently acquired point cloud and the point cloud local map; determining whether the currently acquired point cloud frame is a key point cloud frame based on the first rotation matrix, the first transformation matrix, and the first three-dimensional space rotation angle; wherein, if the currently acquired point cloud frame is determined to be a key point cloud frame, determining the currently acquired image to be a key image; In a case where the currently acquired point cloud frame is a key point cloud frame, the currently acquired point cloud is fused with the point cloud local map to generate a map corresponding to the construction environment.

2. The generation method according to claim 1, wherein: The determining, based on the first rotation matrix, the first transformation matrix, and the first three-dimensional space rotation angle, whether the currently acquired point cloud frame is a key point cloud frame includes: Determining a second transformation matrix between the currently collected point cloud and the point cloud local map according to the first rotation matrix, the first transformation matrix, and the first three-dimensional space rotation angle; determining transformation matrix difference information according to the second transformation matrix and the first transformation matrix; Determine whether the currently acquired point cloud frame is a key point cloud frame according to the transformation matrix difference information.

3. The generation method according to claim 2, wherein: Determining a second transformation matrix between the currently collected point cloud and the point cloud local map according to the first rotation matrix, the first transformation matrix, and the first three-dimensional space rotation angle includes: Converting the first rotation matrix into a second rotation matrix in the radar device coordinate system according to a calibration rotation matrix between the image acquisition device and the radar device, and determining a second three-dimensional space rotation angle corresponding to the second rotation matrix; performing weighted averaging processing on the first three-dimensional space rotation angle and the second three-dimensional space rotation angle to obtain a third three-dimensional space rotation angle; The second transformation matrix is ​​determined according to the displacement vector of the first transformation matrix and the third three-dimensional space rotation angle.

4. The generation method according to claim 2, wherein: The determining, according to the second transformation matrix and the first transformation matrix, transformation matrix difference information includes: The product of the inverse matrix of the first transformation matrix and the second transformation matrix is ​​calculated as the transformation matrix difference information.

5. The generation method according to claim 2, wherein: Determining whether the currently acquired point cloud frame is a key point cloud frame according to the transformation matrix difference information includes: Calculating the sum of the absolute values ​​of the displacements in the three axis directions in the transformation matrix difference information as the shift information; Calculating the sum of the absolute values ​​of the three spatial rotation angles in the transformation matrix difference information as the rotation angle information; In a case where the displacement information is greater than a displacement threshold and / or the rotation angle information is greater than a rotation angle threshold, it is determined that the currently acquired point cloud frame is a key point cloud frame.

6. The generation method according to claim 3, wherein: The fusing of the currently collected point cloud with the point cloud local map includes: performing conversion processing on the currently acquired point cloud according to the second transformation matrix; The currently collected point cloud that has undergone conversion processing is fused into the point cloud local map.

7. The generation method according to claim 1, wherein: Determining a first rotation matrix between a currently acquired image and a last determined key image includes: The pose of the image acquisition device is estimated based on epipolar geometry constraints, and the first rotation matrix between the currently acquired image and the last determined key image and the three-dimensional space rotation angle corresponding to the first rotation matrix are determined.

8. The generation method according to claim 1, wherein: Determining a first transformation matrix and a first three-dimensional space rotation angle between the currently collected point cloud and the point cloud local map includes: The first transformation matrix and the second three-dimensional space rotation angle between the currently collected point cloud and the point cloud local map are determined by a normal distribution transform (NDT) algorithm.

9. The generation method according to claim 1, wherein: The image acquisition device periodically acquires images based on a preset sampling time interval; The radar device periodically collects point clouds based on the sampling time interval.

10. The generation method according to claim 1, wherein: Time synchronization of acquired images and point clouds includes: The image acquisition device and the radar device are time-calibrated using a Network Time Protocol (NTP) server to synchronize the acquired images and point clouds.

11. A device for generating a construction environment map, comprising: A data acquisition module is used to obtain images of the construction environment collected by the image acquisition device and point clouds of the construction environment collected by the radar device, and perform time synchronization and coordinate system conversion processing on the collected images and point clouds; an information determination module, configured to determine a first rotation matrix between a currently acquired image and a previously determined key image, and to determine a first transformation matrix and a first three-dimensional spatial rotation angle between a currently acquired point cloud and a point cloud local map; a key data determination module, configured to determine whether the currently acquired point cloud frame is a key point cloud frame based on the first rotation matrix, the first transformation matrix, and the first three-dimensional space rotation angle; wherein, if the currently acquired point cloud frame is a key point cloud frame, determining that the currently acquired image is a key image; A map generation module is used to fuse the currently acquired point cloud frame with the point cloud local map when the currently acquired point cloud frame is a key point cloud frame, so as to generate a map corresponding to the construction environment.

12. A device for generating a construction environment map, comprising: Memory; and a processor coupled to the memory, wherein the processor is configured to execute the method according to any one of claims 1 to 10 based on instructions stored in the memory.

13. An engineering machine comprising: A device for generating a construction environment map as claimed in claim 11 or 12.

14. A computer-readable storage medium storing computer instructions, wherein the instructions are executed by a processor to implement the steps of the method according to any one of claims 1 to 10.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images