Map construction method and device, electronic equipment and storage medium

Abstract

The invention discloses a map construction method and device, electronic equipment and a storage medium. The method comprises the following steps: determining target data and image data; performing fusion processing on the target data to obtain a reference three-dimensional map; and coloring the reference three-dimensional map through the image data to obtain a target three-dimensional map. According to the method, the reference three-dimensional map is constructed according to the acquired surrounding environment of the target object and the related data of the motion of the target object, and the reference three-dimensional map is colored according to the acquired image of the surrounding environment data of the target object, so that the high-precision three-dimensional map can be acquired in real time, and the map construction efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a map construction method, device, electronic equipment and storage medium. Background Art

[0002] 3D map construction systems have a wide range of applications in many fields, such as geographic information systems, architectural surveying and mapping, indoor navigation, virtual reality, etc. In the field of autonomous driving, they are of great significance for many applications such as obstacle detection, traffic scene understanding and autonomous navigation. The main goal of these systems is to obtain 3D geometric information of objects in the environment for analysis, modeling and visualization.

[0003] Three-dimensional map construction technology is mainly divided into contact and non-contact. Contact technology reconstructs objects by touching the surface of the object with a special probe. These technologies are slow, and direct contact with the surface of the object is not feasible in many applications. In order to overcome these limitations, non-contact technology has been developed: it can be divided into passive technology and active technology. Passive technologies such as stereo vision construct three-dimensional maps from multiple two-dimensional images, and they only rely on ambient light. Active technology reconstructs the scene by using artificial light sources such as structured light such as lasers. It can obtain the distance and depth information of objects by sending laser beams and measuring the time of reflected light, which makes them more accurate, especially for textureless surfaces, which is almost impossible to use passive technology for map construction. Therefore, in the existing technology, lidar is widely used for three-dimensional map construction, but pure visual sensors are easily affected by light and cannot directly obtain accurate three-dimensional information. Pure lidar sensors can only provide point cloud information without color information, which leads to low authenticity and accuracy of the obtained three-dimensional map. Summary of the invention

[0004] The present invention provides a map construction method, device, electronic equipment and storage medium to solve the problem of low accuracy and authenticity in three-dimensional map construction.

[0005] According to one aspect of the present invention, a map construction method is provided, comprising:

[0006] Determine target data and image data; the target data is used to characterize the operating state data of the target object acquired by different sensors; the image data is used to characterize the environment around the target object;

[0007] The target data is fused to obtain a reference three-dimensional map; the reference three-dimensional map is a point cloud image of the target object and the surrounding environment of the target object; the fusion process includes: posture estimation and target positioning data fusion; the posture estimation is used to determine the target posture change according to the target point cloud data and the target motion data; the target posture change is used to characterize the changes in the position and posture of the target object during the motion process; the target positioning data fusion is used to fuse the target positioning data with the target posture change;

[0008] The reference three-dimensional map is colored by the image data to obtain a target three-dimensional map.

[0009] According to another aspect of the present invention, there is provided a map construction device, comprising:

[0010] A data acquisition module, used to determine target data and image data; the target data is used to characterize the operating state data of the target object acquired by different sensors; the image data is used to characterize the environment around the target object;

[0011] A reference map construction module is used to perform fusion processing on the target data to obtain a reference three-dimensional map; the reference three-dimensional map is a point cloud image of the target object and the surrounding environment of the target object; the fusion processing includes: posture estimation and target positioning data fusion; the posture estimation is used to determine the target posture change according to the target point cloud data and the target motion data; the target posture change is used to characterize the changes in the position and posture of the target object during the motion process; the target positioning data fusion is used to fuse the target positioning data with the target posture change;

[0012] The three-dimensional map construction module is used to color the reference three-dimensional map using the image data to obtain a target three-dimensional map.

[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0014] at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the map construction method described in any embodiment of the present invention.

[0017] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the map construction method described in any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention determines the target data and image data; fuses the target data to obtain a reference three-dimensional map, and adopts high-frequency data fusion to ensure the accuracy of the constructed reference three-dimensional map; and colors the reference three-dimensional map through the image data to obtain a target three-dimensional map, and coloring the reference three-dimensional map can obtain map information that fits the actual scene. This method constructs a reference three-dimensional map based on the acquired data related to the surrounding environment of the target object and the target object's own movement, and colors the reference three-dimensional map based on the image of the collected data of the surrounding environment of the target object, which can obtain a high-precision three-dimensional map in real time while also improving the efficiency of map construction.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 A flowchart of a map construction method provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of a three-dimensional map provided by an embodiment of the present invention;

[0023] Figure 3 A structural diagram of a handheld map building device provided by an embodiment of the present invention;

[0024] Figure 4 A top view of a handheld map building device provided by an embodiment of the present invention;

[0025] Figure 5 A bottom view of a handheld map building device provided by an embodiment of the present invention;

[0026] Figure 6 A side view of a handheld map building device provided by an embodiment of the present invention;

[0027] Figure 7 A schematic diagram of the structure of a map construction device provided by an embodiment of the present invention;

[0028] Figure 8 A schematic diagram of the structure of an electronic device for implementing the map building method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] Figure 1 This is a flowchart of a map construction method provided by an embodiment of the present invention. This embodiment is applicable to the case of constructing the surrounding environment of a target object. The method can be executed by a map construction device. The map construction device can be implemented in the form of hardware and / or software. The map construction device can be configured in any electronic device with network communication function. Figure 1 As shown, the method includes:

[0032] S110, determining target data and image data.

[0033] The target data is used to represent the running state data of the target object collected by different sensors. Furthermore, the target data is located in the same coordinate system.

[0034] Among them, image data is used to characterize the environment around the target object.

[0035] Furthermore, the target data includes: target point cloud data, target motion data and positioning data.

[0036] Specifically, coordinate transformation is performed on the running state data of the target object acquired by each sensor to obtain the target data. The target object and its surrounding environment are photographed by an image acquisition device to obtain image data.

[0037] S120: Perform fusion processing on the target data to obtain a reference three-dimensional map.

[0038] The reference three-dimensional map is a point cloud image of the target object and its surroundings.

[0039] Among them, the fusion processing includes: pose estimation and target positioning data fusion.

[0040] Among them, pose estimation is used to determine the target pose change based on the target point cloud data and target motion data. The target pose change is used to characterize the changes in the position and posture of the target object during the motion process.

[0041] Among them, target positioning data fusion is used to fuse target positioning data with target posture changes.

[0042] Specifically, the target motion data is first pre-integrated to obtain the posture change of the target object between adjacent frames, and the target point cloud data is dedistorted according to the acquired posture change, and a local plane map of the target is constructed. The target positioning data is fused with the reference local map to obtain a reference 3D map.

[0043] S130 , coloring the reference three-dimensional map using the image data to obtain a target three-dimensional map.

[0044] Specifically, the coordinate information of the point cloud data in the reference three-dimensional map is determined, and the point cloud data is converted into a pixel coordinate system through the transformation relationship between the point cloud coordinates in the point cloud data and the pixel coordinates in the image data, and the corresponding color values ​​are matched from the reference color values ​​according to the converted coordinate information, and the reference three-dimensional map is colored according to the obtained color values ​​to obtain the target three-dimensional map.

[0045] For example, Figure 2 As shown, the information of the vehicle and its surrounding environment can be displayed more clearly.

[0046] Furthermore, the constructed target three-dimensional map can be stored in the controller or in an external USB flash drive and solid-state hard drive, and can be transferred and used in the form of USB, network port, etc.

[0047] Optionally, before determining the target data and the image data, steps A1-A4 are included:

[0048] Step A1: determine a first transformation relationship.

[0049] The first transformation relationship is used to characterize the transformation relationship between the three-dimensional polar coordinates of the point cloud acquisition device and the reference coordinate system.

[0050] The reference coordinate system is the coordinate system of the handheld map building device.

[0051] Among them, the handheld map building device is equipped with a point cloud acquisition device and an image acquisition device.

[0052] Furthermore, if Figure 3 As shown, the handheld map building device includes: a point cloud acquisition device 1: used to collect point cloud data of the target object and its surrounding environment; an image acquisition device 2: used to collect image data of the target object and its surrounding environment; a detachable handle 3: used to achieve interaction with the user; a shell 4: used to encapsulate the handheld map building device and protect the single-chip microcomputer 5 and the controller 6; the single-chip microcomputer 5: used to perform data processing and calculation; the controller 6: used to control the point cloud acquisition device 1 and the image acquisition device 2.

[0053] Further, such as Figure 4 , Figure 5 and Figure 6 As shown, there are respectively a top view, a bottom view and a side view of the handheld map building device. It can be seen from the figure that the point cloud acquisition device 1 is located at the top of the handheld map building device; the single-chip microcomputer 5 and the controller 6 are arranged in the shell 4, located at the bottom of the point cloud acquisition device 1; and the detachable handle 3 is arranged in the groove at the lower part of the shell 4.

[0054] Specifically, the first transformation relationship is constructed according to the association relationship between the coordinate system of the point cloud acquisition device and the reference coordinate system.

[0055] Furthermore, the first transformation relationship is: [x, y, z, roll, pitch, yaw].

[0056] Among them, x, y, z are the distance values ​​of the point cloud acquisition device from the reference coordinate system (the center of the handheld map building device) in three directions; roll, pitch, yaw are the attitude angles of the point cloud acquisition device. In the calibrated laser point cloud, the ground is in the plane of z=0, and xy is within the detection range.

[0057] Step A2: calibrate the point cloud acquisition device according to the first transformation relationship to obtain first reference point cloud data.

[0058] Specifically, the point cloud data acquired by the point cloud acquisition device is converted according to the first transformation relationship, and after the conversion is completed, the first reference point cloud data is obtained.

[0059] Furthermore, since the collected point cloud data is represented in the form of three-dimensional polar coordinates, it is necessary to convert it into a Cartesian coordinate system through a coordinate system conversion formula before calibration, and then calibrate it.

[0060] Among them, the coordinate system conversion formula can be expressed by the following formula:

[0061]

[0062] Among them, x i is the x coordinate of the i-th point in the point cloud data; y i is the y coordinate of the i-th point in the point cloud data; z i is the z coordinate of the i-th point in the point cloud data; r i is the Euclidean distance from the i-th point in the point cloud data to the center of the laser radar; θ i is the yaw emission angle of the i-th point in the point cloud data; is the pitch emission angle of the i-th point in the point cloud data.

[0063] The above steps calibrate the point cloud acquisition device according to the reference coordinate system. This is because the handheld map construction device constructs a three-dimensional map with the center of the handheld map construction device as the origin, and the point cloud data collected by the point cloud acquisition device is relative to the coordinate system of the point cloud acquisition device. Therefore, using the point cloud data collected by an uncalibrated point cloud acquisition device to construct a three-dimensional map will result in positional deviations, thereby causing the problem of low accuracy of the obtained three-dimensional map.

[0064] Step A3: construct a second transformation relationship based on the first reference point cloud data and the historical motion data.

[0065] The second transformation relationship is used to characterize the relationship between the point cloud data and the motion data.

[0066] Specifically, the second transformation relationship is constructed according to the transformation relationship between the first reference point cloud data and the historical motion data.

[0067] Step A4: construct a third transformation relationship based on the first reference point cloud data and the historical image data.

[0068] The third transformation relationship is used to represent the transformation relationship between the point cloud coordinates in the point cloud data and the pixel coordinates in the image data.

[0069] Specifically, the third transformation relationship is constructed according to the conversion relationship between the coordinate information of the point cloud in the first reference point cloud data and the pixel coordinate information of the pixel point in the historical image data.

[0070] Furthermore, the third transformation relationship can be expressed by the following formula:

[0071]

[0072] Among them, Z c The axis is parallel to the camera optical axis and perpendicular to the image plane; (u0, v0) is the coordinate of the origin of the image coordinate system in the pixel coordinate system; d x and d y are the length and width of a single pixel in the image plane; R is the 3x3 rotation matrix, T is the 3x1 translation matrix; f is the focal length of the camera

[0073] In the above steps, the conversion relationship is constructed because the coordinate systems of various sensors are not the same. Using data in different coordinate systems is prone to time deviation problems during data fusion, thereby reducing the accuracy of three-dimensional map construction.

[0074] Optionally, determining target data includes steps B1-B3:

[0075] Step B1, determining second reference point cloud data, reference motion data and target positioning data.

[0076] The second reference point cloud data is point cloud data collected by the point cloud acquisition device.

[0077] Specifically, the second reference point cloud data is acquired from the point cloud acquisition device; the reference motion data is acquired from the motion data acquisition device; and the target positioning data is acquired from the positioning device.

[0078] Among them, the point cloud acquisition device can adopt a laser radar sensor; the motion data acquisition device can adopt an IMU (Inertial Measurement Unit).

[0079] Furthermore, since the sampling frequency of the original data points of the laser radar is very high, it is unlikely to process every sampled point in real time. Therefore, the data collection interval is set to 20ms (i.e. 50hz), that is, the data points are accumulated within a certain period of time, and then processed at one time. Such a group of accumulated points is called a scan; since the IMU collects motion data and point cloud data tightly coupled, the IMU output is set to 100hz to ensure that each frame of laser point cloud data matches the posture provided by the IMU.

[0080] Step B2: performing coordinate transformation on the second reference point cloud data according to the first transformation relationship to obtain target point cloud data.

[0081] Specifically, the coordinates of the collected second reference point cloud data are changed according to the first transformation relationship, and the second reference point cloud data are converted into a reference coordinate system to obtain the target point cloud data.

[0082] Step B3: performing coordinate transformation on the reference motion data according to the second transformation relationship to obtain target motion data.

[0083] Specifically, the coordinates of the collected reference motion data are changed according to the second transformation relationship, and the reference motion data is converted into a reference coordinate system to obtain the target motion data.

[0084] The above steps convert the acquired reference motion data and reference point cloud data, which can better associate the data collected by the lidar sensor and the IMU and provide more accurate map information.

[0085] Optionally, the target data is fused to obtain a reference three-dimensional map, including steps C1-C5:

[0086] Step C1: pre-integrate and dedistort the target data to obtain corrected point cloud data.

[0087] Among them, the pre-integration process is used to determine the relative pose transformation between two key frames. The de-distortion process is used to correct the distortion data in the target point cloud data.

[0088] Specifically, the target motion data is pre-integrated to obtain the posture change of the target object between adjacent key frames, and the target point cloud data is dedistorted according to the obtained posture change. The distortion-corrected target point cloud data and target motion data are subjected to posture estimation to obtain the target posture transformation.

[0089] Furthermore, before the target motion data is pre-integrated, the target motion data at two adjacent moments are used, and the discrete median method is adopted as the acceleration and angular velocity input into the motion equation, and the zero bias is removed.

[0090] Step C2: construct a reference local map based on the corrected point cloud data.

[0091] Specifically, first, a local map is constructed based on the corrected point cloud data; then, point cloud data of key points are obtained from the corrected point cloud data, and feature description information is constructed based on the point cloud data of key points. Feature description information corresponding to point cloud data of different frames is matched, and the position of the corrected point cloud data in the local map is fine-tuned based on the matched feature description information. Furthermore, by continuously adding new corrected point cloud data and corresponding feature description information, the local map is gradually updated to obtain a reference local map.

[0092] Step C3: Match the corrected point cloud data with the reference local map and determine the point cloud matching error.

[0093] Specifically, the distance between each point in the corrected point cloud data and the distance between each point and the reference local map are constructed according to the corrected point cloud data and the reference local map, and the point-to-surface residual item and the point-to-point residual item are determined according to the calculated distance information, and the point cloud matching error is determined according to the point-to-surface residual item and the point-to-point residual item.

[0094] Step C4: Update the reference local map according to the point cloud matching error to obtain the target local plane map.

[0095] Specifically, the target point cloud data with a position deviation in the reference local map is corrected according to the point cloud matching error to obtain the target local plane map.

[0096] Step C5: determining a reference three-dimensional map according to the target local planar map and the target positioning data.

[0097] Specifically, the target positioning data is fused into the target local plane map according to the position of the point cloud to obtain a reference three-dimensional map.

[0098] Furthermore, the fusion of target positioning data realizes the world coordinate determination of the map, which is applicable to a variety of environments.

[0099] Optionally, pre-integration and de-distortion processing is performed on the target data, including steps D1-D3:

[0100] Step D1: align the target point cloud data and the target motion data with the timestamps.

[0101] Specifically, timestamps are aligned according to the acquisition time of the target point cloud data and the target motion data.

[0102] In the above steps, the timestamp alignment is performed to ensure that the target point cloud data and the target motion data are under the same time reference.

[0103] Step D2: pre-integrate the aligned target motion data to obtain the initial posture change.

[0104] Among them, the initial posture transformation can represent the relative changes in the posture, speed and position of the target object within a preset time period.

[0105] Specifically, the angular velocity and acceleration corresponding to every two adjacent key frames of the aligned target motion data are pre-integrated to obtain the initial posture change.

[0106] Step D3: Correct the distortion data in the target point cloud data according to the initial posture change.

[0107] Specifically, the attitude matrix of the target point cloud data at time t is determined according to the change in initial posture, and the coordinates of each point are transformed according to the acquired attitude matrix and the coordinate information of the target point cloud data to complete the correction of the distorted data. After the correction is completed, the corrected data is converted to the coordinate system of the point cloud acquisition device.

[0108] Furthermore, each target point cloud data in a frame is projected to the last moment. The pose of each point cloud is obtained. There will be many target point cloud data between two frames of target motion data. Through interpolation, the time difference between the target point cloud data and the head frame of the target motion data is obtained, and the pose of the target point cloud data under the target motion data is obtained by integration, and then transferred to the coordinate system of the point cloud acquisition device through external reference.

[0109] The above step of dedistorting the target point cloud data is because the target point cloud data is not obtained instantaneously. Therefore, when collecting data, if the position of the handheld map building device changes but is not captured by the point cloud acquisition device, it will cause position deviation in the acquired point cloud data.

[0110] Optionally, matching the corrected point cloud data with the reference local map and determining the point cloud matching error comprises steps E1-E4:

[0111] Step E1: Determine a reference point corresponding to the target point from the reference local map.

[0112] The target point is any point in the target point cloud data.

[0113] The reference point is the location of the target point in the reference local map.

[0114] Specifically, the corresponding reference point is matched in the local map according to the target point star reference.

[0115] Step E2: determining a first distance according to the reference point and the target point.

[0116] The first distance is used to represent the Euclidean distance between the reference point and the target point.

[0117] Specifically, the first distance is calculated according to the reference point position information and the target point position information.

[0118] Step E3: determining a second distance according to the target point and the reference local map.

[0119] The second distance is used to represent the distance from the target point to the reference local map.

[0120] Specifically, the second distance is calculated according to the location information of the target point and with reference to the local map.

[0121] Step E4: determining the point cloud matching error according to the first distance and the second distance.

[0122] Specifically, the point-to-surface residual and the point-to-point residual are determined according to the first distance and the second distance, and the point cloud matching error is determined according to the point-to-surface residual and the point-to-point residual.

[0123] Furthermore, the point cloud matching error can be characterized by the root mean square error and the mean absolute error.

[0124] Optionally, coloring the reference three-dimensional map by using the image data to obtain the target three-dimensional map includes steps F1-F4:

[0125] Step F1, determining a reference color value according to image data.

[0126] The reference color value is the color value corresponding to each pixel in the image data.

[0127] Furthermore, the reference color value corresponds one-to-one to the position information of the pixel points in the image data.

[0128] Specifically, the color information of each pixel is obtained from the image data to obtain a reference color value.

[0129] Step F2: convert the target point cloud data into pixel coordinates according to the third transformation relationship to obtain reference image coordinates.

[0130] Specifically, coordinate transformation is performed on each point cloud in the target point cloud data according to the third transformation relationship, and converted into pixel coordinates to obtain reference image coordinates.

[0131] Step F3: Match the target color value from the reference color value according to the reference image coordinates.

[0132] Specifically, a color value corresponding to the reference image coordinate is matched from the reference color values ​​according to the reference image coordinate as the target color value.

[0133] Step F4: Color the reference three-dimensional map using the target color value to obtain a target three-dimensional map.

[0134] Specifically, the obtained target color value is assigned to the point cloud corresponding to the reference image coordinates, and the point cloud is colored according to the color corresponding to the target color value. When all the point cloud data are colored, the target three-dimensional map is obtained.

[0135] The technical solution of this embodiment determines the target data and image data; fuses the target data to obtain a reference three-dimensional map, and adopts high-frequency data fusion to ensure the accuracy of the constructed reference three-dimensional map; and colors the reference three-dimensional map through image data to obtain a target three-dimensional map. Coloring the reference three-dimensional map can obtain map information that fits the actual scene. This method constructs a reference three-dimensional map based on the acquired data related to the surrounding environment of the target object and the target object's own movement, and colors the reference three-dimensional map based on the image of the collected data of the surrounding environment of the target object. It can obtain a high-precision three-dimensional map in real time and improve the efficiency of map construction.

[0136] Figure 7 This is a schematic diagram of the structure of a map construction device provided by an embodiment of the present invention. This embodiment is applicable to the case of constructing the surrounding environment of a target object. The map construction device can be implemented in the form of hardware and / or software, and the map construction device can be configured in any electronic device with network communication function. Figure 7 As shown, the device includes: a data acquisition module 210, a reference map construction module 220 and a three-dimensional map construction module 230, wherein:

[0137] Data acquisition module 210: used to determine target data and image data; the target data is used to characterize the operating state data of the target object acquired by different sensors; the image data is used to characterize the environment around the target object;

[0138] Reference map construction module 220: used to perform fusion processing on target data to obtain a reference three-dimensional map; the reference three-dimensional map is a point cloud image of the target object and the surrounding environment of the target object; the fusion processing includes: posture estimation and target positioning data fusion; posture estimation is used to determine the target posture change according to the target point cloud data and the target motion data; the target posture change is used to characterize the changes in the position and posture of the target object during the motion process; the target positioning data fusion is used to fuse the target positioning data with the target posture change;

[0139] The three-dimensional map construction module 230 is used to color the reference three-dimensional map through image data to obtain a target three-dimensional map.

[0140] Optionally, a map building device includes:

[0141] A first transformation relationship determination module: used to determine a first transformation relationship; the first transformation relationship is used to characterize the transformation relationship between the three-dimensional polar coordinates of the point cloud acquisition device and the reference coordinate system; the reference coordinate system is the coordinate system of the handheld map building device; the handheld map building device is equipped with a point cloud acquisition device and an image acquisition device;

[0142] Reference calibration parameter determination module: used to calibrate the point cloud acquisition device according to the first transformation relationship to obtain first reference point cloud data;

[0143] A second transformation relationship determination module: used to construct a second transformation relationship based on the first reference point cloud data and the historical motion data; the second transformation relationship is used to characterize the relationship between the point cloud data and the posture data;

[0144] The third transformation relationship determination module is used to construct a third transformation relationship based on the first reference point cloud data and the historical image data; the third transformation relationship is used to characterize the transformation relationship between the point cloud coordinates in the point cloud data and the pixel coordinates in the image data.

[0145] Optionally, the data acquisition module 210 includes:

[0146] Data acquisition unit: used to determine the second reference point cloud data, reference motion data and target positioning data; the second reference point cloud data is the point cloud data collected by the point cloud acquisition device;

[0147] A target point cloud data determination unit is used to perform coordinate transformation on the second reference point cloud data according to the first transformation relationship to obtain target point cloud data;

[0148] The target motion data determination unit is used to perform coordinate transformation on the reference motion data according to the second transformation relationship to obtain the target motion data.

[0149] Optionally, the reference map construction module 220 includes:

[0150] Correction point cloud data determination unit: used for pre-integration and de-distortion processing of target data to obtain correction point cloud data; pre-integration processing is used to determine the relative posture transformation between two key frames; de-distortion processing is used to correct the distortion data in the target point cloud data;

[0151] Reference local map construction unit: used to construct a reference local map according to the corrected point cloud data;

[0152] Point cloud matching error determination unit: used to match the corrected point cloud data with the reference local map and determine the point cloud matching error;

[0153] Target local plane map determination unit: used for updating the reference local map according to the point cloud matching error to obtain the target local plane map;

[0154] Reference three-dimensional map determination unit: used to determine the reference three-dimensional map according to the target local plane map and target positioning data.

[0155] Optionally, the correction point cloud data determination unit includes:

[0156] Time alignment subunit: used to align the timestamps of target point cloud data and target motion data;

[0157] Initial posture change determination subunit: used to pre-integrate the aligned target motion data to obtain the initial posture change;

[0158] Distortion correction subunit: used to correct the distortion data in the target point cloud data according to the initial pose change.

[0159] Optionally, a point cloud matching error determination unit comprises:

[0160] Reference point determination subunit: used to determine a reference point corresponding to a target point from a reference local map; the target point is any point in the target point cloud data;

[0161] A first distance determination subunit: used to determine a first distance according to a reference point and a target point; the first distance is used to represent the Euclidean distance between the reference point and the target point;

[0162] A second distance determination subunit: used to determine a second distance according to the target point and the reference local map; the second distance is used to represent the distance from the target point to the reference local map;

[0163] Point cloud matching error determination subunit: used to determine the point cloud matching error according to the first distance and the second distance.

[0164] Optionally, the three-dimensional map construction module 230 includes:

[0165] Reference color value determination unit: used to determine a reference color value according to the image data; the reference color value is a color value corresponding to each pixel in the image data;

[0166] A reference image coordinate determination unit: used for converting the target point cloud data into pixel point pixel coordinates according to a third transformation relationship to obtain reference image coordinates;

[0167] A target color value determination unit: used for matching a target color value from a reference color value according to a reference image coordinate;

[0168] Target three-dimensional map construction unit: used for coloring the reference three-dimensional map by target color value to obtain the target three-dimensional map.

[0169] The map construction device provided in the embodiment of the present invention can execute the map construction method provided in any embodiment of the present invention, and has the corresponding functions and beneficial effects of executing the map construction method. For detailed process, please refer to the relevant operations of the map construction method in the above embodiment.

[0170] Figure 8A schematic diagram of the structure of an electronic device for implementing the map construction method of an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or claimed herein.

[0171] like Figure 8 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0172] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0173] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a map construction method.

[0174] In some embodiments, the map construction method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the map construction method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the map construction method in any other appropriate manner (e.g., by means of firmware).

[0175] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0176] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0177] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0178] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0179] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0180] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0181] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0182] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A map construction method, characterized in that: include: Determine target data and image data; the target data is used to characterize the operating state data of the target object acquired by different sensors; The image data is used to characterize the environment around the target object; Performing fusion processing on the target data to obtain a reference three-dimensional map; The reference three-dimensional map is a point cloud image of the target object and its surroundings; The fusion processing includes: posture estimation and target positioning data fusion; the posture estimation is used to determine the target posture change according to the target point cloud data and the target motion data; the target posture change is used to characterize the position and posture changes of the target object during the motion process; the target positioning data fusion is used to fuse the target positioning data with the target posture change; The reference three-dimensional map is colored by the image data to obtain a target three-dimensional map.

2. The method according to claim 1, characterized in that Before determining the target data and the image data, the method includes: Determine a first transformation relationship; the first transformation relationship is used to characterize the transformation relationship between the three-dimensional polar coordinates of the point cloud acquisition device and the reference coordinate system; the reference coordinate system is the coordinate system of the handheld map building device; the handheld map building device is equipped with a point cloud acquisition device and an image acquisition device; Calibrate the point cloud acquisition device according to the first transformation relationship to obtain first reference point cloud data; Constructing a second transformation relationship based on the first reference point cloud data and the historical motion data; the second transformation relationship is used to characterize the relationship between the point cloud data and the motion data; A third transformation relationship is constructed based on the first reference point cloud data and the historical image data; the third transformation relationship is used to characterize the transformation relationship between the point cloud coordinates in the point cloud data and the pixel coordinates in the image data.

3. The method according to claim 1, characterized in that The determining of target data includes: Determine second reference point cloud data, reference motion data and target positioning data; the second reference point cloud data is point cloud data collected by the point cloud acquisition device; Performing coordinate transformation on the second reference point cloud data according to the first transformation relationship to obtain target point cloud data; The reference motion data is coordinate transformed according to the second transformation relationship to obtain target motion data.

4. The method according to claim 1, characterized in that: The target data is fused to obtain a reference three-dimensional map, including: Pre-integration and de-distortion processing are performed on the target data to obtain corrected point cloud data; the pre-integration processing is used to determine the relative posture transformation between two key frames; the de-distortion processing is used to correct the distortion data in the target point cloud data; constructing a reference local map based on the corrected point cloud data; Matching the corrected point cloud data with a reference local map and determining a point cloud matching error; The reference local map is updated according to the point cloud matching error to obtain a target local plane map; A reference three-dimensional map is determined based on the target local planar map and the target positioning data.

5. The method according to claim 4, characterized in that The pre-integration and de-distortion processing of the target data includes: Perform timestamp alignment on target point cloud data and target motion data; Pre-integrate the aligned target motion data to obtain the initial pose change; The distortion data in the target point cloud data is corrected according to the initial posture change.

6. The method according to claim 4, characterized in that The step of matching the corrected point cloud data with the reference local map and determining a point cloud matching error comprises: Determine a reference point corresponding to a target point from a reference local map; the target point is any point in the target point cloud data; Determine a first distance according to the reference point and the target point; the first distance is used to represent the Euclidean distance between the reference point and the target point; Determining a second distance according to the target point and the reference local map; the second distance is used to represent the distance from the target point to the reference local map; The point cloud matching error is determined according to the first distance and the second distance.

7. The method according to claim 1, characterized in that The step of coloring the reference three-dimensional map by using the image data to obtain a target three-dimensional map includes: Determine a reference color value according to the image data; the reference color value is the color value corresponding to each pixel in the image data; The target point cloud data is converted into pixel point coordinates according to a third transformation relationship to obtain reference image coordinates; matching a target color value from a reference color value according to the reference image coordinates; The reference three-dimensional map is colored by the target color value to obtain a target three-dimensional map.

8. A map construction device, characterized in that: include: A data acquisition module, used to determine target data and image data; the target data is used to characterize the operating state data of the target object acquired by different sensors; The image data is used to characterize the environment around the target object; A reference map construction module is used to perform fusion processing on the target data to obtain a reference three-dimensional map; The reference three-dimensional map is a point cloud image of the target object and its surroundings; The fusion processing includes: posture estimation and target positioning data fusion; the posture estimation is used to determine the target posture change according to the target point cloud data and the target motion data; the target posture change is used to characterize the position and posture changes of the target object during the motion process; the target positioning data fusion is used to fuse the target positioning data with the target posture change; The three-dimensional map construction module is used to color the reference three-dimensional map using the image data to obtain a target three-dimensional map.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the map construction method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the map construction method according to any one of claims 1 to 7 when executed.

Images